How do I install mods?

Depends on the mod, generally all of them will go into the data folder, but if you get pcx files with them they go into your gfx folder "galciv/gfx or galciv/data, if you download a invasion tactic then it goes into your galciv/data/invasion tactics folder. All the techs, improvements and anomalies go into your data folder but the images pcx go into your gfx folder.

Thx

Are the different mods compatible with eachother? Should I try them one by one?

I will say be careful with the ones that raise your PQ (planet quality to high) those will give you an anomalous score, I have installed 6 or 7 of the whole total, some may cause your game to crash.

Note that any PQ mods on the library have been tested by stardock, so using them independently is no problem. It's just using lots of +PQ mods simultaneously you have to worry about.