Planets that run with automatic planetary governors build everything faster and provide massive bonuses.  

Planets run without planetary governors allow you to control things, but you don't get the bonuses.  

The amount of control/bonuses you get depend on the civilization and the amount of resources you're willing to spend on propaganda/press and listening to the whims of the planetary governors ("useless" pork barrel projects that drain your Treasury, for example).

Tearing down buildings built by the planetary governors pisses them off and makes them less likely to listen to you in the future.  Also, it makes everyone on the planet depressed, leading to riots and other economic problems.  

For example: You didn't like those useless stadiums built by the planetary governor because they serve no discernible purpose? Well that's just too bad.  Tear them down and you get a planetary riot.  

For this to work, you'll have to eliminate or severely nerf adjacency bonuses because otherwise a human would still super-specialize their worlds despite the handicaps associated with it.