I had a similar experience while trying to play the Iridians.  In several different scenarios I was wiped out by both the Altarans and the Earthlings, due to influence.  Despite building several influence based starbases as well as a bunch of influence buildings.    In one instance I had 6 planets in a unified sphere and the Altairans came in and colonized a level 5 planet next to 2 of my level 14 and 15 planets and within about 30-40 turns had turned both of those planets.   Despite the fact that I had built 2 culture starbases and loaded the planets with Free Markets and Iridium stores.   

I submitted a ticket with 2 game files both of which have examples of culture not necessarily working to the optimum. 

@Raynman, I've noticed that after working up 4 or so levels in the Benevolence/Influence track influence changes its behavior entirely and seems to work as expected.  I lost my save game where I observed this happening.  I'd look at the code around that for some issues.  Some of the levels on that track actually seem to cause influence penalties.  I lost a planet in a game after leveling up, but I can't remember which step.  That was unexpected.

I submitted ticket SWH-548-35275 on this stuff. A tiny, tiny pocket of unconnected influence in my vast sea of dominance is about to capture one of my planets.

When will the influence bug get fixed?

If they didn't fix it in .71, you will probably have to wait until Beta 5. They are much more interested in getting the next iteration out than fixing and polishing the current one.