Redviper, do you build any labs on your specialized manufacturing worlds?
usually, yes, if I have room for them, 1 or 2 of them per planet, more for the research worlds. Eventually, as my empire size increses, I will remove labs from manufacturing worlds (where I have bonus) and put more labs on research world (less industries there).
Then, once I have research all tech, I remove the labs and put something else in their place, like industry to planets attached to shipyards, economic buildings where needed, more farms&morals buildings when I can.