ONLY the Core World that built it contributes to it. Everything else gets funneled through the Core World before it gets sent to the Shipyard. Which means only those Colony Worlds and Asteroid Mining Bases that send their output to that Core World matter. 

Anything that goes to a different Core World will be sent to a different Shipyard, assuming that it has a Shipyard.

So do not make those really small size Colony Worlds into Core Worlds. It varies by the Player and what is available on the Map, but I have seen other Players say not to turn Colonies smaller than 6 to 8 size into Core Worlds. At least early in the game. Many of those smaller worlds will not have many tiles to terraform (IF ANY), once you get those Techs.

A CORE World IS a World that you have assigned a Governor (Leader) to. You could make a size 1 Colony World into a CORE World if you wanted, but it would normally/usually a very bad/worthless idea. You might want to Core a fairly small World if it had a special resource available that you wanted to develop, but usually they do not any to develop.

A Core World is one where you can give orders to build Districts or Special Buildings on the Tiles, including Terraforming any Tiles. However, really small size Worlds often have no or very few (1 or 2) tiles that be Terraformed even with the high level Techs. I do not know if there might be some way to do something with those really small Worlds, with some really High level Techs/Events/Ideologies/etc. Maybe, but only develop them (turn them into Core Worlds) if you are sure it can be done.

Where you run into problems, is when you have Two (or more) Worlds that so good that you have to make Cores out of them, but they are in the same star system or nearby star systems, say 2 Precursor worlds of 15 and 18. Worlds like that need to be developed, which means they have to become Cores, in order to build on their surfaces.

The problem is that they will split the local colonies sending resources to them into two parts, which leads to smaller available production values for each of them, which makes it harder to build stuff at them. Particularly, they will have 2 shipyards building independent of each other, instead of one ship yard getting production from both like in GC 3.