Under the Vasari Tech tree there is an ability called 'Returning Armada'. I tried using the ability and I didnt really see it do anything. Could someone please explain what this actually does?
Returning Armada is an added ability to your phase gates. Basically, once they fill up on antimatter, they'll spawn a few random ships that will phase jump to the gravity well of the phase gate. This shouldn't bring you over the fleet cap, but it's free ships for every phase gate you have

Also, what exactly are Artifacts and when do you get them? In all of the solo games I have played have been on reality small systems with 12 or less planets. I have explored all of my planets til I had no slots left and I controlled the entire system but I have yet to find an artifact.
By default, artifacts are set to appear on 15% of planets I believe. You get them by doing the exploration upgrades. Since they're random, you're not guaranteed too much on the smaller maps, but on large maps there are quite a few artifacts floating around. As to what they are is basically permanent boosts to your empire. One artifact increases all ship damage, another cuts research costs by 25%, another increases your antimatter regen rate by 10%, etc