Do you know which ships jumped and which stayed? There are two things that could've happened:

1)A few ships already initiated jump by the time you gave the attack order. They can't cancel the jump powerup, so they end up jumping anyway and then flying back.

2)There was an issue (I'm note sure if it was resolved yet) with support cruisers. The way fleets work, if you give an attack order with a mix of combat/support ships selected, only the combat ships will execute attack. This is done so that support ships focus on their primary support roles (repair, disable, whatever) instead of trying to tickle enemies to death. But if you give a jump order, then attack, the combat ships will go attack and the support ships will keep jumping. I reported this one, but I'm not sure if Ironclad had time to fix it yet.

For now, to avoid the issue in 2) if it hasn't been fixed, when you have a large mixed fleet and you're giving a direct attack order - give a move order first, that way if they were supposed to jump they wouldn't, and then you can give your attack order.