In my experience the sentry mode alerts you by becoming an idle ship that requires an order. I am assuming that you are referring to your ships stationed nearby the planet to be invaded. To wake up they must be quite close, warships usually don't have a lot of sensor range.
If tehy do wake up they will not do a thing, but you have an opportunity to give them an order to attack any transport or fleet that is threatening your planet. If you are talking about ships beyond range they will do nothing. It doesn't and shouldn't work that way. If you are at war the only reasonable defense is to station ships in orbit around your planets and bases.