Sentry puts the ship to sleep. It wakes up when a foreign ship enters its sensor range.
Guard puts the ship to sleep. It stays there until you wake it. If someone attacks, it (naturally) defends itself.
Sentry is semi-useful. Guard less so. Have used guard to "park" ships so that I'm not bugged every turn to move them. This has the danger that you forget they exist...
My ideal implementation for "sentry" and "guard" would have been:
Sentry: Sleep until enemy ship comes into sensor range. Activated sentry ships are the first to move.
Guard: If a foreign ship enters the movement range of a guarding ship, it must attack it. With the implication that if the foreign ship's faction does not want to go to war with the guarding ship's faction, it can't enter the guard's movement range. (Obviously the AI must also be made to use the guard feature against you...)