Oldschool D&D has rules for morale, such as these from Labyrinth Lord:
Monsters have a listing for morale, which represents how likely they are to fight or flee when in an aencounter. Morale is rated from 2-12, and while a score of 2 indicates that the monster never fights (unless absolutely cornered), a score of 12 indicates that the monster will fight until killed, with no morale roll necessary in either case.
The Labyrinth Lord usually makes a morale check under two conditions, when one side of an encounter has lost a member due to death, or when half the group on one side is either killed or otherwise incapacitated. The Labyrinth Lord will roll 2d6, and if the roll is higher than the morale of the monster, the check is a failure and the monster will either attempt a full retreat or a fighting retreat. If the result is equal to or lower than the morale score, a monster will continue to fight. If this roll is made successfully two times in one encounter, the monster will fight until killed.
Is there an equivalent variant rule in D&D 3e / D20?