In AD&D 2nd edition there were morale checks to determine whether an NPC or monster continues to fight, or tries to surrender or flee. In core D&D 3.5 there is nothing of that sort (if I've missed nothing).
Are there any rules to determine whether monsters fight or flee in any other source related to 3.0, 3.5 or Pathfinder?
In a core-only Dungeons and Dragons 3.5, campaign there are no morale rules. Creatures fight to the death if the DM thinks they would and opt to flee or surrender if the DM thinks they would. This can be problematic for the DM and players alike. For example, a beginning DM may not realize exactly how fragile beginning combatants are, and, subsequently, PCs may grow to think they must behave more like battlefield executioners rather than heroic adventurers, and should root out and murder every kobold baby lest it, too, see the PCs and fight to the death.
So, instead of a strict, codified morale system, the DM's expected to have creatures behave in a believable fashion that fits current events and circumstances. For example, a group of Huge fiendish sharks (each Int 3, by the way) tasked by a water druid to guard his undersea lair may not be open, initially, to negotiation with the PCs, but, after half their number are slain—some spectacularly1—, those same sharks may change their minds and negotiate their surrender or flee.
However, some published monsters and adventures do provide guidelines for the DM detailing when a creature flees or surrenders, but these are, comparatively, rare rather than the norm.
The Dungeons and Dragons 3.5 supplement Heroes of Battle includes extensive rules for morale, but urges the DM to steer away from it in the typical dungeon-based adventure:
You can also take these [morale] rules beyond the battlefield if you wish, using them in dungeon-based adventures, urban settings, or elsewhere. In most cases, using the morale rules means that monsters run away more often than they do in a typical D&D adventure. That situation is common in organized warfare, but rare in a dungeon because the monsters are found in their lair and have nowhere else to go. In contrast, a smart commander encourages her units to retreat (or at least make a strategic withdrawal) when they’re overmatched or needed elsewhere. If you use these morale rules in a traditional D&D adventure, make sure you account for the greater likelihood of NPCs retreating. (72)
In other words, it kind of sounds like the authors of Heroes of Battle didn't often expect their dungeon-delving PCs to give any quarter nor their NPCs to ask for any.
Likewise, Paizo's SRD includes morale as part of its mass combat system, but that, too, is largely unsuited to the traditional dungeon.
Undoubtedly, there are many third-party sources for morale rules. For example, this 2011 EN World thread suggests adapting these homebrew Dungeons and Dragons, Fourth Edition morale rules to Dungeons and Dragons 3.5. The original poster of that thread appears to have been convinced of the playability of those rules.
1 The PC disguised as a local approached the sharks and began a conversation while studying it. When the sharks smelled through his disguise, initiative was rolled and the PC went first and made a successful unarmed strike that was also a death attack against the shark to whom the PC was speaking. The shark failed its Fort save and died. We named the dead shark Bob.
No, from 3e onward there are not morale rules in the core books. Everyone shouts "DM decides" when this comes up, but I like to decide what I want to decide and what I want to randomize. (You may as well have the same fiat-vs-roll argument about everything from random encounters to hit rolls. To me it's easier to have a way to randomize it and then overrule it if I want to fiat, that way both use cases are covered.)
I ported a two-part morale system forward into 3.5e/Pathfinder for my own use as a house rule, published on my blog Geek Related - the basic idea is two d20 values, one for how likely the foe is to attack in the first place, and one for how likely they are to press the attack once wounded/pressed (people and animals vary widely on both those axes). So:
First value, roll d20 over to see if they attack, second value, roll d20 over to see if they keep attacking. And you get a bunch of more interesting behaviors quickly defined…
Apply modifiers for the usual status effects - being shaken, being taunted, all these usually have clear +-2-4 values that would apply on morale rolls. Tack on -2's for whatever's relevant (wounded, leader dead, more than half of group dead...)
This keeps it simple and doesn't use Will saves or whatever because it's designed for you to tweak/override. If you want to default the second number "10 + Will save" or whatever that's your right!
The pathfinder mass combat system has rules for morale including when armies route/disband and the pathfinder performance combat system has morale rules as well you could try adapting either of these to suit your purposes (presumably you are looking to apply this to regular combats).
A number of adventure paths will tell you how NPCs should behave (when they try to flee, if they fight to the death, etc.) If you are using an AP you can follow its guidance (if it's your own campaign you could set these numbers yourself before combat if you like).
There is also this unofficial system created for 4e that seems reasonable if you want to use it (for reference bloodied is less than half hp)
