The DM typically determines if a creature's routed
In a core-only Dungeons and Dragons 3.5, campaign there are no morale rules. Creatures fight to the death if the DM thinks they would and opt to flee or surrender if the DM thinks they would. This can be problematic for the DM and players alike. For example, a beginning DM may not realize exactly how fragile beginning combatants are, and, subsequently, PCs may grow to think they must behave more like battlefield executioners rather than heroic adventurers, and should root out and murder every kobold baby lest it, too, see the PCs and fight to the death.
So, instead of a strict, codified morale system, the DM's expected to have creatures behave in a believable fashion that fits current events and circumstances. For example, a group of Huge fiendish sharks (each Int 3, by the way) tasked by a water druid to guard his undersea lair may not be open, initially, to negotiation with the PCs, but, after half their number are slain—some spectacularly1—, those same sharks may change their minds and negotiate their surrender or flee.
However, some published monsters and adventures do provide guidelines for the DM detailing when a creature flees or surrenders, but these are, comparatively, rare rather than the norm.
Morale from other sources
The Dungeons and Dragons 3.5 supplement Heroes of Battle includes extensive rules for morale, but urges the DM to steer away from it in the typical dungeon-based adventure:
You can also take these [morale] rules beyond the battlefield if you wish, using them in dungeon-based adventures, urban settings, or elsewhere. In most cases, using the morale rules means that monsters run away more often than they do in a typical D&D adventure. That situation is common in organized warfare, but rare in a dungeon because the monsters are found in their lair and have nowhere else to go. In contrast, a smart commander encourages her units to retreat (or at least make a strategic withdrawal) when they’re overmatched or needed elsewhere. If you use these morale rules in a traditional D&D adventure, make sure you account for the greater likelihood of NPCs retreating. (72)
In other words, it kind of sounds like the authors of Heroes of Battle didn't often expect their dungeon-delving PCs to give any quarter nor their NPCs to ask for any.
Likewise, Paizo's SRD includes morale as part of its mass combat system, but that, too, is largely unsuited to the traditional dungeon.
Undoubtedly, there are many third-party sources for morale rules. For example, this 2011 EN World thread suggests adapting these homebrew Dungeons and Dragons, Fourth Edition morale rules to Dungeons and Dragons 3.5. The original poster of that thread appears to have been convinced of the playability of those rules.
1 The PC disguised as a local approached the sharks and began a conversation while studying it. When the sharks smelled through his disguise, initiative was rolled and the PC went first and made a successful unarmed strike that was also a death attack against the shark to whom the PC was speaking. The shark failed its Fort save and died. We named the dead shark Bob.