Animate Undead only creates two types of creatures, zombies and skeletons, neither of which are very smart (INT of 3 and 6 respectively), leaving them with few resources to figure out if a creature is hostile without that creature taking an obviously hostile action.
I don't know that comparing them to beasts of equivalent intelligence is really relevant either. The two creature types are fundamentally different, and would have different reactions to many different scenarios. Even different types of beasts that could be expected to be nearly identical could have wildly different reactions to creatures approaching them. Some beasts (like say, a wild wolf) could be incredibly territorial, and assume that anything that dares to enter their territory is hostile, while others (a domesticated dog) may assume that any creatures they come across are friends, and attempt to play with them, rather than fight.
Unless a creature does something immediately harmful, such as attacking them with a weapon or spell, the undead aren't likely to realize that the creature is hostile or even unfriendly.
So in this case, a "hostile creature" is one that directly attempts to harm the undead, or takes some other obviously hostile action.
How do these undead "defend themselves" against hostile actions? Well, typically with fighting back. To defend oneself is not to simply attempt to avoid harm, but also to attempt to prevent anyone who is attempting to inflict harm from continuing to do so. Think about this: self-defense laws are laws in place that can exclude you from punishment under the law for attacking another person, as long as you were attacking that person to defend yourself from their attacks. Zombies generally aren't known for their self-preservation instincts, so any attempt to defend itself would likely entail fighting the hostile creature until either it dies, the enemy dies, or the enemy runs away.
If one of the undead are attacked, they will fight back.