Hostile is how enemies perceive you, Enemy is how you perceive others
When the rules use the word hostile, it comes from the frame of reference of a creature that is not you.
For the next minute, you know the direction of the nearest creature hostile to you within 60 feet
For the duration, you have advantage on all Charisma checks directed at one creature of your choice that isn’t hostile toward you
The creature is neither friendly nor hostile, and it isn’t under your control
Page 244 of the DMG.
A friendly creature wants to help the adventurers and wishes for them to succeed.
An indifferent creature might help or hinder the party, depending on what the creature sees as most beneficial.
A hostile creature opposes the adventurers and their goals but doesn't necessarily attack them on sight.
Now, notice you have no control over the hostility of other creatures.That's what hostility is. Other creatures attitude defines whether they are friendly, hostile, or neutral toward you.
So, hostile refers to attitudes towards you, enemy refers to really anyone your DM allows you to consider your enemy. Its not so simple as to claim any creature is your enemy, but it is something defined by your character and usually entails combat, whereas hostility can work outside of combat.
