From the Player's Basic Rules, page 76:
If you have temporary hit points and receive more of them, you decide whether to keep the ones you have or to gain the new ones.
So in this situation you can choose to have 5 temporary hit points or 10 - it's up to you.
Armor of Agathys specifies that its effect of damaging creatures that attack you happens 'while you have these hit points'. If you replace the hit points it gave you with different hit points, you don't 'have these hit points' any more, you have some other set of hit points, so you wouldn't get the damage effect.
Cases like this are presumably why you get to choose between temporary hit points instead of automatically getting the larger amount.