The warlock does not take damage to his hp, but rather to the new temporary HP.
Armour of Agathys has the following text (this is the whole text of the ability, bar the 'at higher levels' note)
A protective magical force surrounds you, manifesting as a spectral frost that covers you and your gear. You gain 5 temporary hit points for the duration. If a creature hits you with a melee attack while you have these hit points, the creature takes 5 cold damage.
Emphasis mine. 5E does not have explicit order of operations, and there are several notes that if there is dispute about how an ability works the DM simply decides. However, in general, an attack is rolled and compared to AC before damage is rolled. An attack 'hits' and then you roll and apply damage. Armour of Agathys specifies hits, and does 5 cold damage to them when the attack hits. Not when it does damage. Not after the attack. If it hits.
With Heavy Armour Master or other abilities, it's entirely possible for an attack to hit and do zero points of damage. Some attacks don't do regular damage but instead have other effects. A character may be immune to the kind of damage an attack does (such as Fire, or Necrotic). Armour of Agathys would trigger on any of those attacks, before damage, and any effects triggered by that (such as Dark One's Blessing) would naturally happen when Armour of Agathys happens.
Ergo, in our example, the Warlock would take all 5 damage from his new temporary hp (or some from his new temporary hp, and others from his regular hp total). The Dark One's Blessing temporary HP would replace the armour of agathys temporary hp before damage is rolled.
It would be something like this;
The enemy attacks, and they hit.
Having been hit by an attack while having Armour of Agathys temporary hp, the spell deals 5 cold damage to the attacker. They die. The warlock, being a Fiend Pact warlock, gains Cha mod + level temporary hp from having killed someone. Having gained new temporary hp that is greater than their existing hp, they replace that temporary hp with the new, higher total (temp hp does not stack).
They receive damage from the attack.