Rules are pretty unambiguous about this. There is no "hitting the armor" or "hitting the shield" concept in D&D 5th edition rules. If there is no need to determine the amount of damage done, there is no hit. If attack does not hit, ie. the attack roll result is less than the armor class, there is no hit, and effects which depend on hitting do not trigger.
As a side note, what happens if attack roll is enough for a hit, but then damage is determined to be 0... now that's a different can of worms, and not directly relevant to the core of this question.
What your player is doing, is trying to argue for an advantage based on combining the "rules" and "common sense" in a way that is beneficial to them. It's a very human thing to want to do. There is this very cool feature, and a player has an image of it working in a very cool way.
In general, your unfortunate job as a DM is to say no to this kind of attempts to gain unearned power. Spells and classes are designed and play-tested for a degree of balance. Breaking the balance will just lead to an unhappy table.
And in this particular case... Armor of Agathys is already a very powerful combat spell: It can be cast before battle because of its long duration. In the right circumstances it can deal massive amounts of total damage without using up action or reaction during the battle, and even in the worst case (being hit only by ranged attacks) it provides good temp HP. And it does not need concentration! It is indeed quite essential for balance, that it can only trigger when taking damage in return and having the temp HP get used up.
So, what to do with your situation:
- Your job as a DM is to come up with a good enough explanaton, "fluff" or "flavor", so the spell makes magical sense. In this case it is easy, already in the mechanical meaning of the rules: the caster taking damage is what causes the spell to slash back with cold damage. Hammering at the armor in a way that does no damage to the caster does not trigger the spell. It just doesn't, that's how the spell's magic works.
- If the player decides they still are unhappy with the spell, and you want to make a point, allow them to swap the spell for False Life... It is a lot like Armor of Agathys, except it gives tiny bit less HP on average, and does not deal damage. That should drive home the realization how good a spell AoA really is even without the buff they are asking.