There is no official description of "harmful" so let's break down the condition. The complete Charmed condition is:
- A charmed creature can't attack the charmer or target the charmer with harmful abilities or magical effects
- The charmer has advantage on any ability check to interact socially with the creature.
I would take this condition to mean that the charmed creature is favorably disposed towards the charmer and would take no action to cause the charmer to come to harm. Like a family member (well, usually). The Charmed condition is a stronger version of the Friends spell, so the target will regard the charmer as a really good friend. You don't make bad things happen to a friend. Also note that the Friends spell results in bad feelings after it's over and the charmed person learns they were charmed. It makes people mad, presumably because people feel like they have been harmed.
I would separate "abilities or magical effects" into three categories:
Good for the target
Obvious buffs or ability gains to the target. Fly, Guidance, Healing Word, Polymorph but only if used with intent to help, such as how it might be used on a party member, to give them hit points or abilities. These would be okay to target the charmer with.
Neutral to the Target
Utility spells that normally wouldn't be used in combat, like Telepathy or Rary's Telepathic Bond. I'm having trouble coming up with more, but these or similar would be fine to target the charmer.
Bad for the target
I put in this anything that would be used against the target or anything that would hurt the target in combat or disadvantage the target in interactions. Fairie Fire, Magic Missile, Friends, Polymorph if used with a target save, Bane, etc..
I would not allow rules lawyering to include AoE spells like Fireball, either. You wouldn't throw a Fireball at your children.
It doesn't seem hard to know as a DM what the intent of a spell is. If it is bad for and is targeted at the charmer, it would not be allowed. This would not prevent spells that are target at the charmed creature itself, like Armor of Agathys or Invisibilty, even if useful in combat against the charmer.