The charge is expended.
Invalid Spell Targets (XGtE p. 86)
... If you cast a spell on someone or something that can't
be affected by the spell, nothing happens to that target,
but if you used a spell slot to cast the spell, the slot is
still expended. ...
I think it is safe to reason that the reason you are losing the spell slot for casting on an invalid target is that you completed your Cast a Spell action.
Charges (DMG p141)
Some magic items have charges that must be expended
to activate their properties. The number of charges an
item has remaining is revealed when an identify spell
is cast on it, as well as when a creature attunes to it.
Additionally, when an item regains charges, the creature
attuned to it learns how many charges it regained.
I think that you can make 2 slightly different arguments for why the wand loses charges.
- You must spend the charge(s) to activate the wand to even begin casting its spell. This, if the spell were interrupted in a way that would not normally cost you a spell slot, you would still lose the charge.
- Spending the charge, activating the wand, and casting its spell are all part of the same action. You successfully cast the spell, in so much as you released the energy in your attempt, but it somehow did nothing. Spell cast == charge spent.
I do not know of any RAW that explicitly answers your specific question, but I think it can be answered through straightforward logic of the rules we do have.
I can also provide an argument against allowing the charge to be unspent.
Imagine you have a Wand of Hurt Shapeshifters. If you didn't have to spend any charges to activate the wand against invalid targets, then the wand is also a Wand of Detect Shapeshifters which automatically uses up a charge if and only if it is used against a Shapeshifter to hurt it. Now you can go around and use the wand on everyone, and it'll only go off when you find a Shapeshifter. I don't think this would make any sense.