Or do I have to target the caster of Mirror Image, who then decides which missiles are redirected onto the illusions?
You create three magic darts. Each dart hits a visible creature of your choice.
So there is no "trying" to target the caster, you simply target the caster and automatically hit them, as that is the creature you are targetting. (Illusions are not creatures.)
After that, Mirror Image actually does nothing, as per Does Mirror Image affect spells that have no attack roll?
So no, you can't use Magic Missile to get rid of Mirror Image duplicates.
Magic missile hits the caster
This has been discussed here, but the net reason is that magic missile isn't an attack (there is no roll.) Without an attack roll, there is no D20 to determine what you hit - you simply hit the target.
This is also supported by Jeremy Crawford:
The mirror image spell has no effect on magic missile, which doesn't involve an attack.
Can you even target an illusion?
Magic missile requires (emphasis mine):
Each dart hits a creature of your choice
The following is from Xanathar's Chapter 2 under Spellcasting:
If you cast a spell on someone or something that can’t be affected by the spell, nothing happens to that target, but if you used a spell slot to cast the spell, the slot is still expended. If the spell normally has no effect on a target that succeeds on a saving throw, the invalid target appears to have succeeded on its saving throw, even though it didn’t attempt one (giving no hint that the creature is in fact an invalid target). Otherwise, you perceive that the spell did nothing to the target.