Yes, it is valid to negate a spell's, such as Mirror Image, benefits by making it obvious what the result of the spell is.
Spells in D&D 5e do exactly what they say they do. There is no 'just flavor text' in 5th Edition. They do no less, but also no more than the words as described. This does not, however, mean that spells are always effective. Mirror Images provides some examples of how to overcome itself, both magically and mundanely.
Mechanically, Mirror Images does not provide a comprehensive list of caveats for methods that would identify you. I'm not saying it should be defeated by something like throwing a sack of flour over the spellcaster's square (the images likely conform to however you look at that moment); however, another creature being affected by the spell is far beyond its power and there is little other way to explain what a target sees when you successfully Grapple one of the images.
Notably, the description of Mirror Image's function is
it’s impossible to track which image is real.
You do not have to 'track' which image is real if you're simply attacking whatever your ally is holding. This is closer to "[relying] on senses other than sight" or "[perceiving the] illusions as false".
There are other instances of spells, especially illusion spells, being overcome and their effects no longer being effective (even if the spell is still active). While it is initially strange that the second paragraph of the spell is negated, that is the result.
Additionally, successfully grappling a target affected by Mirror Images deserves some form of reward, and being able to show your party "look, this is the Wizard" until they remove themselves from your grasp seems appropriate to me.