According to the Counterspell text, the Casting Time is "1 reaction,...
...which you take when you see a creature within 60 feet of you casting a spell.
In 5e, because of the importance of the action economy, reactions are very strict, and are only allowed under very specific circumstances, as written. In this case, you must see a creature that is casting a spell.
There are several indications in the rule books that spellcasting in and of itself does not present any visible signs in the absence of verbal and somatic components. The sorceror metamagic ability Subtle Spell (PHB p.102) says:
When you cast a spell, you can spend 1 sorcery point to cast it without any somatic or verbal components.
If spellcasting activity was innately identifiable despite the absence of these components, then this particular ability wouldn't be very subtle, would it?
Also, in the Spellcasting chapter of the PHB (p. 204) under Targets, we have:
Unless a spell has a perceptible effect, a creature might not know it was targeted by a spell at all. An effect like crackling lightning is obvious, but a more subtle effect, such as an attempt to read a creature's thoughts, typically goes unnoticed, unless a spell says otherwise.
Lastly, according to the Sage Advice Compendium,
If a spell that’s altered by Subtle Spell has no material component, then it’s impossible for anyone to perceive the spell being cast. So, since you can’t see the casting, counterspell is of no use.
So, you must perceive the spellcasting activity. However, since the visible components of spellcasting appear to only be the somatic components themselves (barring any visible effects like a fireball), there is no indication I have found in the rules that there would be any way to distinguish between the caster and someone who cannot cast the spell making the same noises and hand gestures.
By this argument, the OP's question makes sense. It is possible to see someone casting a spell, but not know whether they are actually casting or just acting. That means the real question is
...do I have to aim for one of them specifically?
The spell doesn't use the word "target". However, the first line of the spell says:
You attempt to interrupt a creature in the process of casting a spell.
Since the spell only affects 1 spell being cast, and not an area, it seems straightforward that you would have to specify the creature you intend to interrupt.
So yes, you have to choose.
Now, having said that, if the other creatures aren't casting the same spell but are just acting then, as a DM, I would have them make a deception check to get the moves right to fool you. And that would be easier if you didn't know anything about spellcasting, so the obvious opposing skill roll I would ask for from you would be an Arcana check. If you don't have that skill, I might allow Perception to figure out if one or more of these fellows seem more adept than the others in their movements. Depending on the relative die rolls, you might eliminate some or all of the imposters from contention and improve your odds.