It Could be Countered Sometimes, But it has to Actually be Detected
XGtE p. 85 (as previously quoted by BlueMoon93):
To be perceptible, the casting of a spell must involve a verbal, somatic, or material component.
Hence one or more of these components is a necessary precondition to perceiving the casting (to the extent we enforce materials in Xanathar's Guide to Everything). An arcane focus of any kind may make it possible to perceive. However it is not clear that all arcane foci under all circumstances would actually be perceived, simply that they are one of the things that make it possible to perceive.
For counterspell specifically, we see:
Casting Time: 1 reaction, which you take when you see a creature within 60 feet of you casting a spell.
It is not sufficient simply that the spell be perceptible in some way; it must actually be perceived and be perceived via the sense of sight specifically to be a valid target for counterspell.
This leads us to your "small arcane focus". If the component can be hidden from view then while in a state of being hidden from the counterspell caster they may or may not see it or the spell. The onus is on them to do so, the mere fact that an arcane focus of some sort is involved does not make them see it, it merely makes it possible.
It should require a perception roll by the counterspell caster whenever no components are being used in an obvious manner, and is only possible where visual perception is possible (eg: no greater invisibility). Counterspelling when the focus is small and easily concealed should require a perception roll, perhaps contesting a slight of hand check.
If any spell with material components was automatically perceptible and automatically visually perceptible, then one could counterspell through walls, while blind, or against invisible opponents. Components are a necessary condition, not a sufficient one.
If it was not possible to discreetly cast a spell with material components under the table, from time to time, then the question arises: Can I do anything with minor illusion whatsoever?