There is no specific mechanic for this, but...
As Jon Aristotle says, there is no specific mechanic for attempting to hinder the recognition of spell identity. However, what you are describing fits within the general rule of the GM assessing whether environmental "circumstances" merit rolling an ability check with advantage or disadvantage (PHB 173). Thus, the rule that applies is the GM deciding whether the caster's attempts to obfuscate their spell identity are significant enough to merit disadvantage on the roll of the creature attempting to identify the spell.
A few considerations based on the action economy
What you don't want to have is a caster being permitted to impose disadvantage on spell identification automatically or 'for free'. If it was that easy to do, you would assume that all casters did it all the time. Doing something confounding enough to impose disadvantage must be exceptional, and that means it must either cost something, or be extremely situational, or both. In terms of the cost, it bears mentioning that the creature attempting identification has to spend its reaction to do so. Frustrating their effort should have at least an equivalent cost.
However, the caster is already spending their action on casting the spell. If they are going to cast the spell and in addition disguise it, they must be able to spend something in addition to their action (and see "Other Activity on Your Turn", PHB 190). Possibilities (in my opinion) could include:
Their reaction. "As you begin your spell, you notice that the enemy mage is paying close attention to your gestures." "I would like to use my reaction to turn away from them provided I can still see the spell's target." "Okay, you have obscured your gestures and it is harder for them to hear your voice - they will have disadvantage on their attempt to recognize the spell as you cast it although someone in another position will not be affected."
Their object interaction. In addition to their spell-casting action, the caster might be doing other things with their hands (cf). However, if they aren't, for example if they are casting one of the sixty-some spells that have verbal components only, they could be allowed to spend their object interaction making meaningless gestures so as to throw someone off.
The "brief utterances" they can make on their turn. Especially if they are casting one of the twenty-some spells that don't have a Verbal component, they could fill their turn with nonsense words or even words from other, actual spells so as to confuse the issue.