Let's look at it this way; an archer can, as part of their attack action do the following, while maintaining a grip on their two-handed bow with one hand:
- Reach to their quiver
- Draw an arrow Align the nock with the string
- Put the nock on the string
- Change their grip on the arrow/string
- Draw the string back Release the string
Here we have gripping and ungripping of a weapon with one hand, while manipulating an object with that hand, and it is completely incidental to the use of said weapon. At higher levels, the archer can repeat this process multiple times during the same action.
To argue that a wizard can't:
- Release their versatile staff with one hand,
- Make a gesture with their free hand,
- Return their hand to their staff
Here we have the gripping and ungripping of a weapon with one hand, without manipulating an object with that hand, but the argument is being made that it takes some form of action to do this?
That's a pretty hard argument to make. Both are grabbing/releasing the weapon with one hand the same number of times. The archer additionally has to manipulate an object with their other hand. (That's fine motor control in a high-stress, combat situation.)
If the archer can manage it, the wizard certainly can.
Additionally, interpreting the requirements of a Somatic component that way would make it impossible for a wizard to cast a spell with both Somatic and Material components without spending the prior round 'drawing' the spell component(s) and freeing a hand to make the gesture.