You will need the free use of a hand without the focus
This was elaborated on in the Rules of spellcasting sage advice column.
Another example: a cleric’s holy symbol is emblazoned on her shield. She likes to wade into melee combat with a mace in one hand and a shield in the other. She uses the holy symbol as her spellcasting focus, so she needs to have the shield in hand when she casts a cleric spell that has a material component. If the spell, such as aid, also has a somatic component, she can perform that component with the shield hand and keep holding the mace in the other.
If the same cleric casts cure wounds, she needs to put the mace or the shield away, because that spell doesn’t have a material component but does have a somatic component. She’s going to need a free hand to make the spell’s gestures.
If we extrapolate the above example to include any spellcasting focus, which we are allowed to do since there is no mention anywhere of a holy symbol being an exception to any general somatic component rule, then any spell with a somatic component and no material component requires a hand that isn't holding anything.
Why should a spell with fewer components be harder to cast?
The in-universe reason why you can't use the focus for non-material spells is, in some way, setting dependent. A DM is well within their purview to ignore this rule entirely, or justify it in whatever way they see fit. One way that you might justify this rule, and the way I do it in my own games, is by saying that material components focus the magical energies used to create the spell to some extent that it makes it easier to cast the somatic components. Basically, because of the extra focusing power of the material components, your gestures don't need to be as exact to get the job done and cast the spell.
Without material components, your somatic (and potentially verbal) components don't get that extra boost, so they need to be absolutely precise for the spell to cast properly. As a result, without special training à la Warcaster feat, your hands need to be completely free to assure a proper cast.