While spell attacks are affected by the unseen attackers rule (PHB 194) as any other attack, saving throws follow different rules. In the general rules for saving throws we read that:
A saving throw can be modified by a situational bonus or penalty and
can be affected by advantage and disadvantage, as determined by the DM. (PHB 179)
But while it is left up to the DM, there is also the arcane trickster feature magical ambush:
if you are hidden from a creature when you cast a spell on it, the
creature has disadvantage on any saving throw it makes against the
spell this turn. (PHB 98)
It is strongly implied that this is a special rule overwriting the general case. Granting this to all characters (and NPC-s) would make this feature redundant.
There are not many features that grant disadvantage on saving throws. The above rogue feature is acquired on level 9, and the sorcerer metamagic that does it is generally the most the expensive one. It is clear that such a thing should be regarded as really powerful. Consider that traps also grant a saving throw as normal, even though they need no components to cast and are usually hidden.
Thus I advise against ruling that being hidden should grant disadvantage on saving throws.