I think your intuition is right: if there are more seekers, there are more chances to find a hider. And if there are more hiders then there are also more chances to find a hider.
The checks should reflect that fact.
Normally I would say that each hider succeeds or fails on their own merits, meaning an individual Stealth check. That is how it would work in real life: in the children's game, hide and seek, the seekers find the hiders one at a time, not all together as a group. Back in the world of D&D, if the seekers don't know there are any hiders but then discover one, they would be alerted to the possibility of further hiders; so they should now get advantage or a small numerical bonus to their Perception checks (I favour +2).
I would make one exception to this ruling: if a party are hiding in situ (ie they are not moving) and they have been doing so for at least a minute, I would allow a group Stealth check as the stealthier members help the less stealthy to be inconspicuous. So long as they stay still, the less stealthy members have much less chance of mucking things up for the party.
Seekers can benefit from the most advantageous kind of check: a teamwork Perception check, which gives a bonus to the check of the creature with the best modifier. But this doesn't come cheap in combat. It requires the use of the Search action by the lead creature and the Help action by one other creature. That's only two seekers, so what are the rest of them doing? I'd be inclined to give a +1 bonus for each additional seeker - but to what limit I don't know!
When the seekers are relying on passive Perception I'd use the passive Perception of the lead creature and be done with it. The +5 from advantage granted by a single helper is way too strong. I could see an argument for a +1 bonus for each additional seeker as suggested in my previous paragraph.
The DM may also want to think about what different seekers can perceive based on their senses and their position in the environment (eg the hiders may have obscurement and cover from some seekers, but only obscurement from others).