From Dispel Magic (emphasis mine)
Choose one creature, object, or magical effect within range. Any spell of 3rd level or lower on the target ends. For each spell of 4th level or higher on the target, make an ability check using your spellcasting ability. The DC equals 10 + the spell's level. On a successful check, the spell ends.
At Higher Levels. When you cast this spell using a spell slot of 4th level or higher, you automatically end the effects of a spell on the target if the spell's level is equal to or less than the level of the spell slot you used.
From the Sage Advice Compendium (all emphasis mine)
Can you use dispel magic on the creations of a spell like
animate dead or affect those creations with antimagic
Whenever you wonder whether a spell’s effects can
be dispelled or suspended, you need to answer one question: is the spell’s duration instantaneous? If the answer is
yes, there is nothing to dispel or suspend. Here’s why: the
effects of an instantaneous spell are brought into being by
magic, but the effects aren’t sustained by magic (see PH,
203). The magic flares for a split second and then vanishes.
In contrast, a spell like conjure woodland beings has a
non-instantaneous duration, which means its creations can
be ended by dispel magic and they temporarily disappear
within an antimagic field.
Although that doesn't really clarify if it dismisses one summon or all of them. However, I would argue that they all end because Dispel Magic is a spell to remove magic, and the sage advice even reinforces that Dispel Magic is meant to break spells, not a single effect from a spell.
Can you use dispel magic to dispel a magical effect like
a vampire’s Charm ability?
Dispel magic has a particular
purpose: to break other spells. It has no effect on a vampire’s Charm ability or any other magical effect that isn’t
a spell. It also does nothing to the properties of a magic
item. It can, however, end a spell cast from a magic item
or from another source. Spells—they’re what dispel magic
is about. For example, if you cast dispel magic on a staff of
power, the spell fails to disrupt the staff’s magical properties, but if the staff’s wielder casts hold monster from the
staff, dispel magic can end that spell if cast on the target of
However later they explicitly answer that Dispel Magic does not end the entire spell.
If dispel magic targets the magical effect from bless cast
by a cleric, does it remove the effect on all the targets?
Dispel magic ends a spell on one target. It doesn’t end the
same spell on other targets.
So strictly by the Sage Advice, only the creature targeted would be dispelled.
In my games though, I rule that the entire parent spell ends.
Notice Dispel Magic says "the spell ends" not "the creature is no longer affected by the spell" or "the spell ends on the target". "Any spell of 3rd level or lower on the target ends" has a different meaning from "Any spell of 3rd level or lower ends on the target". The first says any spell that has that creature as a target ends, the later effectively says that that creature is removed from that spells targets. The later wording "you automatically end the effects of a spell on the target" still doesn't say "ends on the target" to clearly indicate that the "on the target" part only applies to what is ending, but the change of "spell" to "effects of a spell" opens up the ambiguity if the effects of a spell is the spell, or if the effects can be itemized. Though since this is one of 3 statements of what the spell does, and is clearly just suppose to buff the first instance, it would be weird for this spell to do something different at higher levels and should probably just be considered poor wording since it adds ambiguity. And as we all know, the golden rule is that spells do exactly what they say they do. No more, no less.
So (A) that is exactly what Dispel Magic says it does and (B) Allowing otherwise opens a pandora box of "how much of a spell does Dispel Magic actually end" (from experience, the more ground you give a rules lawyer to argue, the bigger the problem will be), and arbitrarily makes swarm/multi-target spells artificially more powerful because they require multiple Dispels to cancel after the fact (and means that using multiple casts for each target actually has tactical meaning). You would also have to track each effect separately if they can be manipulated separately. So for simplicity and easy to enforce consistency, treat Dispel Magic as reducing the parent spell's duration to 0 (aka, the duration is up, the spell ends).