I believe that there's a mix-up in the moving while grappling RAW, and is actually broken as written.
The RAW says 'speed.' Which is confusing. The speed stat? Doesn't matter unless I add more movement mid-turn (like with a Dash action).
I believe they wanted to limit your movement this turn, while grappling, as they did with the Difficult Terrain in Combat rules, but botched it. Basically, it's a resource naming problem.
I look at it like this:
Speed is the character sheet value that generates a consumable resource each turn, called "movement" (though I prefer "movement potential." "Movement" is what happens when you spend "movement potential.") This resource goes away at the end of your turn (unless your DM allows you to specifically reserve it). The speed stat can be changed during your turn, but doing so doesn't (usually) affect anything this turn. It's already been used. It would affect your next turn, or if you decided to (or got the capability to) Dash after it was changed.
Dragging while Grappling is intended, I believe, to slow your character down, immediately and quite a bit, with one grappled target halving your movement per turn, starting from when you grapple. So it has to affect "movement potential" rather than "speed" (or the speed stat). Likewise, the ability to cease grappling at any time is intended to allow you to regain movement, giving you the capability of leaving the grapple target in a particular place (such as under that big ice boulder over there) and leaving in the same turn.
I believe in the interest of RAI, the written rules for Dragging while Grappling should be written and treated more like the rules for Difficult Terrain in Combat.
So, a proposal for RAI: "While Grappling a creature equal to or one size smaller than your size, you may drag them, but 1' of linear motion costs an additional 1' of movement. Each additional creature grappled, of your size or one size smaller, costs an additional number in feet, equal to twice as much as the last. Creatures two sizes or more smaller have no effect on your movement. Moving a creature larger than your size while grappling costs an additional 3' of movement for each 1' of motion. You cannot grapple more than one creature that is larger than you."
One bigger target: Cost is 1'+3'=4' (a basic medium can move a large 5')
One target: Cost is 1'+1'=2'
Two targets: 1'+1'+2'=4'
Three targets: 1'+1'+2'+4'= 8'
No messing with the speed stat is necessary. If you remain grappled, the additional costs cut your movement. If you let go, you can move more freely.
Using the above rules, let's take the example of a Loxodon (medium) with a 30' speed. It can easily grapple three small-sized creatures (one with each hand, and one with its trunk), and drag them.
But only by expending all of its base movement, and probably whatever else it can scare up. With a base speed of 30' he can drag them a total of 5' (3' actual, with a fraction remaining, so round up). That depletes his movement for the turn. Assuming it somehow got ahold of a fourth, and tried to move on a subsequent turn, it would require more than 40' of movement to move them 5', and more than 120' of movement to move them 10'. So he's going nowhere quickly.
I believe that sorts everything.