Not at all
The relevant section of the grappling rule is:
Moving a Grappled Creature. When you move, you can drag or carry the grappled creature with you, but your speed is halved, unless the creature is two or more sizes smaller than you.
So, you can move a grappled creature when you move maintaining the same orientation and distance between you and you can move without moving the grappled creature allowing you to change the orientation and distance between you but you can’t move the grappled creature on its own.
Some specific ‘grappler’ monsters have specific rules about moving their victims that operate in addition to this general rule.
Is this rule restrictive and unrealistic? Yes, but then so is every rule in the game for any personal level of restriction and realism.
There are various schools of thought on this which, at the extremes go:
- That's the rule - we're playing a game so arbitrary cut-offs that create cognitive dissonance are inevitable if we want consistency and certainty.
- This rule is stupid so I by my power as DM am going to rule differently, the new rule is X.
Put yourself on the continuum and make the ruling.