A grappled creature has a movement speed of zero, so it can only be moved by the grappler (or an effect that forces it to move, like thunderwave).
Since the only movement that matters in a grapple is the creature/player doing the actual grappling, that's where the focus lies. The pertinent rules are as follows.
For the grappled target, under the PHB pg.290, title Conditions:
• A grappled creature’s speed becomes 0, and it can't
benefit from any bonus to its speed.
And for the rules as they apply to the creature/player doing the grappling; PHB pg.195:
Moving a Grappled Creature.
When you move, you
can drag or carry the grappled creature with you, but
your speed is halved, unless the creature is two or more
sizes smaller than you.
This makes the physical location of the grappled creature entirely irrelevant by RAW since it's movement speed is zero anyways. All that matters is the grappler's location, and the terrain the grappler is moving across.
In addition, your specific scenario would also deal damage to the creature since Spiky Growth does damage. Your movement speed would follow the grappling rules, but the creatures location with respect to the grappler is up to the creature/person that has established the grapple. If you control the target's movement, you can position them wherever you please.