For reference, I'm really inspired by some of the iconic battles in Star Wars, such as the Duel on Theed or, more notably, the infmaous Duel on Cloud City.
I want to design combats that span the distance from the Carbon Freezing Chamber to the Reactor Shaft, and everything in between, and I want my combatants to naturally transition from one location to the next. I want to design monsters and enemies that can constantly gain ground, pushing the proverbial "Luke Skywalkers" across the catwalk with their backs to the walls, and I want to give my players the resources to do the same.
I know there are obvious rules for pushing combat from zone to zone, but there are only so many ways you can use thunder wave and dissonant whispers, and there aren't a lot of attack actions that also push/trip when they resolve.
Do any of you have experience doing this? Are there game mechanics or homebrew resources (official or otherwise) that make this easier to accomplish? Or should I design these combats as gridless and simply describe the gaining/losing of ground as part of my narration?
PCs have no reason to move during combat unless you give them one
In my game experiences, I have seen very little movement by creatures during combat that wasn't forced upon them. There really isn't any incentive to change position (mostly because moving away/out of reach generates Opportunity Attacks.)
What you need to consider are environmental or other effects/objects that require interaction or movement with/around by all the creatures. Otherwise, the risk of moving away is greater than the reward of the more cinematic feel.
So what next?
Consider the environment is moving itself. Whether it's on a slope, there's high winds/water that is moving ALL creatures around or something else that creates the requirement of movement without putting creatures at risk of generating OAs to move. Or even bits of the environment are breaking down and remaining stationary will kill you.
Consider adding elements that cry out for interaction. Whether it's the discovery of Glyphs (via Glyphs of Warding) that will buff/debuff creatures (giving an incentive to move to one, or to move someone to one), environmental interactions like statues that could be toppled, or chasms/bridges that can be closed/withdrawn, etc.
Consider Simultaneous objectives in separate locations. My concern with this is in splitting the party, but an encounter that requires split parties to fight simultaneously - and possibly require a positive outcome from one to improve odds in the other is an interesting device.
Utilize an alternate grid system. As an example, in one of our sessions the DM prepared a series of cards layed out like a grid. The story was we were in the Plane of Air and were travelling via the strong winds. The problem was that we didn't really know the path and the winds could take us places we didn't expect. We would choose which 'direction' we wanted to go and flip the card. The card would have a direction choices of which way we could go next, or have an encounter. Utilizing this system, you could have grids that do different things to create movement/action as they cross them.
Consider giving bosses scripted special movement abilities that trigger at certain HP thresholds. Maybe when a boss reaches 3/4, 2/4, 1/4 he teleports/does a massive leap/knocks back the party (with some chance to save)/breaks the floor and the party finds themselves suddenly somewhere else or is forced to chase it down. Just be sure that if the party has to chase that they have a reason to do so. Either a boss they don't want to get away or are still in danger from it. The actions should be part of the narration and cinematics - not just a standalone action.
Think outside the box for how you approach designing combat...
I can speak from my experience here.
I've essentially made 3 additions to my games, 2 of which you touched on in your question:
Spells - as you've already alluded to, certain spells can really make the combat more cinematic. A well timed Thunder Wave can not only force the players away, but can have effects on the environment as well. Oh, your party is having an intense battle on the top of a cliff overlooking a city? (Enemy thunder claps) well now you're having an intense battle over the city, with citizens watching and cheering on the PCs. Is the battle inside a space station? Great, a wall of the station has erupted instead, and now your PCs have to figure out how to clog the leak or else risk being pulled out into empty space. Even spells like plane shift or dimension door can be used in similar ways, so I would urge you to consider the effects of these spells beyond the PCs themselves and on the environment as well.
Grid changes - this is another thing I've implemented in my games with little extra effort. Prepare in advance a few different environments in which to battle, and then using spells or other means, have the PCs moving throughout the different terrains. For example, maybe your PCs are fighting the BBEG in his temple. Lay out the grid, and add the relevant additions (eg statutes, fire pits, etc). Then, part way through the battle, have the BBEG plane shift the party to his true home on the Abyssal plane. As you're describing the new environment, make the relevant changes on your grid. The PCs now see they are in the middle of a coliseum (change the square temple border on your grid into a larger circle), with a giant pit in the middle (remove the statues and instead draw out a large pit). By describing the new terrain as you change the map, you can keep the PCs interested while also engaging them in more interesting combat scenarios (as you realize you're in a coliseum, you also notice the huge crowds of demons and imps populating the stands, cheering for their champion, the BBEG).
Music - this was a game changer for me. Depending on the setting, I'll find a relevant movie, and use its soundtrack throughout the game (eg fighting in space? Star Wars soundtrack. Elves battling Orcs? Lord of the Rings, etc.). This really helped make the game more cinematic since the music added a level of immersion into the game that otherwise would not have existed. For me, the addition of music is what really brought the combat up to the next level for my PCs, and has become a regular expectation at our table (so be prepared to be using music in your future games as well!). And the music doesn't have to be limited to combat either. The Skyrim soundtrack has been the music of towns and inns throughout my campaigns, and the PCs really appreciate the world-building that music adds to the game. Music can also help change scenes - music in movies is used to familiarize the viewer with a particular environment, character, or mood. Changing the music allows the director to easily transition between scenes, characters, etc, and there is no reason why the same cannot also be implemented at your table.
Finally, in these larger, more cinematic combats, I find that the party can sometimes get split up (eg I'm battling clone troopers by the Carbon Freezing chamber while you're in the next room throwing shade at Jabba the Hutt). Keeping some extra yarn handy (this is what I use to denote boundaries on my grid) can help keep all players immersed by ensuring they're all still on the grid.
The target or goal of the PCs needs to either be where the PCs or not, OR have said target/goal itself be on the move.
Step Two: Introduce a sense of crisis.
This can be the obviously overwhelming firepower/enemy behind them giving chase, the juggernaut in the labyrinth hot on their heels, a natural disaster rapidly approaching... whatever works for your players.
One of the simplest methods I have to share is to play psychological warfare on the PLAYERS themselves. Bring a bag with you. Once the chase scene starts, disclose from the bag a simple egg timer (up to one hour, given the amount of time it takes to resolve combat/chase maneuvers). Set it. Melodramatically place your timer in a prominent place. Glance meaningfully at it every time the players start to debate, and sigh, or hum cheerfully. If they really argue, stop them, then adjust the timer down a bit, then indicate they can proceed with their argument... if they really want to.
Bonus points if you have a cool hourglass or other quirky timer.
Step Three: Location, Obstacles, Attitude!
Get into the swing of things. Literally. Blow things up, present sudden obstacles that will give a sense of crisis, yet require a split second decision to resolve, get around, blow up, jump/fly over, etc. Bend the laws of physics (this is a fantasy game, btw, you shouldn't be sweating the physics). It won't be cinematic if YOU aren't thinking cinematic! Villains get to taunt as they run away (with the target, or better yet, a fake target while the real target is elsewhere) regardless of range of voice or speed of wind.
Things fall over. Embarrassing clothing gets caught on peoples heads. People get in the way. Chickens, ducks, and goats come out of the woodwork. You have time to snap off a wink and a smoldering look to that hot young lady you just passed by. Angry mobs chase you. That wizard whose foot you stepped on throws a fireball at you in passing. Things blow up. You swing on ropes that randomly fell from a passing airship (even in the middle of the desert), and slide down cables. Improvised weapons. Angry townsguards join the chase or block off your route inconveniently. Somebody parks a cabbage cart in a really inconvenient location.
Step Zero: Beg, Borrow, Steal. And prepare ahead of time, all really good villains... err, DMs do.
Watch cinematic movies. Add fantasy if it isn't already one. Study the chase scenes. Question them, why are they moving? Why are they here, then there? What is the goal, the motivation? What are they using? Did the transition from part to part seem natural or forced? Why? How could fantasy elements make this chase scene even more awesome or better?
Go watch the Tintin movie. Now that was a chase scene. If you can't get inspired by that, then you are dead to cinematic. ;D
A game I'm in had quite an interesting scenario. The party was on a boat transporting a prisoner to his home country for trial. We were boarded by mercenaries after our prisoner (led by a rather OP lizardman) who proceeded to fight their way through us down to the brig of our ship, we were forced to follow in order to try and fight them off.
This could be a good way of going about it, having a strong NPC who's goal isn't to kill the party but to get to somewhere on the other side of the party (which obviously the party want to stop). The NPC attacks the party but every other turn or so they'll move towards their goal forcing the party to follow them.
Here's the thing. When you add motion to combat, you introduce a set of variables that are hard to control and that make the whole process even more complex.
But you have a great idea. I would limit the possibilities for moving. You don't want members of your party getting far away from each other. You don't want characters abusing your rules or getting abused by unforeseen consequences.
For this to work, both sides need goals of some kind. Vader wanted to recruit Luke, not kill him. Luke wanted to kill Vader, but that goal quickly changed to escape. Goals give you a narrative, and a narrative gives characters reasons to move.
Here's a proposed, untested rule for handling combat movement. If there's something similar out there, well, great minds think alike.
Some may complain that this rule hurts player agency. If the alternative is certain death, I'll take the humiliation. Keep in mind that this rule should be used sparingly.
The Harry & Parry Rule
Prerequisites: The DM must approve each use of this rule, mainly to avoid abuse. Reasons to disallow include two combatants who are evenly matched, a defender who's avoiding combat unnecessarily, a defender who can't retreat, a tight schedule, and lack of a clear goal for the attacker. Reasons to allow include an attacker backed into a corner, an attacker who needs to reach a location, and a defender who would otherwise be killed.
First, a character must declare that he will use the rule in a given round. The rule remains in effect during the round unless it becomes irrelevant, e.g. opponent is killed. The character must take the role of attacker or defender, which forces his enemy into the opposite role.
Second, the attacker makes a normal attack, with all applicable modifiers. Damage is calculated normally but not applied. The attacker is harrying, which involves footwork, taunts, brute force, and intimidation, as well as the usual weapon skills. The defender is parrying, with every move devoted to avoiding the onslaught.
Third, compare the damage total to the defender's Stability Rating. (This is the untested part.) The Stability Rating could be the defender's level multiplied by some constant. It's a measure of how hard it is to make the character move. If the damage equals or exceeds this number, the defender must retreat one space. If the damage is twice this number, the defender must retreat two spaces, and so on. The defender chooses which direction to move. The attacker has the option to follow. If the damage is less than the Stability Rating, there is no effect.
Fourth, the defender makes a dexterity check (assuming he had to retreat). If he fails, he is considered off balance. At this point the attacker may take a single action to accomplish his goal. If the terrain allows, he may push/trip the defender. On a sloped or uneven surface, the defender will fall, with the obvious consequences. At the edge of a drop, the defender gets a saving throw, with success meaning he falls but grabs the edge.
An attacker may continue to Harry for any number of rounds. A defender who is off balance will continue to be off balance, with a penalty to his Stability Rating. The defender may regain his balance by passing the dexterity check, although he still must retreat. The attacker or the defender may call off the action at the end of any round and resume normal combat.
The defender and the attacker take no damage from the Harry, representing the attacker's superior skill. They may, of course, take damage from falls, spells, attacks by other combatants, and other dangers.