RPGs don't leave the GM helpless.
A computer is helpless against its programming. If you're playing a computer RPG and on a combat map, and you're losing, you have to choose from the options programmed for you. If there's no retreat, all you may be left with is to get lucky and win or die and reload.
A GM is a human being. They're not following programming. They have the responsibility and the power to-
Let's start there.
Player Roles, or: How to Win at RPGs
An RPG of the type you likely think of when you think of RPG is played by players with very asymmetric powers and desires. Most players are regular players, each controlling usually one character in a world, the Player Character. They make decisions about what their character thinks, feels, does, and wants, but the outcome of their actions is often in doubt. One player is the game master, usually called GM but the name varies based on system. They're responsible for presenting the rest of the game setting: the whole world and everyone else in it, friends and foes, gods and devils, and they have the power to make vast, unilateral declarations like "the town gate is locked" or "these two kingdoms are at war" or "the sun explodes".
You can only win an RPG if you're a player character, and usually it's more of a self-directed goal? You're trying to get your character to somewhere satisfying to you, even if the entire world is against you. Death is usually something you don't want to happen, but there are also plenty of little things your character wants. Fame, fortune, love, respect. The role of the GM is to play the world believably, and giving you opposition to overcome that makes sense to the place and time you're at in the world is part of that.
It might seem like there's a mirror to the player's goals, that the GM can win too by stopping them, but that's not the case. The GM may play people dedicated to stopping you, running them honestly to the extent of their capabilities, but if they in their role as GM really wanted to stop you? They can blow up the sun.
Everyone's there to have fun playing an RPG, but you can have fun playing something even if you don't win. You can have fun playing pickup basketball if you don't win, but if you play pickup basketball and just try to have fun but not win, that might not work out so well for the fun of your teammates and opponents who are trying to win. So it's expected that everyone plays to win. It's "being a good sport".
In RPGs it's expected on a similar "being a good sport" level that the GM will play fairly but that the players will win... at least in the short term. Most of the obstacles a GM presents are not intended to be tests of life or death but rather to use up the players' resources and thus curtail their ambitions. So how do you get to the point where you're at a test of life and death that the players are failing, and what do you do then? Well...
Dice Hate Me, or: Disclaiming Power and Responsibility
Successfully playing a game where you cannot win but must present a credible challenge is something you might expect out of a true virtuoso player. Not everybody who picks up an RPG book to GM is one of those virtuosos, which is why most RPGs include some random element. The random element, usually dice, is given the responsibility and thus the power to decide what happens under certain circumstances. "You have a 70% chance to climb this cliff." "I'm climbing this cliff!" "Oh, good job, you made it!" feels weird as a conversation between just a player and a GM - how is the GM deciding that? Are they really being fair? But with a dice roll in the middle of it it feels fairer all around. There's a risk of failure that's visible to everyone, and it feels good when it doesn't happen!
And of course, when death, the ultimate end to all ambitions, is on the line, many RPGs go heavy in the dice and impose a lot of structure on the play - you know, "the combat system" - so that the chance of failure is there but the odds can be rigged in the PCs' favor.
As an aside, this is why most GMs don't just offer up a "load game" option, where time rewinds to before the combat and the party has another go. They could, of course, and if there was some misreading of the rules or something they sometimes do. And GMs always have the power to do that! But doing that means overruling the dice, which are supposed to be the fair arbiter here, outside the GM's fancy. What sort of credible opposition breaks the rules to give their opponents the victory?
This is different from something else which would probably show up in a computer game as "undo last action". Since it's assumed that players aren't deliberately trying to die, a GM can pause before a player e.g. opens fire on forty horse riders charging them down on an open plain and ask - wait a second. This is forty horse riders charging you down on an open plain. How are you planning to survive this? Often it'll turn out that a player had a different view of the circumstances than the GM - that they could hide to the side in the cloud of dust the horses were kicking up and take potshots which would be taken for more arbalest fire - so pausing to reflect and reframe things in this way is just making sure everybody has a clear picture of what's going on, not denying the dice the power they're supposed to have over life and death.
Oh, right. We were talking about how you gave the dice the power over life and death and now they're trying to kill the PCs. What's up with that and what do you do?
The Long Odds, or: Failure Is (Not) An Option
The sad truth is, a lot of RPG system and scenario design has a common problem: the players are "expected to win", so the designer may not spend a lot of time providing ways to deal with what happens when the players don't, or even just throw lethal obstacles in the players' path and expect that the players can just have the dice fall their way.
Like: a fiend from hell, bound to your destruction, that can teleport faster than you can run. It throws a frustrating wrinkle into a tactical combat, sure! But after a run of bad luck, even if the system has a formal numbers-based offramp for escaping from a fight, that whole "teleport faster than you can run" thing is probably going to seal the PCs' fate.
So a skill a GM has to develop is to recognize what they're leaving up to chance. Anything that's possible could happen! If you throw that fiend into combat with the PCs you are leaving it up to chance whether they live or die! Do you want that? But what can the GM do?
Remember, the role of the GM is to play the world believably. Something that wants nothing more than to kill the PCs is fine as long as it fails, but there's the chance it will succeed. In that case what you need is something the creature wants more than to kill the PCs, and a way for the PCs to get that if it's not out there already. The riders want to escape the town more than they want to kill, so they'll be satisfied with leaving a PC for dead and riding off. The fiend from Hell has been compelled into killing the characters by a contract, and it wants a better deal - maybe it binds the PCs to tear up its original contract, or demands a pact from them.
But again, if a system isn't designed to support a more graceful out from combat than "one side or the other is dead", even playing the world in this way can seem like you're making an exception for the players that the system never covered.
Systems That Cover, or: Our New Offramp
Here are three modern engines that support an alternate out from combat than "one side or the other is dead".
Powered by the Apocalypse/Forged in the Dark: "What combat system?"
Both of these engines support players taking actions at a dramatic pace. Characters can fight for their lives, sure, but they're not doing it in any special kind of rules structure. If things look bad they can just try to run or talk it out with no special interruption.
Both also generally don't have the GM rolling dice to determine what their characters do. The players roll dice to adjudicate their actions, and on player failure the GM is given a free hand to impose negative consequences - so the GM keeps a tighter hold on the power of life and death.
Fate: Pull the Ripcord
Players in Fate campaigns have a generally free hand to demand the GM give them an exit from combat on their own terms. The Fate conflict system can be used to resolve all sorts of things, not just fights to the death, but within it any player can choose to concede unless they're staring lethal damage in the face.
Conceding means that the opposition accomplishes their goals but the characters keep control over their own fate, and the GM gets to decide how that happens, not the dice. If just conceding would leave them in too bad of a position, or they have reason to want to leave combat without conceding, players can also stop fighting and run, kicking off a different structured play element.
Torchbearer: Setting Stakes
Torchbearer's conflict resolution system doesn't have an out. What it does have is a variety of ins. It is a game that tries to evoke an earlier, deadlier style of play, where Cord the Fighting-Man could just drop dead in the dungeon and with a minimum of mourning and reasoning, Bord, brother of Cord, would be right back in the fray next play session, but for all of that death doesn't have to be on the line.
A Torchbearer conflict starts with the PCs setting goals, and their lives are on the line only if they decide they'll be fighting to the death instead of trying to drive off, capture, run from, argue with, or outwit their enemies. (That's not strictly true. Characters who are already injured can still risk "a fatal injury" in any physical conflict, but again, they know that going in.) PCs aren't necessarily expected to win the non-fatal conflicts, either; one of the most important mechanics of the game is that suffering setbacks based on your personal quirks motivates you to do more during your downtime in the dungeon, and it's easiest to suffer setbacks when you already have terrible odds and they're not likely to hurt any worse.
Even in a fatal conflict, though, PCs still have something to fight for when things look bad, because the winners of any conflict owe the losers a compromise depending on how close it was, and the compromises for fatal conflicts are spelled out in the rules. A close fatal fight can result in the losers just being left for dead instead of dead outright, but even in something that seems like a straight-up stomp, Cord the Fighting-Man can at least try to force a minor compromise and hide a pouch of gems and the family axe in a place only Bord, brother of Cord, would think to look.