When to Give Up
To answer the title question directly: When it ceases to be fun.
That seems a little glib, but it is fundamentally the truth. It's obvious that the players and/or the GM came to find the situation unfun.
Specifics from the described situation
The description provided indicates several break-downs in the story flow:
- The players don't know the part isn't needed.
- The players have been told the part is irreplaceable and irreparable.
- The GM has fixated on a single story direction.
- the story direction is fixated on overcoming the engine problem
- the GM allowed failure point on the "railroad"
Taking these in turn...
The players don't know the part isn't needed.
This is a GM communications failure. Simply put, if the part isn't needed, then they should have been told the part isn't needed.
You needn't come right out and tell them, but you can make it clear upon an inspection.
The Part is Irreplaceable and Irreparable.
While not a problem in and of itself, it's generally bad form to present a "big shiny thing" that doesn't work, and then tell them they can neither fix nor repair said shiny part. It's an instant frustration factor.
Some better ways to handle it are:
- to set an insane repair difficulty; on a fail, the PC breaks it.
- to make it so that the story doesn't fixate on the part at all
- to describe the part as safety factor only - for example, the flywheel cover, or radiation shielding. Or the Air Cleaner housing or fuel filter.
- to outright tell the character with the highest repair skill that it's not an essential part.
In other words, communicate more effectively that the part isn't the solution.
The GM Has A Single Story Direction
This is only a problem because the GM failed to present the problem in a way that the players could decode. There's nothing wrong with a bit of story-railroad from time to time, provided the stops along the way are meaningful down that railroad.
The problem is, when the GM puts the story on the rails, the GM needs to make bloody certain he's presenting solvable challenges. And make clear the branches and sidings of that railway.
Sometimes, however, it's better to "say yes". It's turning into a hoary old chestnut, but Vincent's Admonition is very applicable here: "Every moment of play, roll dice or say yes." When they sought to fix it, you should have assigned a difficulty and let them roll, or simply let them fix it.
Mind you, Vincent also says "Bang! Something’s at stake. Launch the conflict and roll the dice." If the conflict is fixing the engine, it needs to be set up as a conflict, not a puzzle. Set clear steps, and then guide the players through them.
"Who's inspecting the engine? Who's attempting to fix the irreparable part?" Once they select, "Tom, while inspecting the engine, you realize that you can work around. If you succeed, the part won't be needed. It's a hard repair roll, and on a fail, you're out time, and . Rich, while examining the part, you realize it's just a safety system. You can attempt to repair it, at Insane difficulty, but if you fail, you break it and hurt yourself. Guys, Going to try?"
If they fail, give them an option to make a sail. And when they do, just say yes. Or...
Failure Doesn't Always Mean Failure
Sometimes, the task can be redefined so that failing the roll isn't failing the Task.
For example, telling Tom "If you fail, you'll still get it working, but by using servos from your left hand." The stakes are now personal, rather than story impeding. If he fails, he still gets it repaired, but now has a permanent injury to later repair.
Meaningful stops on that railroad - ones that matter.
The Players Need To Have Input
The players set themselves a goal - to get it working, period. Once they make it clear it's their goal, the GM needs to give up his story in favor of their fun.
So, in this case, once they got fixated, the GM needs to realize that his "minor encounter" is now a "big deal encounter" and treat it as such. "Say Yes" and get it moving, or make it a series of meaningful rolls which have clear consequences for the story.
Those consequences need not be "you failed to fix it," as mentioned above. They can be, "You hurt yourself" or even "You figure you can repair it in about 4 weeks of machining new parts" - but some way forward needs to be evident.