As a GM, Never Decide A Correct Course of Action, Only Decide Consequences
As a GM your calling is to create a great story. Stories are about conflicts that arise as characters with motivations make choices. Thus, the best thing you can do as a GM is to structure the 'adventure' to be about the story and the conflicts, and not about something concrete being done.
When you structure your adventures this way, you entirely avoid the issues of the players not doing the 'correct' thing: everything is, by definition correct, and the interesting part is the results of the choices being made. Let's look at your concrete examples:
- “We turn around and go home.” In this scenario, assuming nothing is naturally stopping them from going home, you should let them. But ask yourself why they were there in the first place. Was it to stop an assassination? Then perhaps the assassination goes forward, and they see the throne war that begins as a result of a dead king. Was it to put an undead creature to rest? Bring the point home as tales of children going missing rise and priests turn their backs on the PCs.
- “We leave the monsters alone and decide to go back.” What happens when the monsters aren't stopped by the PCs? Either they continue to wreck havoc, or someone else kills the monsters, gets the attendant rewards and then uses those rewards in a way that impacts the PCs.
- “We do not open the strange door.” This seems reasonable! Realistic characters won't do dangerous things without a motivation. What is their motivation? Give them a motivation, or tie one they already have to opening the door.
Try and consider all the characters in your story (including the monsters, and indeed the 'environment') and what their motivations are. If the PCs mess with those characters, what do they do? If they don't mess with the characters, what do they do?
Telegraph Consequences
Suppose you set up a story about a dragon ravaging the countryside. If the PCs are implored to go deal with it... and they don't, what will happen? At the point where they have the choice to not deal with it, make sure the PCs have the information to know what would happen if they turn away: in the case of the dragon, this is easy. More livestock is eaten, perhaps a fair maiden or three. Someone's father.
Why do I call this out particularly? Because as a GM the difficult part is tying the consequences of actions to the stakes the PCs have. If the PC has a little brother, or a particularly large field of wheat, these are things that become at risk when the PC chooses not to deal with the dragon. Make sure you draw a line between a 'freely roaming dragon' and the stakes the PCs have. This allows them to have a real motivation to engage the issue.
And in this scenario, maybe they don't deal with the dragon. The little brother is eaten. What does the mother do? Does she silently accept the loss of her child, or does she demand the PCs seek vengeance? If they refuse, what is the grieving mother likely to do? How quickly does the community turn against the cowards? Or perhaps a knight comes along and does the deed, earning the love of all, and is granted the village as a fief. What kind of knight is he? Perhaps he thinks killing the dragon means he gets free stuff from everyone all the time, until the town is in open revolt. Or maybe he's just so goodly that everyone else guilt the PCs into doing things. Or maybe it's a mixture. The point is the world should keep spinning until the PCs are caught up in the situation and start acting on their own motivations. And once that happens, the world should still keep spinning, to add velocity to those motivations, wherever they take the characters.
But to do that you have to telegraph the situation: if the knight mistreats the tavernkeep on his way through to kill the dragon, bragging about how the town will become his after he does the deed, the PCs now know what to expect if they kill it first (the knight goes away empty-handed) and what happens if they don't (they come under the jurisdiction of a brutish thug). Now they can make their choice without being in a vacuum.
Avoid Railroads
There is a door ahead of them, and they choose to turn around and go home. Oh, no! Cave-in. They have to go forward.
The problem with this scenario is that it forces the players into what I call 'airplane mode'. They're basically watching a story unfold before them, and come to understand they don't have much say (like sitting on an airplane). This will reduce the engagement of the players. But stories are about conflicts: if you follow the above advice they're choosing between two different conflicts, rather than one conflict or nothing. Don't make them do that.
The worst thing you can do is put something in front of them that you want them to do, but give them the choice to do something else... only to veto that choice every time they make it until they make it 'right'. Consider:
"I turn away from the door."
"The tunnel collapses behind you, leaving only the door."
"I start to dig my way out with my shovel."
"You get a few feet and then your shovel breaks."
"I cast my 'teleport home' spell."
"The magic builds around you and then inexplicably fades."
"Fine. I sit down and wait for it to come back."
"You find yourself falling in love with the door. You can't help but turn the knob..."
Obviously this is ridiculous, but shows how railroading and forcing a player to make a choice turns the whole thing farcical. (Note: this is functionally different than them failing to get out for other reasons, such as bad dice rolls, which can lend a sense of desperation that is very healthy. I'm only talking about fiat decisions on the GM's part.)
Hide Unavoidable Railroads
If they're moving through the dungeon and they're getting close to the door and you're afraid of them backing out... then make sure that you telegraph that they won't be able to. The roof should be visibly unsteady, or the earth should start to quake. If you have to collapse the door behind them, collapse one exit but not the other... and underscore why they might want to go forward instead of back. Make it feel as though they have an out, but keep highlighting the 'interesting' option.
Making this work is very tricky because players will quickly pick up on the fact they have no choice. You have to be very skilled to hide the fact from them that they're not making a choice. You have to know the characters and the players very well. Players often feel betrayed if you're promising a choice and don't deliver. For that reason this is best avoided.
Declare When There Is No Choice
If hiding the railroad is not possible, and the railroad is unavoidable, then make sure the players know there isn't a choice there. Take responsibility for not giving the players a choice.
"You find yourself standing before the cave of the dragon. How did you get here!? It was all such a rush from the moment the farmhand rushed into the tavern. Worse, you know dragons have a keen sense of smell, and now that you've been to the lair, nothing in the world is going to save you. You'll have to deal with the dragon somehow."
By narrating this you're telling them, as a GM that the dragon is simply going to be a problem for them until they deal with it, and you don't even ask them to confront it: you simply put them there. To do this you have to earn the trust of the players: they need to understand that this isn't closing off their choices but simply setting out a set of choices involved with dealing with the dragon, rather than a set of choices amongst which 'not dealing' is an option.
Being upfront about this is like having a good video game teaser video: you don't expect a video of isomorphic city building to be about first-person card-playing. In the former they'll expect to make choices about resources, and in the latter choices about which card to play and where to look.
Using this reasoning, you can also fast forward past boring parts of play: by definition those parts that have no choices to make, no conflict to resolve and no telegraphing to be done.
Accept Offers
Finally, recognize that when players choose something unexpected it's for a reason. Here are some reasons:
- They are unengaged with the game and don't care about the main plot. This is a meta-concern and you need to find a way to make the main plot interesting, rather than one thing amongst other things.
- They are actively messing with you. This is an out of game concern: there is no cure for a player who doesn't want to play than to not play.
- They find something else much more interesting. This is actually great! By 'accepting their offer' to look at that other interesting thing you could open up a huge avenue of adding material to the game. Remember roleplaying is collaborative, and if you are collaboratively finding that finding the buried treasure is more interesting than the dragon eating livestock, by all means follow the buried treasure thread.
For this last reason, if the players are making unexpected choices, try and honor those choices: try to ascertain why they're being made, reinforce that it's a valid choice, and finally add a complication that articulates and expands on the new set of choices that result from the player choosing to go in that direction.
The Dragon Isn't There To Be Killed
If you take away one thing from this, recognize it's all about motivation: the dragon isn't there to be killed. It's a dragon. It has things it wants, things it's good at, things it avoids. If it's not doing anything the player characters care about, they're (probably) not going to go kill it just for the sake of it.
The corollary to this is that you shouldn't add a dragon to your story unless it is there to do something with the story. Don't add it just to be there for the PCs to kill. Smaug went to the Lonely Mountain because it was the biggest stash of gold around, which motivates him. He kicked out the dwarves. He was only killed because the dwarves came back with a hobbit that from him. Smaug went on a rampage in anger... which led to a character shooting him to protect his town from burning down. It would have been a way worse story if a random human said, "There is a dragon. I shall shoot it."
So follow the motivations back. Add them to characters where it's reasonable to justify interactions you want to have happen. Add characters if it doesn't make sense to add it to any of your current ones. And then, when PCs do something unexpected, just see how the various characters' motivations interact with the choices made by the PCs.