In game, make it clear that they won't always win
The cause of your problems isn't open world, the problem is the preconception that PCs can always solve the problems in front of them with combat.
When a level 1 PC sees a dragon, the correct reaction should be to run. That much is obvious. But what about when they are asked to track down and kill a certain wizard. Why should they be careful?
Help the players understand that they will not always win
The first issue is having your players aware that danger exists. Often DMs in D&D drip feed their players combat at appropriate challenge ratings. This gives the PCs the idea that no matter the foe, victory is always possible. This idea is your first enemy.
There are a few classic ways that game designers pull this off:
- Have them fight something and lose, only to be saved by someone else. Eg, have a dragon beat the snot out of them, then have a friendly wizard teleport them to safety, consuming some magic item to signify that they got lucky this time.
- Have them struggle to beat something, only to have that same thing beaten easily by something stronger. Eg have the players fight a bandit captain, the fight is close, it seems like the players may just win, but then the bandit captain drinks a potion and heals up completely. All is lost, but then a dragon shows up and swallows the bandit captain. The PCs should now understand that dragons are way above them
Give the players an out
Especially at the beginning of the game, the players may not understand how to disengage from a fight. You will need to teach them that it is ok to run away. Eventually you will want them to not fight at all if something is stronger than them, but it can be difficult to judge the power of enemies.
If they do get into a sticky situation, they need to be able to get out of it without winning the combat. They need to retreat, lick their wounds, train, level up, and prepare to fight again.
- Give them an explicit out. The city watch shows up, attracted by the fighting, and says they will hold off the foe while the players escape.
- Give them a mechanism to escape. The party is fighting in a dungeon, each room is separated by heavy wooden doors. Previously the party tried to break one down so they know how tough they are. If they can just retreat to the next room, they can bar the door.
- Make them afraid to continue fighting. One goblin runs to a big gong and starts banging on it. The players hear a distant gong answer. Within a turn or two they can hear footsteps and warcries, a goblin host is approaching.
Give the players a means to scout, and help them learn to use them
Because combat is often drip fed at "appropriate" levels, players don't feel a lot of need to scout. Why do we need to research the enemy wizard? Whatever he can do, we can beat him. Why do we need to look in to the abilities of the creature we are going to fight? It's nothing we can't handle.
Once the players know they can't beat every foe, they need to be able to figure out what they can beat.
- Encourage the players to talk to NPCs. Have an NPC volunteer some information about a foe. Oh you are going to the marsh? Beware the goliath toads, the city guard has to deploy an entire regiment to fight even a single one.
- Encourage your players to do good old fashion detective work. Your NPCs should be doing things. The bandit captain leads their bandits in a raid, giving the PCs an opportunity to see them fighting the city guard. The PCs can clearly see the captain's power. They could also sneakily scout the bandit camp and use various abilities to observe them up close, and their equipment.
- Give them a helping hand while they get used to it. Limited charge items that can be used to scry or detect magic or be used to scout are great to help the party until they get into the swing of doing legwork themselves.
For all these points, have NPCs do the same. Have them be afraid of other NPCs. Have them run when the fight gets bad. Have them scout and investigate before a fight. Players learn fast by copying NPCs.
Once the players understand that this is not a safe world, they can take steps to avoid a TPK
Ultimately it is up to your players to actively prevent a TPK. If they do not learn to do their homework before fighting, and have preparations in order to escape, then they will get themselves into bad situations. They need to learn that the GM isn't there to hand wave their bad decisions.
But, so long as they have the tools and support from the GM they can choose when to enter encounters, they can take the necessary steps to avoid a TPK.
A note about teaching players
One way to teach players is to introduce the mechanic to the players before the first encounter. For example have an NPC at the tavern tell the party "goblins are cowardly but cunning, they will seem to run at the first sign of trouble, but don't turn your back because they will be waiting just out of sight with an arrow nocked"
The have the players encounter this situation in a fairly safe way, play it straight. Have the party encounter a goblin scout, when the goblin sees them it will run. If they leave it be, it will put an arrow in them. Now it's up to the party to figure out what to do next. How can they chase away this goblin? Can they hurt it? How can they deal with it?
After that, give the party novel situations that expand the concept of stealthy goblins that avoid direct confrontation. Have the party ambushed at night, allow the party a chance to turn the tables and be stealthy, have a party of goblin raiders use stealth proactively and aggressively.
Finally after the players have mastered the situation they may feel comfortable to fight the final boss. How will the handle defeating the notorious Shadow Gobbbler, a goblin famed for their stealth and underhand tactics?
Once this "mechanical arc" is complete, your players will hopefully remember stealth and how it works, and apply it to other situations. In this way you can drip feed your players if you feel they aren't up to the challenge yet. Remember that even in an open world, the DM has a certain responsibility to the players. You need to teach your players so they can have expectations about the way the world functions.