In the moment, ad-lib something.
This happens to me all the time. Despite all my preparations, my players often completely subvert or circumvent my planned sessions. Just last session, they were at the start of a whole dungeon I planned, but they went "This looks too scary," left, and went to scam some vampires in a city I had barely named.
When something like this happens at your table, in the game, all you can do is make something up. It's not easy, but if you just grab something reasonable-sounding off the top of your head and flesh it out, it might be sufficient to get you through the session. Sure, you're probably not going to blow anyone's minds, but you can still have an entertaining improvised session.
Stuff like that gets much easier with practice, however. You'll eventually get the hang of coming up with what you need to keep the game running, and even have some fun/interesting encounters.
Prepare for the unexpected by fleshing out settings
The long-term solution to this is to have a flexible setting, instead of a single railroaded quest. In addition to planning out specific quests and events, you should also decide specifics of your setting: what are the major industries in the area? How active are the gods? Is there a centralized bandit gang, and do they have weird rituals? If they set that inn on fire, what sort of guards will come after them?
You probably won't use 99% of what you come up with, but thinking about such things ahead of time allows you to ad-lib much more effectively, because you already have foundations and prompts to work off of. It also doesn't hurt your actual session planning, either.
Know your players and your characters
It's true that in a TTRPG, anything is possible. However, the realm of what is likely is a much, much smaller subset of those possibilities. It is important to have a session 0 in order to determine the kind of game and characters your players want to play, and you can prepare around that.
For example, if your party is full of lawful do-gooder paladins, you probably don't have to worry about them committing random acts of evil. Likewise, if you know that your players spend lots of time every session messing around with NPCs, you can spend more time focused on making interesting NPCs instead of planning a ton of combat encounters.
In this way, you can give your players freedom to do what they want, but also be prepared for what they'll likely do.
If all else fails, railroading isn't always bad
Railroading is frequently bad, but it's not always bad. If you've planned for a world-ending cataclysm to happen, then let it happen regardless of what the PCs are doing. For instance, after opening a portal to the Abyss, my players decided to simply leave and do something else. As a result, I had the portal steadily chew up huge swaths of the game world until the PCs decided to do something about it.
For your bandit example, your players can mess around for a day or two, but the town is going to get ambushed regardless of what the PCs do. If the PCs decide to go to a completely different town, you can just reskin your bandit ambush to target that town instead, and the PCs will be none the wiser. After all, the players only see what they play--they won't know that random town A is actually town B in your notes.