Focus your preparation on what you expect them to do, but have contingencies ready so that they can do other things if they choose.
I run my current 4E Neverwinter campaign as a bit of a sandbox. I say "a bit" because everything is very time sensitive. This means that I don't have to worry about my party showing up and saying "hey, we've had a month to think about it, and we're getting on a boat to Kara-Tur, go". When one session ends, the next session starts right where it left off. We've been playing for nearly a year, and have had less than two weeks of in-game time pass.
This structure allows me to ask the party what they think they're going to do next at the end of each session: "Okay, you defeated the patrol. We'll stop here. What do you think you will do next?" "Well, that castle over there seems interesting, we'll probably go in there."
So, for the next session, I spent a lot of time on the castle, specifically the first few rooms. However, because this is supposed to be an open-world, I also updated and prepared the surrounding area. I was aware that there was a scout watching them fight the patrol in the road, and decided that that scout would go get reinforcements once the party left the area. I expected this patrol to follow them into the castle. However, they decided not to go into the castle, and then went down the road further into town. Luckily, I had prepared the reinforcements encounter and set up security checkpoints around town. I also had a general idea what the important NPCs in the area were doing. The whole session ended up focusing on the party's fight with guards at a checkpoint, followed by fleeing and evading the reinforcements. They got help from one of the nearby NPCs, and ended up at their base.
Now I ask them what they think they'll do next. I get an escape through the sewers ready, and slightly update all the other things I had ready in the area. They'll get to the castle eventually, and it will still be ready.
Pay attention to what interests that party. Add more detail to these things.
During the escape through the sewers, the party asked about where other side tunnels led. Because of the way they responded to my explanation, I think they'll probably come back here and check it out at some point. Therefore, I'll write some notes on what is down that tunnel, and be more prepared if they ever get around to it.
Get more specific the closer your party is to the thing.
I know that the Netherese are out in the woods. I have a general idea of the kind of encounter I might like to occur if they go out there. However, the party has yet to express any plans or interest in heading to the woods, so I don't expect it to happen any time soon. This means that I haven't wasted any time putting together a specific encounter. If I did, I wouldn't know what level to make it. Instead, I put together a few notes from an in-universe perspective, and if they somehow manage to jump through a portal and get out to the woods before I expect, at least I have some notes to help me wing it.
On the other hand, if the last adventure ended with them standing at the door to a temple, and I know that they're going in the temple, then I'm going to decide exactly what is in that first room. I will know how many enemies, what level, where any traps are, what kind of loot exists, if there are any secret doors, etc. I'm going to spend a little time on the buildings and streets adjacent to the temple, and a little time on things that might be at any exit, and a little time thinking about the purpose, role, and history of the temple, in case they do something crazy and I need to make something up.
If they wander out into the unknown, make some stuff up. These are now facts.
"What's the dwarf's name?"
"Oh. Crap. Name. Uh... Malin. His name is Malin."
There is now a dwarf named Malin in your campaign. If the party cared enough to ask his name, they thought he was interesting. Malin is now no longer a random nameless NPC, he's a person the party knows. Write this down. Next time you need a dwarf in the party's lives, maybe Malin will be a good fit.
"Is there any loot on the tiefling?"
Crap, I just made up the tiefling because I didn't expect them to cross the bridge.
"Sure, he has a diary in his back pocket. It's written in Netherese."
Haha! No one in the party knows Netherese. Now I have time to make up some diary entries before they find a translator. I can use these diary entries to give them a quest hook. It will seem to the party that they have stumbled on something important that I put together for them, when really they made up the tiefling when they crossed the bridge, and they made up the diary when they asked about loot.
Don't get too far ahead with your story.
Every session, your players will do things you did not expect, and all of these things should have consequences.
Therefore, don't plan out a long story in advance. Make the story more specific and detailed the closer you get to it. The easiest way for me to do this is to plan the situation, not the story. After each session, I look at all of my NPC's plans and see if the party messed with them. If not, I conclude that the NPC's plan progressed a bit, and update their situation. If so, I get in the NPC's head and see how their plans will change. But plans are not things that will happen in the future, they are ideas that the NPCs have right now. The NPCs' plans are part of the situation. When it comes to writing scenes and dialog, I almost never go further ahead than the next session, and a lot of times I don't even end up getting to that.