You don't have to envision each and every possible way that a PC could derail something. Just get in the heads of your NPCs
I know, it seems like it's really weird, but knowing how your NPCs think instead of trying to plan for every eventuality can help you roleplay them when things come out of nowhere or when your players do something completely off-the-cuff, and it can also show the characters that you really care about the world they are in. Consider that your players are doing this all the time and don't even realize it. For them, it's easy: they only have to do it for one character, though they do have to do it for the whole campaign and often change to reflect changes in character.
You as the GM, on the other hand, have to roleplay most or all of the NPCs in your world (thankfully not all at the same time), so you can afford to not roleplay as-deep as a Player Character would.
With Minor NPCs, like thugs, mobs, and shopkeepers, you can often get-by just describing their alignment, goals, and personality, even using a group mentality.
The Angry Mob (CN) is upset with the Party because the PCs are preventing them from attacking the Tiefling community they believe is behind a series of disappearances and murders tied to dark-magic. Tertius the Mayor (LG) acts pretentious, but is insecure and will cow before a suitably-forceful challenge.
This example shows motivations, makes it easy for you to set-up responses in your head, and allows you to know how to easily convey this information to your party (likewise, if the character is secretive, it helps you know what to avoid). So, if your PCs confront the mob as cooler heads, you can just think of how any mob in a movie reacts. If the PCs do something unexpected, like clobbering a bunch of them, the mob will disperse, but the people will transfer their fears onto the PCs, calling them "servants of Devils" and possibly refusing to help them find the real killers. It also shows that they aren't truly evil, and reminds you that powerful-enough good-based arguments will work better than platitudes. ("You'll get to the bottom of this? That's what the Mayor's been saying for months, and has he? No!")
If you want to flesh them out a bit more, individualize them a little bit.
The thugs who attack the PCs want their money, not their lives. Anton (LE), their Leader, has a cruel streak a mile wide while Michael (CN/LE around Anton) is loyal to and takes orders from Anton, but Johan and Andrea (CN/CG) are only doing this to survive, repressing their objections to Anton's excesses. They would help the Party, but they are terrified of Anton.
Consider that, in this example, you've anticipated that the PCs might feel empathetic towards some of the thieves they first encounter, and also have created the potential for them to feel that empathy. You don't have to necessarily write down what they are going to do in any possible circumstance, but just knowing that they're terrified of Anton means that you already know how to flavor anything they do. Also, if your PCs end-up attempting to influence them, you already know how they have to influence each member.
For Major NPCs, such as major antagonists, campaign bosses, or just NPCs who are really important I would go ahead and suggest making entire backstories for your major characters, because they definitely deserve it. Plus, you will probably only be playing a few of them at a time, so it won't be so difficult. Something else I'd highly recommend are "Character Questions," (also known as OC Questions) designed by authors to help other authors with making characters, and playing in an RPG is very, very similar to writing a story when it comes to character design.
This PDF is really good if you want to go extremely deep into your character. It's really (REALLY) long, however, so if you're looking for something shorter I'd recommend either of these two D&D-specific character question sets.
For example, the Arch-Magos Eluthrina (CE) was a good mage who travelled far down the path of things Man was Not Meant to Know. Where she once called rains and sun to ensure a perpetual bounty, she now sends storms and fire to destroy without pattern, leaving some patches of crops standing in otherwise barren fields or blowing-out specific windows, seemingly without any rhyme or reason but her own agenda... or that of a being much worse.
One thing I'd immediately suggest doing in this case would be to tie her to an "Greater Old One" (3rd Ed has a lot of these, or you can design your own), as if she was a Warlock of that Pact, to represent its corruption (and possible extra powers). Whether that being directly controls her or if she is just heavily-corrupted by the influence of that being is up-to you, as is whether she can be restored to sanity, or if she'll ever be the same again. This can be used to help guide her actions from being completely unpredictable (which often comes-across as arbitrary) to having some sinister pattern (which fits the theme of the overall campaign).
Then, I'd think about what caused her to fall? Did she actually make a pact with a lost deity (and why)? Did a path of study reveal to her some horrific truth that shattered her mind and sanity and opened her consciousness to the influence of some ancient power (and is it reversible)? Does she carry some artifact from where a long-dead god was felt so close that the very ground warped from its presence, which flows through her and twists her into something very different? Is she trapped in an endless illusion (a la Mad Hatter) where she thinks she is doing one thing while she is actually doing something else?
The point of all these steps is to make two things easier:
- Roleplaying the character
- Designing encounters
If you know the ins and outs of your major characters, you can think of how they plan to influence the world, and how they will react to the unpredictable influences of the Party. You can also begin planning a boss encounter from the very beginning, which is very important to the story as a whole. It wouldn't do, after all, to have planned your NPCs for a fight at the end only to have to change to a puzzle which would make all the magic weapons and combat-oriented information basically worthless.
One last thing I'd have you consider is that this works for entire towns and countries, or even unpopulated areas, as well. If you run a campaign for Hillsfar, for example, you can expect the people to be distrustful of or even outright hostile towards non-Humans. That is going to be a major plot-point for anyone who exists in Hillsfar or its surrounding area.
Meanwhile, even a wasteland can have personality. If your wasteland was the site where an ancient and powerful demon fell, your PCs might see demonic specters at night, or find that the place has an unnatural connection to evil and whispers into their heads, or they might look up at the sky at night to see, instead of their night sky, the fire and smoke of the Abyss.