As many other people have noted, it really depends on your group and the game you're playing. Some people enjoy games that are railroaded - they're playing to get a story told to them, and maybe make a few choices in a fight scene. Some people want to be able to do anything that's appropriate within the game genre/setting, and do not want a story dictated to them, at all.
More often than not, wanting different things along these lines, or misunderstanding what you should be doing or getting from a given game, leads to big problems in play.
Being very clear as a group what you're trying to do and how you do it, solves a LOT of issues that people crop up as problems in their gaming.
(side note: yes, you can in fact have great game sessions doing nothing but improvisation with no planning. There's a lot of games well suited to do this: 1001 Nights, Inspectres, Primetime Adventures, Breaking the Ice, Universalis, etc. There's a lot of other games that easily can do that, as well.)
Tools to use
A lot of time gets wasted with players feeling lost about what to do next. When you have a game that has a core mission structure, it becomes a lot easier. Inspectres is about ghost-busting, Mouse Guard is about clear missions given to you by the leaders to carry out. When you have a goal, it becomes a lot easier for players to be proactive and for the GM to know how to feed direction to them.
Flags are clear mechanics for players to tell you what they want the story/conflicts to be about. Burning Wheel uses Beliefs, Primetime Adventures uses Issues. Instead of having to plan what the "problem" is for each scene, you simply look at the general Flag and think of ways to make scenes around it.
"Korsak has 'Loyalty to the King' as an issue. What happens if he finds out the King ordered his brother's execution?"
You simply take a situation, the NPCs you've got, and the player's Flags and build around that. Half the time it's simply NPCs acting and reacting to the PC's actions.
Some things simply provide ongoing conflict easily - Dogs in the Vineyard's Towns and Apocalypse World's Fronts both do this. They're situations cause many types of problems, and you don't have to prep each scene or encounter, you just make some notes before play and during play, you can easily improvise problems during play.
"You're on a starship that's falling apart, and there's space monsters. It's not hard to figure every scene is either hazards or monsters or both."
Scene framing is the trick nearly every game does better with. You know how a movie will cut from the end of one scene, and then the next scene is the character driving up to a new location, with anything from minutes, hours, days, months cut out in between?
Do that. Cut to the part that's interesting. You can do this soft, or hard. Soft scene framing involves asking the player "Where are you going next? What are you generally doing?" and framing on that.
Hard scene framing puts them into the middle of the action, even if it's not when/where they want it to be. You have to have some kind of trust and knowledge as a group to do this - it is a style that doesn't work with players who want their characters to be "safe" all the time, or who have had abusive play situations in the past.
"It's the next day, you're sprawled on the training ground, and your rival Mikka is standing over you, his fist dripping blood, your blood. He's decided to take the sparring a bit too seriously. What do you do?"
It can also be putting people into positive/opportunity situations as well, though:
"You lower yourself into the office through the open window. You hear footsteps and hide yourself behind the filing cabinets quickly before the Don comes in. What surprises you is the voice you hear with him. It's the police chief."