I'm running a fairly sandbox game, where the party can wander around easily from place to place. When the party visits areas I've prepared something for, they generally have a good time. When they go off-map, we run into trouble.
How do I keep the party within the confines of the map?
I don't want it to feel like railroading, so (for example) a big magical wall surrounding the kingdom is a very bad idea.
A couple more are more "corny" but can work...
Some expansion on these ideas...
If your map is big enough that crossing it is a major effort, then players won't hit the edges. For example, if the best transport is a horse for 60 miles per day (which is hell on the horses, BTW), then a map some 1200 miles on a side will tend to result in not going all the way across without some prep time, especially with the encounter and getting lost checks of most old school games.
These can either be the "here be dragons" unpleasant, or "here be the endless desert of roasting" and "Here be the endless ocean" type unpleasant. Or simply the vast-boring "endless waves of grassland"...
Noting, however, that boring is less effective than unsurvivably vicious. If the "edge of the world mountains" are backed up by the lands of the walking dead, where ghosts are present in huge numbers, then the horror factor may be lost, but players will either turn back or die.
Don't give any hooks to the outlying areas unless you've prepped those particular ones and given a bit of an overview to the neighboring edges. And, just because YOU know what's there doesn't mean your players need to.
In general, you also need a variety of prepped leads. This also works best when you have a clear idea what players want to go after. GM-player communication is a vital part of even old-school games - by knowing what kinds of adventures they want, you can put those in places where you want them to stay.
Some campaign worlds have very literal borders that simply can not be crossed. For example, in certain settings, there are areas where living beings simply can not go, and in going there, either the world turns them around, kills them, or makes them realize they can not go on. It can even be the Mythological barrier at the end of the world...
Examples in game worlds include the mountains edging the central parts of Athas. Many of the Great Wheel D&D/AD&D cosmology's planes have literal barriers at the ends - in some cases, leading to another plane, in others, simply marking the end of that plane.
Examples in literature are fewer, but not entirely non-existent - Norse Sagas include the edges of the world, and U.K. Le Guin's Earthsea runs to stormy oceans without land that lack even fish to eat (The Farthest Shore) and turn one back, whilst also trying to break up one's boat. Another, Sci-Fi, example is the barrier at the edge of the Galaxy in Star Trek.
Once one crosses some part of the border, the rules change, and the Gods punish the characters. This can be, as with AD&D's Ravenloft, that the mists simply return one whence one came from through level-drains, or even simply being struck dead for crossing the line.
One game I ran, going off-map resulted in being teleported to a particular dungeon... and the map had a literal line drawn by the Gods. Players saw the line, and decided not to push it.
The world literally simply ends at the edge of the map. Nothing past it. Characters leaping off simply float away forever.... There might be a clear barrier, or not.
Decide if the world has a reverse side... and it's thickness... because players in such a case likely will ask... and also how far off the atmosphere continues.
While this is a cosmology issue as much as a map issue, it can still be expanded by other, large, mobile chunks (each conveniently map-sized) grafting on as you develop them, and having rumors of them before they graft on as they become visible to the peoples near the edge.
To some degree, Ravenloft works this way... So do several other planes in the Great Wheel cosmology... the end is literally the end, and past it, nothing.
Fundamentally, this has, combined with a big enough map, been my best solution. Simply let your players know you would like it if they didn't go off map until you've prepared a particular direction. Also, be clear, with rumors, etc, when you're ready for them to go off-map in a given direction, but also go ahead and tell them.
I think a good question should be "why are they going off-map?". You're running a sandbox campaign, so you're generally waiting for the characters' own motivations to lead to the next adventure. These motivations can be one of several things: they can be hunger for adventure, gold or power - in which, case, you're in control, since you determine where these things lie. Why would they insist in going out to the uncharted desert, if you've dropped hints that the abandoned dwarven mines in the mountains hold mountains of gold?
Secondly, they could be pursuing their own character's plot hooks. There, again, you're in control. Are they chasing the murderers of a characters' family? Leave plot hooks that will lead them where you want.
Or are they exploring for exploring's sake? That's fun too, but then you will have to anticipate some of their moves - let's say, ask them at the end of a session what their plans are for next time, and plan ahead just enough to leave clues that will interest them enough to pull them back into the places where you have thicker plots planned.
In short, characters (and players) rarely do things randomly. They have their goals. Plan accordingly.
You have a variety of choices.
Tell the party which areas you have stuff prepared for, and let them pick from those; if you have more than 3-4 choices, this generally won't feel like rail-roading.
Come up with a story about an ancient blessing on the kingdom keeping out the more ferocious monsters. If the party wanders out of the area, hit them with enemies of a much higher level (while making it clear that they're intended to retreat). This could even be a plot hook: as the party gains levels and naturally finds more dangerous monsters to keep things level-appropriate, they could realize that the blessing is fading, and go on a quest to renew it.
Come up with a few generic adventures that can easily be modified to work in any area the party might find off the map. Then when they wander off the map, just pick a generic off the list.
Expand the map. This can be done on your own, so that you know what's beyond the current map even though the players don't, or with the players' cooperation (my group uses a slightly modified version of Dawn of Worlds to build a world/setting as a group, so everyone knows approximately what they'll find in any given part of the world).
When I run sandbox games I like to have a set of scenes prepared which are fairly flexible with regards to where they can take places and the people that would be involved. With each scene I make a note of the hooks that would allow it to be inserted in the games, like: "at sea", "in a bar", "in a city at night", "when confronting militia", and so on.
Then, if the player characters go off-plan, or even just when the game slows down because the players can't decide what to do next, I look over at my list of flexible scenes and pick something I can use in the current situation the characters find themselves in.
Part of the magic of being a referee is that the players don't know what is part of the plot and what you made up on the spot.
a) don't act fazed when they leave (or plan to leave) your pre-planned area. It ruins the illusion that the world is real, that is essential in sandbox play (IMO).
b) delay; this can be done in several ways. The best way is to get the party into a discussion about something irrelevant. Tell them, for example, that the place they are going to is well known to not have any supplies and they had better prepare, as once they are there there will be no chance to buy extra rope, etc. Some groups will spend the rest of the session poring over the equipment list. If this does not work, remind them of another plot line by having an NPC turn up with a new piece of information that needs immediate but not long term attention (else they will think you are stopping them going). And finally have something happen on the way: bandits collecting a toll on the road, etc.
c) once the session is over, then plan for what will be where they are going. Remember whatever you told them in session—they don't know you just invented it, and it does not necessary have to be true.
Collaborate. You and the players should be working together to tell a story. If they're interested in something, they should let you know far enough in advance that you can plan it out.
I've outright told my GM "that NPC that charmed us and stole our secrets was really intriguing to me. Expect my character to try to make friendly contact with it." We haven't actually gotten to play out that scenario, but I felt it was important for my GM to know that what he thought was a one time misadventure was actually something he should expect to use again.
Something that I'm going to be trying out in my next game is: there is no map, or not one the players can see. Maps - ones that are any use to travellers, that is - are a pretty recent development in this world. And if you haven't got a map, then without any railroading, travelling becomes genuinely difficult. They can't say "we travel up through this pass, and then down around the Great Swamp", because they don't know where the pass is, or how big the Great Swamp is, or indeed, if it's there at all, and isn't a myth.
An example:
The characters want to head to the West Coast. There's a mountain range in the way, though, so they have a few options - they can follow the road all the way down south, for hundreds and hundreds of miles, and go around them, or they can look for the (possibly mythical) passes, or they can propose to actually climb over them, or they can look for entrances to the old dwarven mines below.
So if they go south, that's the opportunity for lots of adventures and encounters on the way. They don't know when they've gone far enough south, unless they're travelling right up in the foothills, which raises its own problems. A single river in a gorge can add two days to the trip. Some rivers just can't easily be crossed - think of coming to the bank of the a big river like the Mississippi in springtime, say. You can barely see the far side, you've no idea how deep it is... that's stop-and-build-a-boat time. If they try to go upstream and around, well, that's back into the foothills, and in any case, another detour. If they go north, man, that's even worse - everything you have going south plus ice and snow.
The passes may well be mythical. Some passes through mountain ranges were only found after the advent of flight, so even if they're not mythical, they might be really hard to locate. And mountains are tough to travel in. Wild beasts, rockfalls, avalanches, just plain difficult terrain, not to mention a possible lack of food and water. Climbing over is next to impossible. Not actually so - that's no fun - but really, really hard. Look up some mountaineering stories from the early 20th century, even, and that's with some level of technology.
And the dwarven mines - well, that's Moria, right there. Or maybe the mines don't exist, either, just a few dead-end caves, or a rumour spread by the hillfolk equivalent of shipwreckers to bring in nice fat adventurers.
If the players can't see the map, the player characters are unlikely to go far off it, and they'll have to travel slowly enough that you'll have plenty of time to prepare for the places they do reach.
Actually having a player map can be a really good tool in this case. Have a map (either found via the magic of google, or hand-drawn to fit an existing campaign) with say, a dozen towns divided among two or three kingdoms, a few neutral towns, some mountains, caves, rivers—whatever it is you need for your campaign world. Come up with a bit of detail that everyone in the area knows about places. Come up with a scale, probably in days of travel—a month or two from edge to edge works for my group, but then again they always forget horses. Mark the party's position on the map, and at the end of every session ask them where they want to go next. Unless you've given them a plot hook that lies outside the picture they're looking at, they'll probably point somewhere on the map.
If they start to stray close to the edge of the map, ask if they're planning to go outside the borders of your little self contained world, and if they say yes, ask why. See if whatever they want can be found inside the map. If not, then make a new one for whatever is outside the borders. This gives you a bit of control over the options they have while allowing them as much freedom as their characters would have (i.e., if you don't know about a place, you can't go there).
Roleplaying Games have Players playing individual characters. During the course of the campaign, those characters will develop connections with NPCs, and locales. In some campaigns those connections can be as prized as the +2 sword found on level 4. An observant referee and design those connections to keep the character focused on the prepared area. Manipulate things so that the the rational choice would be to stay.
The caveat is that like all things involving human choice, all you do is set up the likely outcome. More often than not the unexpected thing happens and the campaign veers off in a new direction. However even this can be minimized by increasing the number of connections.
Sure the intrigue with Dame Black got resolved in an unexpected way but the ongoing responsibility to keep the Temple of the Green Fire safe from the local thugs will still keep the characters in the region.
You don't want it to feel like railroading, but it still will be.
Simply break the urge to enact an illusion and tell them 'there is no more map this way. Not yet, anyway'.
In Star Trek: TNG, the Enterprise was transported to an uncharted part of the galaxy, well off the map. There they encountered the Borg, a race many, many levels above the Humans. The Enterprise barely escaped alive. However, the Borg now had an interest in the human race and prepared their invasion
Give your players a reason to not venture too far into the unknown, and if they do, make it have consequences.
