Three basic techniques come to mind:
- keep a "Big Enough" map
- keep the edges really unpleasant
- keep the central areas really interesting
A couple more are more "corny" but can work...
- a literal barrier at the edges
- Wrath of the Gods at the edges
- End of the world at the edges
- Have your players agree not to go off the map
Some expansion on these ideas...
A Big Enough Map
If your map is big enough that crossing it is a major effort, then players won't hit the edges. For example, if the best transport is a horse for 60 miles per day (which is hell on the horses, BTW), then a map some 1200 miles on a side will tend to result in not going all the way across without some prep time, especially with the encounter and getting lost checks of most old school games.
These can either be the "here be dragons" unpleasant, or "here be the endless desert of roasting" and "Here be the endless ocean" type unpleasant. Or simply the vast-boring "endless waves of grassland"...
Noting, however, that boring is less effective than unsurvivably vicious. If the "edge of the world mountains" are backed up by the lands of the walking dead, where ghosts are present in huge numbers, then the horror factor may be lost, but players will either turn back or die.
Interesting Central Area
Don't give any hooks to the outlying areas unless you've prepped those particular ones and given a bit of an overview to the neighboring edges. And, just because YOU know what's there doesn't mean your players need to.
In general, you also need a variety of prepped leads. This also works best when you have a clear idea what players want to go after. GM-player communication is a vital part of even old-school games - by knowing what kinds of adventures they want, you can put those in places where you want them to stay.
Some campaign worlds have very literal borders that simply can not be crossed. For example, in certain settings, there are areas where living beings simply can not go, and in going there, either the world turns them around, kills them, or makes them realize they can not go on. It can even be the Mythological barrier at the end of the world...
Examples in game worlds include the mountains edging the central parts of Athas. Many of the Great Wheel D&D/AD&D cosmology's planes have literal barriers at the ends - in some cases, leading to another plane, in others, simply marking the end of that plane.
Examples in literature are fewer, but not entirely non-existent - Norse Sagas include the edges of the world, and U.K. Le Guin's Earthsea runs to stormy oceans without land that lack even fish to eat (The Farthest Shore) and turn one back, whilst also trying to break up one's boat. Another, Sci-Fi, example is the barrier at the edge of the Galaxy in Star Trek.
Wrath of the Gods
Once one crosses some part of the border, the rules change, and the Gods punish the characters. This can be, as with AD&D's Ravenloft, that the mists simply return one whence one came from through level-drains, or even simply being struck dead for crossing the line.
One game I ran, going off-map resulted in being teleported to a particular dungeon... and the map had a literal line drawn by the Gods. Players saw the line, and decided not to push it.
World Ends at the Edges
The world literally simply ends at the edge of the map. Nothing past it. Characters leaping off simply float away forever.... There might be a clear barrier, or not.
Decide if the world has a reverse side... and it's thickness... because players in such a case likely will ask... and also how far off the atmosphere continues.
While this is a cosmology issue as much as a map issue, it can still be expanded by other, large, mobile chunks (each conveniently map-sized) grafting on as you develop them, and having rumors of them before they graft on as they become visible to the peoples near the edge.
To some degree, Ravenloft works this way... So do several other planes in the Great Wheel cosmology... the end is literally the end, and past it, nothing.
Have Them Agree to Not Go There
Fundamentally, this has, combined with a big enough map, been my best solution. Simply let your players know you would like it if they didn't go off map until you've prepared a particular direction. Also, be clear, with rumors, etc, when you're ready for them to go off-map in a given direction, but also go ahead and tell them.