Look at my problem. My party (avg 10 lvl) just had a pre boss fight. That was an opportunity for me to make them spend some of their resources, because they thought that was actually the big bad boss fight. So we finished the session with them defeating the big bad guy's first mate and they are at the entrancce of the cave, in which resides the boss. The boss will be a wizard about 14-15 level. I have heard them planning that they will try to kill him when they meet him, but the actual NPC has heard them for some time now, planning for the event (through a successful series of scrying spells). So their plan is kill or flee. Through teleportation actually. My NPC wish them either dead or them surrending to him an artifact they are carrying. So, imagine a wizard of vast intelligence hearing that his enemies have a backup plan of leaving through teleportation. Is there a way, (spell, magic item or something other) that the npc could use in his cave, to prevent them from teleporting when things go awry?
A fair number of spells prevent teleportation; a wizard will struggle to gain access to some of them, but the skill Use Magic Device and a wand or staff will solve that.
If the wizard doesn't have to use teleportation effects himself to escape he should probably employ the spell dimensional lock. If he does have to use teleportation effects himself, he can use the feat Chain Spell (CAr 76) to modify the spell dimensional anchor.
The best magic items are scrolls, staffs, and wands of the above spells, but specific magic items are also useful.
A Few Notes about Teleportation
Also, the spell teleport [conj] (PH 293) explicitly says, "Areas of strong physical or magical energy may make teleportation more hazardous or even impossible." In that quotation the word teleportation is not italicized--that's teleportation in general. The DM can just say no to teleportation effects in any area and claim, "It's strong magical energy prevents teleportation." The wizard might've chosen the cave in the question for just that reason, although that may seem contrived to the players.
Players who go Dumpster-diving for travel solutions will eventually find the 7th-level Drd spell master earth [trans] (SpC 139), a transmutation spell that sends the caster anywhere on planet by shoving him (harmlessly, obviously) through the planet. I've house ruled this spell so that it cannot penetrate creature-built structures and instead moves the caster to their most common entrances. (The spell itself says, "Underground, creature-built structures don't hinder [the caster] as long as [the caster] can move around them," and I've expanded that to mean the caster can't enter any creature-built structure using the spell master earth.) Further, they might even find the never-errata-ed and never reprinted 9th-level Drd spell lookingglass [trans] (MW 90), whose effects I urge house ruling to actual clairvoyance and to actual teleport without error (i.e. greater teleport) instead of as if those spells. With the existence of these spells in the campaign and without house rules, high-level druids are the ultimate assassins.
Finally, a spell clock (Clockwork Wonders column "The Spell Clock" here) (130,000 gp; 0 lbs.) crafted at a higher level by some ancient epic precursor can have been toted around the entire world to create a worldwide forbiddance effect if you want a way to explain why teleportation effects don't work in your campaign at all without resorting to DM fiat.
Depending on how many can cast teleportation, all the party need to be touching to teleport away.
One suggestion would be to have him divide the party as they enter the cave preventing them all from being within touch range of each other.
An NPC with vast resources should cover (almost) all of his warded area with Forbiddance. (Cleric 6). We can assume the wizard either has a cleric on staff or can hire one (either through services or money), and nothing says "go away" than random enemies taking 12d6 damage every couple hundred feet.
Forbiddance's important sentence reads:
As a note, "planar travel within" could be read as "instigated within" or "both start and end co-located within." If it's the latter, you'll want him to just armor up the cave with dimensional locks.
Everything else, (dimensional lock, anticipate teleportation, refusal) either requires renewing or works on incoming teleports rather than outgoing teleports.
Unhallow combined with dimensional anchor may work well as a medium term solution.
Still, be careful with this tactic. He can lock himself in with the party (though leaving a few one-way closets of "bye bye now" outside of the warded space is always a good idea.
The wizard's cave could be a demiplane. With the Teleport spell, interplanar travel is impossible. This leaves the possibility to Plane Shift, but I don't suppose 10th level characters regularly prepare the spell, especially that it requires specific focus, not part of the components pouch. Also, I'd let the players know they enter another plane.
For Dimensional Anchor you don't need to hit all the party members. Let them decide if they want to leave some of their companions at the evil wizard's mercy...
Dimensional Lock is great (even better with Forcecage), but the area covered is actually quite small (20-ft. radius).
I'd generally vote against any solutions based on this sentence Areas of strong physical or magical energy may make teleportation more hazardous or even impossible, unless this is common in your world or somehow justified within the adventure.
The hallow/unhallow spell comes with a one-year lasting effect the caster needs to choose from a list. Dimensional anchor is amongst the options, allowing the caster to block teleportation of everyone that's not of his own faith, as long as they stay inside the hallow/unhallow radius. Add physical barriers to avoid people just walking out of the area and you have a very useful "nobody (else) can teleport out" trap.
Some other options like Otiluke's suppressing sphere, investiture of the orthon or a selective anti-magic field have a very small range and are not reliable for preventing teleportation.
Deus Ex Machina
He simply can. You as the GM can make up anything you want if it helps advance your plot. If you want him to be able to block teleportation, then he can. Perhaps the chamber he is in permanently blocks teleportation in and out due to some ancient lost magic cast on it centuries ago. You let your imagination run wild here.
3.5 RAW Spell
The only spell I could find that would allow your NPC to do this on his own is a lvl 4 spell called Dimensional Anchor. Any other spells were either too high for his level, or weren't for a wizard.
Dimensional Anchor: This seems the better option for a more ranged encounter. It will allow you to lock the party in at longer range than Anti-Teleport Sphere, but you do need to hit them with it and you need to hit each party member, one per cast. You also have to overcome SR. It lasts minutes per level, which should be enough for any encounter, certainly enough to get them in range of Anti-Teleport Sphere. The big downside to this though is the number of rounds you need to spend to lock the entire party down, 1 per party member, although in theory you just need to lock down their casters, as without the casters no one is teleporting away.
Anti-Teleport Sphere: From reading it seems more like a defensive spell to stop people dropping in on the wizard, but there's no reason why he couldn't cast it to stop them escaping. It does have a limited area of effect though. At lvl 14 it'll only affect 140' radius, or 280' diameter sphere, so you need to keep them close. But with it lasting a day for 5 minutes prep, it seems like a spell I would cast in his place. It offers the party no saves either.
Null Teleport Zone: I takes a day to cast for the area you need, but as you have the time to prepare that isn't an issue. It prevents all types of teleportation. It appears to be the wizards version of Forbiddance.
It is reasonable to expect such a powerful boss to have minions capable of casting spells for you. I've previously mentioned Forbiddance, and having a cleric minion cast that for the boss isn't infeasible. Forbiddance would permanently stop the players from teleporting in or out of the area. Although as @Zachiel has pointed out when commenting to @BrianBallsun-Stanton, who also mentioned this option, the spell specifically states:
It doesn't mention anything against teleporting out, so is open to interpretation. Personally I interprate that to mean you cannot cast teleport whilst in the area.
The Castle Guide, one of the AD&Dv2 sourcebooks, discussed preventing travel spells using exotic materials. For example (off the top of my head since I can't remember the actual example), you could block teleportation into or out of a castle by using basilisk blood in the mortar.
I'm not aware of any RAW for this in the later editions of D&D however I think something like this would be an acceptable house-rule.
In the case of the campaign in @Drunken_Guy's question, the PCs could find themselves unexpectedly blocked from teleporting, which would be a shock. However if they determine how the bad guy blocked them then they have an extremely saleable piece of secret information.
something my gm just used on my party when we teleported in to check on a town aflame was a teleport trap in the area, instead of landing where we were aiming for we ended up underground deep with in the town itself in big metal boxes. you could do something similar where your players end up in another area of the room etc