I'm a DM that focuses heavily on story, description and setting. My goal is to send players home wanting — no, longing — to actually be part of the mythos I've created.
The ultimate goal for me as a DM is to properly instill real emotions in the people playing the game (instead of the characters). If I can invoke fear (most of my stories are horror-themed, almost Lovecraftian) in the players, I've succeeded.
What are the best ways to invoke real emotions (leaning heavily on fear) in players?
I have managed only a couple of times, but as a GM. And the first time is one of my best gaming memories, and one of the reason I keep GMing.
It was not in a Lovecraftian setting at all (Exalted actually, making me even prouder), so I won't be able to give you setting specific hints there, but the method can supposedly be applied to any game. I have found the most effective way to be based on a combination of the following factors (which rely heavily on writing methods):
Never say "it looks scary", and even less "it is scary because..". Just describe it. Describe the scary parts, but without directly saying it is scary. Describe the unnatural aspect of it, but without saying how unnatural it is. People tend to feel stronger about a conclusion they reached by themselves than something you told them. The less you say, the better it is.
"The little girl turns towards you, her eyes silently weeping blood. She lifts a hand towards you, and you can hear in the silence of the room her bones break during the process."
Two reasons for this one. First, humans can be a lot more sneaky and cruel and evil that whatever Ancient One out there. I don't need to give examples, but some people out there do really messed up stuff, they can be everywhere, they are unpredictable, they can target your family, friends, and do horrible stuff to them. Evil Monsters just want to steal your soul and torture it for eternity. Meh.
Second reason is that at the end of the day, monster are just that: monsters. They can have twenty tentacles and be in 17 dimensions and a half, just close the book and they are not there. People are real, and you will have to be with a dozen of them in your subway trip back home.
The best and worst feelings you will have in your life will be caused by someone else. People are powerful emotion-makers, use them.
This one is especially true in RPGs. Rampant fear & ambient fear are really difficult to achieve, as players will naturally adopt strategies to fight it: they will make jokes, focus on something else, stop paying attention to a while...
This isn't a bad thing. This is what scared people do in real life.
But you will have more 'satisfying' results if you use an unexpected / sudden trigger, if possible in a somewhat stressful situation, almost relaxed. You will cause a peak of fear, without any time for your players to adopt any strategy to compensate. Set up a stressful situation, and drop just one element (see Show, Don't Tell) that changes everything, making the situation very dangerous.
"Torches in hand, you arrive close from where your daughter is tied, spots of blood on her white dress. She looks unconscious. That's when the smell hits you. The floor of the pitch black room is not covered in water. Your daughter is not covered in water. Gasoline."
This point work even better when using elements established previously in the story. The torches + pitch black room above are an example, but the farther in the past the elements are, the better it is - the players need more reflexion to conclude by themselves why they should be scared, and thus will feel it more.
"The PCs are infiltrating an enemy military camp, in order to earn the trust of the enemy king. Finally, they are to meet him in a formal ceremony. It is grandiose, there are hundreds of elite soldiers there to celebrate. The enemy king finally arrives, and ask for the PCs to come and pay their respects. As they arrive in front of him, they recognize him: [drop the name of an enemy NPC they met several stories ago, who has a grudge against the PCs and know they are not soldiers]."
Use humans as often as possible. Never say that something is scary, let the players reach that conclusion themselves. Switch as fast as possible from "this is kind of bad but it will be a lot worse if" to complete silence + deer in headlights face. Laugh maniacally.
There's a saying amongst certain roleplayers: "If it has stats, we can kill it."
The accuracy of this saying varies hugely from game to game, but it's relevant to this question because it highlights a very important aspect of D&D: The more the players know about a threat's strengths and weaknesses, the easier it is to overcome, and the less scary it is. Conversely, the less players know about a threat - aside, of course, from the fact that it is a threat - the less comfortable they'll be with the idea of it stalking them.
In addition, many players will react to the first sign of a threat by preparing to fight. Which is entirely reasonable. The idea that evil can be confronted and put to the sword is part of the heroic fantasy genre, and D&D makes it a viable strategy a lot of the time... but having it be too reliable is counterproductive to most horror campaigns. It's therefore an effective tactic to present players with opponents which they can't fight - at least not yet.
One of the most fear-inspiring adventures I've subjected my players to was built around a shadow fiend. (We were using the Pathfinder rules, if anyone's interested.) Shadow fiends have a particular combination of abilities that make them great for horror: They're invisible in anything but bright light, they can create darkness, they're intangible (and therefore move silently and can't be harmed by most weapons) but can still interact physically with objects, and they can speak telepathically.
The party encountered a creature that they couldn't locate, couldn't see, and didn't know anything about - but which could close and lock doors after they passed though them, could leave great gashes in the walls, which was unharmed by sword and spell*, and which spoke into their heads with a voice no-one else could hear offering knowledge and power in exchange for performing apparently innocuous tasks.
Aside from that, remember that successful horror depends on maintaining tension. Humour tends to break tension. Try and avoid humour that breaks suspension of disbelief during periods of high tension, as it's hard to recover from.
*Technically, spells would have harmed it, but it's hard to cast at something that's invisible and inside a stone sarcophagus.
I've only had this happen once so limited exposure but here's what my DM did:
1st, Physical Environment: Lights were out with just small candle style lanterns to see character sheets, maps, etc.
2nd, Auditory Senses: CD with appropriate sound effects. Bonus: We really did have a BIG thunderstorm hit about 20 min after we started, which aligned perfectly with our setting. So, if you can control weather that would be a big help.
3rd, Get & KEEP everyone in character (no OOC talk was allowed). It seemed kinda weird but it really worked for the horror story we were in.
Actual fear is gonna be hard to come by unless you threaten to cut off a finger every time they roll a natural 1 (ala Stephen King's Quitters).
Players often get too accustomed to their gear, skills, co-operation etc. That makes them sure of themselves because they kind of know how things will go.
One of the best experiences for me, was a very interactive session in which DM didn't much do talking himself but we players created the scene.
Somehow, I don't anymore remember how, we got possessed. One player at a time was under control of an evil spirit which tried to lure us into some place in order to devour all our souls.
The DM dealt us a stack of cards before the session and he forbid us from shuffling the cards or reading them in beforehand. In those cards, there was instructions like "You are now under possession, try to lure other players into place X, don't let them know it's not you who's talking."
so only the player who was in control of this spirit knew he was "The One". Actually, we didn't know if there were more than one at a time but it was revealed later on.
Imagine how it was in some dark god forsaken place, when you couldn't actually trust what your fellow players were saying.
Every now and then, the DM asked us to pick the next card in our stacks which had next instructions.
It was really effective. He took away our trust to each other and forced us to try and play as a team. He forced us to try and find a cure which lead us into a quest unexpected. He made each of us to role play and empathize the spirit while we were in possession. This role playing revealed every time a little bit more of this evil spirit's motivations.
We talked about the experiences even during the game, but we never really could tell, if it was the real person talking or the spirit itself.
To make it even harder for us to tell what was going on, the DM gave some of us a pleasant experience with this evil spirit. So, we couldn't even agree on whether this spirit was evil or not. But the experience eventually dictated that we agreed after the game that it was really scary and we were glad we got rid of it.
To tell you the truth, you could even prolong this. At the end of a session, make sure one of the players will leave possessed but act normal. Give him some ability which the possession gives him.
Throw in a couple of sessions more and let your players think the spirit is all gone until at the worst and most unexpected moment in the middle of some hard battle, let the spirit loose once again.
And don't make it the main theme. Use it as a wild card that you throw in every once in a while to shake your players off the track when they're once again too accustomed to the game. Listen to them while they're talking to each other and creating theories about what it is they're target of, they will give you hint's for the next turn and maybe you can use some of the ideas later in the game.
I haven't played a horror adventure in dungeons-and-dragons, but I have been the storyteller for kult, which is thought of as a horror-themed system.
This is my personal success story (YMMV), but you might be able to pick up some good ideas.
I can think of few sources of fear, but they mostly fall into:
Fear of the unknown - For this, you have to rely mostly on your storytelling. A quick Google search suggests that in real life, sources of fear are often associated with death, injury, and pain. And in my personal case, the system helped a lot; because you can't heal HP with a spell: a single gun shot will kill you. My players were aware of this and acted very carefully, almost hesitant at all times.
Sense of desperation - The easiest way to accomplish this is having things go wrong. This one was quite hard to handle, because you still want players to have fun. Knowing that you're simply not powerful enough to deal with it, also helps. Knowing that you are fragile and very likely to end up dead, also helps.
Personal phobias - Use this with caution, you can't really know how deep a phobia goes. That being said, this depends on you knowing your players decently enough. But if a player has a fear for spiders, don't just rush out and pop spiders in their way. Use suspense in your favor (have some telltales, let them guess for a while).
The 2nd edition of the kult RPG had some quick tips. I can't find my book, but these are some I remember:
Things don't work as expected. You step on the brake of the car and it accelerates. Your phone seems dead when you use it but everybody else can use it fine
Twist. You spent a great night with a girl but in the morning you realize she's just a dead decaying body. You hear loud scratching on your door but nobody is outside; you later discover huge scratch marks on the inside.
You're going crazy. Nobody but you notices that the clocks' hands are actually going counter-clockwise. Nobody hears or sees what you do.
I didn't use music, but on several occasions we did play only with the dim light of candles. Does wonders.
Yes, I had to work very hard on immersing the players in the story; and that meant to plan a lot and have a plan B, C, and D at hand.
My player's PCs all lived in different places. This allowed me to play a short one-on-one with each of them ate the end of each day (in-game). If things went wrong, I could call it a dream, and move on.
Maybe @Gamer_Chick is right and you just can't induce real fear. But you might come close to it if you try hard enough.
Many things have already be proposed in other answers, let me just add some approaches that have proven effective in the past (for me as a player and for me as GM). Most of those were used in lovecraftian settings, but they should be adaptable well enough...
We had sequences where the GM would not look at his players and had turned his back on them. A GM can walk around, stand directly behind the players (sitting in a row obviously), while talking to them. We also once had a GM retreating to a corner of the room, continuing as a distant voice from the background.
Change your voice. Whisper. Then shout at your players, as if you were angry. Even though your players know that you are not really angry with them, the standing social contract is not to shout at people -- so doing it anyway puts them off balance.
Dark lights are certainly a good idea, for some sequences no lights can be even better. Pitch black darkness. This works well with the next point.
Bend the rules of the game. Ask them to roll, then continue instantly while the dice are still spinning. GMs cheat, every player expects that -- but rolling dice without even looking at the result leads to more unknown situations.
Don't leave your players enough time to think. Press for urgent instant decisions. Have something horrible happen the first time they fail to react in time.
N.B.
The above approaches can lead to a session that might be a bit discomforting but will at the same time be very intense. I suggest only doing this with people you know really well.
There's different types of fear, and it's worth noting what kinds you can do in a tabletop rpg well.
Something pops out! "Argh!" you're scared for a moment. Movies and videogames do this with the thing that suddenly pops out and makes a loud noise. You could do things to get this effect, but it's really a cheap move and doesn't really make for a good lasting effect.
Anxiety is about not knowing what is going to happen.
On the game side, you see this when a player is very invested in the game and about to make a very important win-or-lose die roll. The Dread rpg uses Jenga blocks as the trick to produce this effect.
On the fiction side, you can get this by having things work in a way you're sure is bad, but you're not sure exactly how/what it is doing. This is where describing the five senses - sound, smell, sight, taste, feeling, become very valuable. "You're not sure why, but every time you look at the glowing rock the back of your eyes hurt. Maybe you shouldn't be too close to this thing."
When suddenly you understand how it all fits together and how... wrong it is. It's when the players put together the implications of all the clues and suddenly understand and it's far worse than they expected. In a movie, it's when you, the audience, have a real understanding of how screwed the characters really are.
This is what a lot of horror roleplaying aspires to, but doesn't do a good job of producing it. Often this is because if you adhere too close to the pacing and tropes of horror novels or movies, the "big reveal" usually means either the protagonists have already failed and now die or manage to find a way to overcome the evil – and rpgs rarely work out this same way. If, for no other reason, than, the players will figure out the evil when they do which might be sooner, or later than you expect.
I've usually only managed to successfully induce this as part of long term action-adventure campaigns – the implications of what has been going on only hit the players after they've been dealing with it for some time. "Wait, so you mean this whole time the cloaks they gave us? Oh, god. I gave one to that kid. Oh god. This is bad. No."
The trick I've found is to either make the sources of horror mundane or simply odd but not necessarily threatening. The best horror is the thing that you've become accustomed to – the thing you've spent a long time in the game feeling "safe" about, and then suddenly it's not safe at all.
Create atmosphere
I had best results with what I experienced as most frightening in horror books and movies as well: Slowly building up. My PCs were students that were cleaning the late grandpas house in order to move in afterwards. A framed picture on the wall fell down when they left the room, so they hung it up again and secured it. When they came back later, it lay on the ground again, with a drop of blood. They open the door to the cellar and look down a steep staircase into the darkness. When they turn on the light it illuminates a human size body, then the lamps burn out. They develop the strange feeling, that there is something outside the house you better should avoid and then the power fails and you got to start the generator in the garden...
Step by step I created an atmosphere of "something is going wrong here", "I should not be here" and "no one could help me" and I stroke terror in their hearts. Most of them told me later they could not sleep that night.
Don't show the monster
Actually, it is pretty easy to destroy this atmosphere. My best hint for this is to show a monster and/or explain what's going on. As others pointed out here, and this is especially true for Cthulhu etc., something unknown is more frightening than a closely described Funghi from Yugghot.
Show the monster
There are situations when actually showing the monster may be quite the right thing on the other hand. I can remember two situations it worked out greatly, and both had the same initial requisites: At first sight, they were ordinary humans.
The first one was a child murderer and the long searched villain, and when they finally faced him in this, until then, pretty realistic but dark setting, he proved that he was able to levitate his knife and direct it right into them.
The other one was a youth camp counselor who started as a nice and helpful NPC, but who took the form of a Pan/Satyr/Faun when he finally revealed his dark intrugues.
The sudden shift from real world to super natural world proved to be terrifying.
First, you have to grasp your players in the story.
Next, a good intrigue and NPCs will let you introduce the Lovecraftian horror. It is all about Unnamed and Irrational Eldritch Evil. Which has no stats, no sight and kills you before you know what is it. Just read G.P.'s stories and note his words.
I strongly advise against using personal player phobias. You would be a jerk in real life for scaring people like that. Also it is the best way to lose friends.
In a recent adventure in my campaign, I was able to give most of my players nightmares after everyone went home. The specific setting was a haunted house in an area where children were going missing. The kinds of things I did were:
Music that ties into the setting: I had the ghost that was abducting children be a sound based sorceress, and she would sing "Come Little Children" from the disney movie Hocus Pocus in order to lure them to her. While ambient music is good, I personally like to tie it into the story somehow.
Break the fourth wall: At one point in the house, the characters were in a dark hallway, and were holding hands to make sure everyone was there. The most logical outcome was to add a hand into the mix. My players and I went back and forth over the number of hands that were present, and when they finally understood the gravity of their situation, they asked who was there. I had the entity reply with the name of the player that asked the question. Scared the crap out of them. It boils down to getting the players invested in the situation. If they forget that they are people sitting at a table sipping soda, it makes it that much easier to evoke a reaction from them.
An off-the-wall suggestion: wear mirrorshades. If the players cannot see your eyes then they lose the personal connection and become nervous and edgy.
