You've only talked about retaliatory responses to players who step out of line. Have you tried a group conversation before the game, so everyone can set parameters and know what they are? That works a lot better than punishing them for crossing lines they didn't know existed.
This is especially important with horror games. Suspense, dread, terror, and the other emotions which horror games thrive on, are delicate to construct and easy to tear down. ALL games require everyone to collaborate if they're going to work, but horror games are less robust in their ability to absorb social contract miscues.
Pre-game flight check: Make sure everyone's playing the same game before you start.
Before a group starts playing a horror campaign, they need to sit down and talk about what it means to be playing that sort of game. Some horror RPG systems, like Call of Cthulhu, have sections about the implications and necessities of horror gaming. There are also essays on the subject which you may find helpful.
Some of your problems may stem from a disconnect in the kind of horror your group is interested in. A gore-heavy story will reward study of the monsters' statistics because it's dismemberment which brings the chills; a dread-heavy story can be utterly ruined by familiarity with the beasts stalking in the shadows. If some participants expect one style and some expect another, the tone will be pulled in multiple directions at once and probably won't survive.
It's important (for reasons I'll get into a little bit later) that this pre-game meeting doesn't just consist of the GM telling the players how it'll be. Good horror stems from things the participants find unsettling, and also requires all the participants to buy into the kind of horror they're agreeing to experience. This means the group should work together to agree on what the game will be like, so it's good for everyone. Your goal should be that everyone at the table, GM and player alike, becomes a co-conspirator working to make the game great for all involved.
Use the right tools: System choice can help or hinder your gamegoals.
Once your group's goals are agreed upon and coherently expressed, you may want to experiment with various systems to find ones which encourage those goals. For example:
A d20 System campaign (which gives hit point and defensive statistics to all creatures) comes pre-packaged with the idea that defeating an enemy is always possible given the right circumstances. That's going to actively oppose any attempt to cultivate "cosmic despair."
Cthulhu Dark, on the other hand, says that if you attempt to fight any monster, you automatically die. This totally re-frames the Investigator's position in the world, and reinforces the implacable nature of the horrors the Investigator faces. (Naturally, Cthulhu Dark is an awful game for zombies-and-shotguns gorefests.)
Make parties that work together.
When players create characters together, they act as a force multiplier to push things in the direction the group wants to go, while also reining in ideas that are counter to the group's needs.
You'll find a number of questions on the site already dealing with the details of party cohesion, including What kinds of rules promote good team cohesion?. It's a great way to help avoid issues like "My character isn't interested in that," "My character fixes the problem before anyone else can help," and "My character has a worldview totally at odds with the sort of story we're telling."
Some systems have this concept built into their mechanics already, but you can add it to most systems with trivial effort.
Don't force horror on the unwilling.
Most important of all: you have to be sure everyone wants to play the game. If some people are uncomfortable with the subject material, or just bored, they shouldn't feel pressured into doing it anyway. That's the fastest way to get players who disrupt the proceedings. They don't usually do it on purpose, but they will fight the tone of the game every step of the way. If you have players who don't want to play horror, or just don't want to play one particular flavour of horror, don't push them into it.
Take breaks from being scared, and guide action toward productive channels.
Don't go overboard on enforcing the horror experience all the time. A little levity is important, as it gives us a chance to breathe and recover so we can be scared again. Without breaks in the terror, we get overwhelmed and stop being scared. System mastery is laudable, as I'm sure you don't want players who are unfamiliar with the game they're playing. Studying the books means they're interested in what's going on. If you punish these behaviours all the time, they'll never find appropriate outlets which enhance the game experience.
