In a horror themed campaign with a Lovecraftian feeling, a big part of the game is a constant feeling of threat, but without any visual stimuli other than glimpses, or sounds vaguely heard of things seething in darkness, and with investigation a big part of the game, contrary to fighting.

Having read Heroes of Horror, I am searching for ideas on how to actually motivate players to assume the role of investigator rather than executioner, and how to instil the horror of the unknown, of something that threatens but is not substantial, that cannot be seen or heard, but is there, and has malevolent intent.

I am thinking of D&D 3.5 as a system, but I am interested in answers that cover any system (Call of Cthulhu is unfortunately out of the question), more as role-playing and storytelling techniques rather than mechanical ones.