I'm running a Dungeons and Dragons 3.5 (not Pathfinder) game, and I want to create a system by which players can use certain abilities, but at the cost of sanity and self-control. Is there a system that models progressive, growing insanity through stat changes? There would be other effects that are specific to the story, but those could be minor tweaks to the existing system.
There are two systems which were made by WotC which I can imagine off the top of my head to model such a system.
Some places and items are so evil that exposure to them marks, or taints, a character in a very real and difficult-to-cleanse way. Taint is evil. It is a corruption so deep it warps the very plane of reality. A weapon used to slaughter thousands of innocents, a forest grown on land soaked in the blood of an evil deity, a book bound in the flesh of an archfiend for his own horrible purposes, and the presence of an evil deity are all sources of taint. Then, of course, there are rings…
Originally appeared in the d20 version of The Legend of the Five Rings to model Shadowlands taint, but can be used to simulate corruption from other sources as well.
Taint starts at 0 and increases by failing Fort saves after being exposed to a Taint source. Each such failed save gives the character one Taint point. Each Taint point is a -1 malus to the character's Constitution and Wisdom scores and consequences for accumulating Taint points range from "Mild paranoia", to "Sores ooze blood, pus, foul-smelling ooze, spiders or insects, thick pasty substance, maggots, or acidic green slime" and finally even "Skin peels off in papery sloughs at the slightest touch, leaving raw, red flesh beneath".
This variant system allows you to introduce an element of dark horror into your d20 game. In campaigns using these rules, characters gain a new attribute called Sanity. This statistic functions like an ability score in some ways, but it has its own unique mechanics that represent the character’s descent from a stable and healthy mental state into confusion, dementia, and mental instability. As a character encounters monsters, witnesses horrible acts, masters forbidden knowledge, or casts spells, his Sanity score, and his corresponding ability to function as a normal member of his race, deteriorates. This gradual descent is balanced in part by the powers that characters gain each time they overcome a horrific foe or grow in skill and expertise, but even as those characters grow in power, they know or fear that an even greater peril lies ahead—the threat of becoming permanently insane.
Originally appeared in the d20 Call of Cthulhu and simulates the growing madness eating away at a character's mind as he witnesses unspeakable events and creatures.
Sanity starts at Wisdom score * 5 and each time a horrible, gruesome or outright weird (on a cosmically scale) scene/creature is encountered the player makes a d% check. If the result is less than the character's Sanity score the check is successful, otherwise the check failed. On a successful check the character typically loses no Sanity, but truely horrible things can even erode the mind of a character even on a successful check.
Alternately I believe if you take a look at the Ravenloft supplements by Swords & Sorcery Studios for 3.0/3.5 that this also included an insanity system.
Yes there is: It was published in Kobold Quarterly #11 for 4e, and a web supplement expanded it to 3.5/Pathfinder (The system is close enough it can be used for either with ease): www.koboldquarterly.com/k/front-page2525.php
I wonder if it would be simpler to model it as a temporary reduction in Intelligence, Wisdom, and Charisma that might, depending on the severity and long-term viability of the characters, turn into permanent loss, such as when an ability hits 0 or debilitation lasts longer than a month. This is inspired by touch of idiocy and the Madness domain (3.5, Pathfinder).
Relatedly, Pathfinder's Game Mastery Guide includes a Sanity and Madness subsystem that could easily be backported to 3.5.
I remember of the prestige class "Alienist" in the Complete Arcane. One would gain power and knowledge from the alien beings of the Far Realms (FR settings) at the cost of sanity (penalty to diplomacy, handle animals and bluff checks).