I searched the player's handbook a while ago for the sleep deprivation rules but I don't remember finding anything, even though I recall having been told about such rules. Apart from asking if any one can cite them, is there anything official on insomniac characters, so it will not be just roleplaying, but actually having an effect on the game?
The thing to remember about sleep deprivation is that it doesn't take long before it really starts messing you up. It shortens your attention span, makes you extremely irritable, makes it difficult to switch focus, and draws out both physical and mental reaction times.
There is a huge body of research on the subject of sleep deprivation because the military has an intense interest in it. This 1994 study states: "The ability to do useful mental work declines by 25 percent for every successive 24 h that an individual is awake." One way to simulate this might be to simply reduce the odds of success by 25% for every 24 hours without rest for any task that involves mental acuity.
It also states: " In contrast, simple psychomotor performance and physical strength and enduendurance are unaffected. For example, a soldier can shoot as tight a cluster of rounds at a fixed target after 90 h without sleep as he or she can when well rested, but if he or she has to shoot at targets that pop up at random at random locations on a firing range, then his or her performance drops to below 10 percent of baseline (Haslam and Abraham, 1987)."
From personal experience I can say that it's not the physical act that is difficult, it's responding to the change in circumstance that becomes radically impaired after long periods of sleeplessness. So you might have only a gradual decline in the odds of success for tasks that are mostly or purely physical.
Another factor you may want to consider (depending on how severe the insomnia is and whether it is connected to some specific illness) is that even the most insomniac characters will probably eventually need to sleep. The sleep will come, whether the character wants it to or not. After being up for 72 hours straight, an officer fell asleep and didn't wake up when a company-sized live fire exercise started. His body told him to sleep, and he could do nothing about it. Giant savage aliens could have attacked and he would have slept while they munched on him.
Perhaps after a certain period of time (36 hours, perhaps), you could start requiring saves to avoid falling asleep at inopportune moments, with a sliding scale of difficulty as the hours march on.
While the idea of a character with non-normal disabilities in a D&D game has always appealed to me, it suffers from some practical problems: you will be severely unbalanced versus the other players. While choosing a Tier I class and adding disadvantages to it is an excellent way to reduce capabilities (I personally favour the sleep addicted wizard) it is a poor fit for most heroic fantasy tropes.
Looking in fiction, the best example I can come up with for fantasy genre with hampered hero in this vein is the Thraxas series. The noir-fantasy mix lends itself to a hero with many weaknesses. (In his case, drug addiction, but the same general guidelines apply.) It absolutely works in setting, but would be very hard to pull off outside a game that was accomidating.
Mechanically, you want to look at the fatigued and exhausted conditions. Failing to get sufficient rest should start looking a lot like the starvation and thirst rules. (Not pretty, but trivially circumventable by magic). While starvation requires "create food and drink" ... insomnia requires Sleep. A much easier spell to cast. If you make sure your party is willing to deal with this, I would recommend a will or fort (depending on the source of insomnia) save to sleep through the night, with fatigue occurring on a restless night. If you are then exhausted after every encounter, it would illustrate an increasing desperation (which will be mirrored perfectly by the frustration of the rest of he party.)
On the other hand, if the entire party is willing to embrace the Noir tropes, fatigue, exhaustion, and ability damage are an excellent way to model PTSD, substance abuse, and insomnia.
I don't think there is anything in the core 3.5 rules. There are some 3.5 compatible rules for rest in the Pathfinder SRD and there's a spell from Rite Publishing called Insomnia (and a lesser version called Restlessness) that causes insomnia that you could certainly use to model it.
Characters who do not get a full night’s sleep may suffer the effects of fatigue. If a PC does not get at least 6 hours of sleep, she must make a DC 15 Fortitude save or be fatigued and take a –1 penalty on all other checks and saving throws against sleep effects. A second night without sleep requires another DC 15 Fortitude save. A failed save results in the character becoming exhausted and the penalties increasing to –2. A third failed save on the next night increases the penalties to –3.
If someone's an insomniac, I'd have them make a Will save to be able to go to sleep (the cumulative penalties on sleep effects would help a little night to night). How bad the save should be kinda depends on why they're having their character be an insomniac...