So far as I'm aware, there are no guidelines published by Wizards of the Coast for a PC to create a special ability, and, while guidelines for PC-created spells and magic items exist, those are only guidelines, the results of using them still subject to DM oversight. (So you know, I also don't know of a way to avoid needing the DM approval for anything a player wants to add to a campaign.) However, below's an example of special ability creation from material beyond that published by Wizards of the Coast. (Although beyond this question's scope, I suspect different guidelines can be found if one's willing plunge into third-party d20 System material.)
Alderac Entertainment Group's Way of the Daimyo (Oct. 2004)—part of that company's officially-licensed-by-Wizards-of-the-Coast Oriental Adventures product line—on D20 System Technique Creation (70) says (along with the typical advice to compare it to other feats and a warning against power creep) that a creature that wants to create a new feat must take 1 year to create that new feat. Each day during this time the creator must take 4 hours practicing the feat. After this time, the creature gains the feat.
Similarly, Way of the Daimyo has more precise—yet still vague—guidelines that enable a PC to create his own kata, a special ability purchased with XP by those that meet the kata's prerequisites. This takes only 6 months, every day of which the new kata must be prepared twice (the longest preparation time for a kata is 120 min., so creating a strong kata will still take 4 hours of the creator's day just like creating a new feat), and at the end of which the creator pays the XP. Way of the Daimyo includes a nearly useless table (almost none of AEG's about 50 published kata fit the table's guidelines) but recommends—as always—that any new kata be compared to a similar existing kata. Kata vary wildly in power and cost, but because they're outside the existing feat structure, their availability is limited only by XP and the campaign.
These rules are specific to Rokugan, which, for example, lacks convenient access to planes with the trait flowing time. Using these rules in a campaign where time is irrelevant could see them abused.
Although the player quit before his character reached level 10, in a campaign that I ran I felt comfortable with a homebrew character class I designed receiving the following special ability:
At levels 10 and 20 the player either picks an existing regional feat or designs 1 regional feat (subject to DM approval). The character gains the regional feat as a bonus feat [in context, without needing to meet the feat's prerequisites], and the feat is added to the campaign’s list of available feats were it absent. Further, this feat is unrestricted [in context, not subject to the normal restrictions of the feat's type] for the character’s subjects, and existing creatures who meet the regional feat’s prerequisites may take the feat when able even if they’re beyond level 1.
The idea was to have the prince (the character class) shape the nation as he advanced in level, in essence guiding his subjects so they'd better withstand the rigors of campaign-specific events or granting them special abilities that would differentiate them from (and give them an edge against) their rivals. This is the kind of tack I'd always take as DM were a PC allowed to create something new for the campaign: Use what exists for guidance and get it approved by the DM.
Take a new creation to an expert
Even with the existence of literally thousands of special abilities in official material (and thousands more in unofficial material), it's possible no special ability does what a player wants. At this stage of the game's existence, there are plenty of forums (and even Q&A sites) willing to offer an opinion on a homebrew special ability for Dungeons and Dragons 3.5. No DM should give a PC the ability to create for the campaign a new feat without oversight (in much the same way no DM should allow a PC to introduce to the campaign a new spell or magic item without oversight), but if the DM's too busy to supply that oversight, pitching the special ability to a forum for evaluation is a fair middle ground.