For example, undead "use their Charisma score in place of their Constitution score when calculating ... any special ability that relies on Constitution". The special abilities section describes several types of special abilities, but does not appear to actually say what counts as a special ability. Are spells special abilities? Is the ability to cast spells a special ability?
There is no real definition, but Special Abilities are listed as such
According to this passage in d20PFSRD, there are four types of Special Abilities, four lower level headers under the "Special Abilities" header:
- Extraordinary Abilities (Ex)
- Spell-Like Abilities (Sp)
- Supernatural Abilities (Su)
- Natural Abilities (No acronym provided)
Hence, if an ability is an (Ex), (Sp), (Su), or a Natural Ability, it is a Special Ability. Perhaps some other books have other ability types that are specifically listed as Special Abilities, then they also belong to the class.
Unfortunately, there is no definition such as "Special Abilities are...": Pathfinder frequently defines an in-game term by listing all of the objects that belong to said class (if it is an object class), or saying how to calculate it (like in case of Caster Level) without giving an actual definition. It might hence be hard to understand what is what sometimes. Especially given how many terms are there that mean different things in D&D but almost the same things in real-life language.
Moreover, "Natural abilities are those not otherwise designated as extraordinary, supernatural, or spell-like" means that sometimes it can be really hard to understand that something is a special ability at all if it is a Natural ability, since there is no special marking. RAW, you have to guess if something is a Natural Ability based on context, formatting, etc. As usual in Pathfinder.
According to d20srd, the situation is different in 3.5e. While using the same or almost the same wording, it doesn't say that Natural Abilities are Special Abilities.
Natural abilities are those not otherwise designated as extraordinary, supernatural, or spell-like.
A special ability is either extraordinary, spell-like, or supernatural in nature.
Since something that is not "designated as extraordinary, supernatural, or spell-like" clearly is not "either extraordinary, spell-like, or supernatural in nature", and since nothing can belong to one of the three said classes and belong to none of them at any moment of time, Natural Abilities are never Special Abilities in 3.5e.