While reading up on the Bog Nixie (Bestiary 3, page 201), I noticed it mentioned that it had the harpy's Captivating Song DC and that the save DC was "Charisma-based."
My question, then, is how do I calculate the save DC of a monster's abilities?
The harpy's save DC, for example, is 16. With 17 Charisma, it should have a +3 Charisma bonus. My best guess, then, is that the formula is DC = 10 + 2×(Stat Modifier).
Glad you could reverse-engineer the save DC calculation. It is actually described in the SRD - and in several other places. Kept that one as an example.
Step 8: Special Abilities and Qualities
Most special abilities that cause damage, such as breath weapons, give a save (Fortitude, Reflex, or Will depending on the ability). The DC for almost all special abilities is equal to 10 + 1/2 the creature’s Hit Dice + a relevant ability modifier (usually Constitution or Charisma depending on the ability). Special abilities that add to melee and ranged attacks generally do not allow a save, as they rely on the attacks hitting to be useful.
So a special attack that doesn't already rely on a hit roll will have a save DC. Some special attacks that are particularly devastating might have both hit rolls and saving throw, but those are rare.
As Geoffrey points out in his answer, some spell-like abilities that emulate published spells don't use the 1/2 HD to calculate the save DC, instead using the spell level of the emulated spell for the DC calculation. Some unique (Sp) abilities still use 1/2 HD though.
Finally, Supernatural (SU) abilities ignore spell resistance, but spell-like (SP) abilities usually need a caster level (almost aways equal to HD, unless otherwise specified) to beat SR.
...Actually, I think I found it after examining the hydra.
The DC formula is DC = 10 + ((Hit Dice)/2) + Stat Modifier. 10 + (7/2, rounded down = 3) + 3 = 16. That's how it's calculated.
So, the Bog Nixie, being a Nixie with the Advanced creature template, would be calculated as DC = 10 + (2/2 = 1) + 6 = 17.
...I recently read something about how answering your own questions is encouraged, so I'll leave this here in case someone else is confused.
The rules for save DCs are given in Appendix 3 of the Bestiary. They vary depending on the type of ability; I posted a summary here a couple of months back.
Most do follow DC = 10 + HD/2 + stat modifier, but spell-like abilities are an exception (they use the level of the spell that's being emulated, similar to how regular spell DCs are calculated) and while most abilities are specifically based on racial HD (i.e. not class levels), a few don't include that qualifier.
