How do a dragon's Frightful Presence and Becoming More Fearful interact?

Question

Scenario

A dragon attacks, which activates its frightful presence. Everyone in the party has at least 5 HD, but fewer HD than the dragon. At least one party member is affected, and they are shaken.

The following round, the dragon breathes fire and activates their frightful presence a second time. What happens?

Relevant rules

Frightful Presence (Ex)

This special quality makes a creature’s very presence unsettling to foes. Activating this ability is a free action that is usually part of an attack or charge. Opponents within range who witness the action may become frightened or shaken. The range is usually 30 feet, and the duration is usually 5d6 rounds. This ability affects only opponents with fewer Hit Dice than the creature has. An opponent can resist the effects with a successful Will save (DC 10 + 1/2 the frightful creature’s racial HD + the frightful creature’s Cha modifier; the exact DC is given in the creature’s descriptive text). An opponent that succeeds on the saving throw is immune to that same creature’s frightful presence for 24 hours. On a failed save, the opponent is shaken, or panicked if it has 4 Hit Dice or fewer. Frightful presence is a mind-affecting fear effect.

Format: frightful presence (60 ft., DC 21); Location: Aura.

Possibly Relevant Rules

Fear (Su or Sp)

...

Becoming Even More Fearful: Fear effects are cumulative. A shaken character who is made shaken again becomes frightened, and a shaken character who is made frightened becomes panicked instead. A frightened character who is made shaken or frightened becomes panicked instead.

Other considerations

Frightful Presence is Extraordinary while Fear is Supernatural or Spell-like. Does this mean that Frightful Presence produces a supernatural fear effect? Or is the fear generated by Frightful Presence unrelated to the supernatural or spell-like Fear effect?

There is a rule for magic effects that in general, effects with the same name tend to only affect duration rather that change the actual effect. So, could the fear generated from Frightful Presence stack with itself (the cumulative effect mentioned in the possibly relevant rule above) to take a character from Shaken to Frightened? Or would that run afoul of the rules about stacking effects?

That rule, however, is listed under the more general heading of Combining Magic Effects, and it specifically mentions spells, so it is unclear if it is at all relevant since Frightful Presence is Extraordinary:

Possibly Irrelevant Rules

Same Effect with Differing Results

The same spell can sometimes produce varying effects if applied to the same recipient more than once. Usually the last spell in the series trumps the others. None of the previous spells are actually removed or dispelled, but their effects become irrelevant while the final spell in the series lasts.

Answer 1

The characters become frightened

Frightful Presence is causing the shaken, frightened or panicked conditions which are the three increasing levels of fear. These conditions in their text state they all are less or more intense states of fear, but provide no rules how to progress from one to the other.

Frigthful Presence (Ex) is not the Fear (Su or Sp) ability however, so why would Fear (Su or Sp) apply here? Fear (Su or Sp) says:

Spells, magic items, and certain monsters can affect characters with fear. In most cases, the character makes a Will saving throw to resist this effect, and a failed roll means that the character is shaken, frightened, or panicked.

This fits Frightful Presence. Fear (Su or Sp) is the only place in the core rules that talks about how fear levels work, and the ability contains a general discussion of fear effects and Becoming Ever More Fearful, before going into details about Fear Auras (Su), Cones (Sp) and Rays (Su). Frightful Presence also states it is a fear effect. It seems leaving out (Ex) in the list of things that can cause fear effects is an oversight, and the aim is to have these rules explain in general how fear effects work.

There is no direct way to become frigthened by Frightful Presence. Creatures either they have 4 or fewer HD, in which case they become panicked, or they have more in which case they become shaken. For Frightful Presence to make creatures frightened as written in its descriptions and to do what it says it does, it must be possible to increase the level of fear, and so the rules for increasing fear levels given in Fear (Su or Sp) must apply.

All characters that succeeded on their saving throw the first time around will be immune to the second try, so we'll ignore them. All characters that failed their saving throw the first time around are shaken, and if they fail again they will become frightened, and if they fail a third time, panicked.

This is a specific rule for fear effects, and in this case the effect is not caused by a spell, so the rule about combining spell effects does not apply here.

Other rules systems

There are alternative optional rules for handling Fear under Other Rules Systeme that expand this to 7 levels of fear, grouped into three lesser increasing levels (spooked, shaken, and scared) and four greater levels (frightened, panicked, terrified, and horrified):

If you are subject to a fear effect of a level equal to or lower than your current fear level, your fear level usually increases by one. However, multiple lesser fear effects can’t force you to progress from a lesser fear level to a greater one. If you are scared and are subject to an additional lesser fear effect, you are staggered for 1 round, rather than becoming frightened. You can, however, accept the frightened condition rather than be staggered while scared if you prefer (such as if you actually want to run away).

So, under this system, if the shaken charactes failed a second save, they would first become scared, and if they faile a third time, they by default would become staggered for one round instead, although they can choose to become frightened. If they became frightened and were affected one more time, they'd become panicked.

Rules Compendium

If you are open to consider the D&D 3.5 Rules Compendium as a valid source of rules, you can find this rule on p. 53:

ESCALATING FEAR
Fear effects are cumulative. A shaken creature that is made shaken again becomes frightened instead, and a shaken creature that is made frightened becomes panicked instead. A frightened creature that is made shaken or frightened becomes panicked instead. Multiple exposures to the same effect don’t trigger this escalation of fear. Exposure to different effects does.

So, while the progression would be the same here, since Frightful Presence is the same effect, it would not do anything under these rules. (Those who saved are immune, those who did not cannot be affected again). This still would mean you never can become frightened, and as Frightful Presence tells you it can make you frigthened, I would likely not include this extra ruleset.

Answer 2

There is probably a typo

As you quoted:

[...] Opponents within range who witness the action may become frightened or shaken. [...] On a failed save, the opponent is shaken, or panicked if it has 4 Hit Dice or fewer.[...]

At it is, this doesn't really make sense. If the intent was the effect stacking with itself, worsening each turn, then it wouldn't say "may become frightened or shaken" but rather "may become frightened or worse", to include both the possibility of failing more than twice and the case where the target has less 4HD.

Two ways to correct it

Either we assume that the intent was:

[...] Opponents within range who witness the action may become panicked or shaken. [...] On a failed save, the opponent is shaken, or panicked if it has 4 Hit Dice or fewer.[...]

or

[...] Opponents within range who witness the action may become frightened or shaken. [...] On a failed save, the opponent is shaken, or frightened if it has 4 Hit Dice or fewer.[...]

It doesn't solve yet our stacking-or-not question, but this step has to be taken if we want to get anything out of that. The first one is more consistent with other abilities that refer to Frightful Presence, though (like Mythic Presence), so let's assume we understand it like that.

Now, what?

As far as I know, the "same effect can't stack with itself" rule only concerns magic effects, and not even all of them. Frightful presence isn't a magic effect, isn't about a modifier to a roll, nor is it any other type of interaction we can see here, thus there is no reason to think the effect would not be able to stack with itself, increasing the level of fear every turn someone fails their saving throw.

Conclusion

The shaken PC who failed again becomes frightened, and will become panicked next turn if they are still in the area and fail one more time.