How does the Battle Dancer Rolling Flurry work underwater?

The Brawler archetype Battle Dancer has the Rolling Flurry feature which states :

When a battle dancer uses her brawler’s flurry, she must move 5 feet before each melee attack or combat maneuver. If she is unable to move 5 feet, she can’t attempt any further attacks or combat maneuvers. She can’t exceed her maximum speed. This movement does not provoke attacks of opportunity if the brawler would be able to take a 5-foot step normally; if she would be unable to (for instance, if she were in difficult terrain), the movement provokes attacks of opportunity as normal unless she succeeds at the appropriate Acrobatics checks.

In the context of an aquatic fight, I wonder if the movement resulting from the Rolling Flurry feature is subjected to the swimming rule ?

The rules about swimming state :

Make a Swim check once per round while you are in the water. Success means you may swim at up to half your speed (as a full-round action) or at a quarter of your speed (as a move action). If you fail by 4 or less, you make no progress. If you fail by 5 or more, you go underwater.

For the example, I have a human Battle Dancer with a base speed of 30 ft. and who doesn't have any swim speed. She plans to make a full attack on an adjacent target (five attacks in total).

Is her flurry limited by the maximum distance she can move depending on her swim check ?

Yes, you are limited by your applicable speed

The important lines are

If she is unable to move 5 feet, she can’t attempt any further attacks or combat maneuvers. She can’t exceed her maximum speed.

You are "unable to move" further than 1/4 (move) or 1/2 (full round) your typical speed while swimming without a Swim speed, so you would be restricted by that.

But how far can I move?

Your "maximum speed" is not a defined term, so I assume they would mean your speed which is how far you move with one move action. So, attempting to dance fight underwater without a swim speed would only allow you 1/4 your movement (usually 5ft).

Luckily, it is fairly easy to gain a swim speed if you're aware that you will be needing one.