I think your original idea was bang on. The whole sea battle was a single prolonged (and difficult) encounter. The only way I can see this encounter being split up into two is if the party retreated out of sight and took a prolonged rest to regain spells/abilities or if you intentionally wanted to split it into two encounters to 'go easy' on them.
I don't think there is any hard and fast rule for when exactly an encounter ends. I think it's a decision which must be made by the GM who should not feel he has to change his decision because the players are doing wonky stuff or failing to engage for 3 rounds so they can get abilities back.
Because the definition of 'encounter' is planning based and a 'game' aspect rather than a story element, I think whatever you planned must usually hold for the duration of the adventure. After the game session you can ask the players if they liked your rules calls and work something out if you wish.
The one exception I'd allow is if the encounter went far differently than you had planned. For instance, if the players cleverly avoided the battle or split it up for a prolonged time it could be treated as two separate encounters (this sounds like what happened to you).
Personally, I would have ruled the whole thing as one encounter because it makes it too easy (and less fun) if it was broken up. That's a personal call. If you feel it's integral to decide exactly when one encounter ends and another begins I think you should consult your players and agree on something reasonable. There is no written rule (that I'm aware of) that says when one encounter ends and another begins. I've always decided that in my best judgement and I hope your players would support you if you did the same.
For myself, a reasonable 'end' to one encounter would be A. one that I'd planned or B. a full retreat out of range of all attacks, danger, and an essential end to the possibility of future conflict in the near future. I.e. if the party escaped the dungeon to a place of relative safety and could then 'choose' to either re-engage or leave the area completely. If the threat pursued them to the location it could reasonably still be called one encounter (which sounds like what happened to you because the second wave would be inevitable).
I hope this helps!