I had a session this week involving a fairly large battle at sea. The encounter involved a couple of boarders while the enemy ship was still at quite a distance, and the larger group still on the ship. I modeled this as a single hard encounter (APL+3) which didn't work out well, as it was split up by a fair number of rounds which made it play out as two smaller encounters, the first of which was a pushover.
How can I know when I should play something as two separate encounters instead of one?
Well, the simplest answer is "if encounter 1's going to be done 3 or so combat rounds before encounter 2 starts." You can't always do this; smart tacticians will try to break up encounters for that exact reason, and that's part of the reward of being a smart tactician.
We did this in our Jade Regent game recently; there was a hobgoblin keep with drawbridge, parapet with archers on it, and melee troops inside. So we dimension doored to the second floor, wall of iced the stairwell, and killed us some archers. Then the melee guys came busting through the ice but only after we'd killed the archers and had a round to buff and BS with each other. It would have been a very difficult encounter without the separation; with it, it was easy (though we did blow two 4th level spells on it).
It can go the other way; dumb or unlucky players can end up "pulling" bunches of adjacent foes in a dungeon or whatnot. I consider this to be a feature of earlier D&D rather than the "set number of army points!" approach of some later editions .
Sometimes you eat the bar, sometimes the bar eats you.
All you can do is do best-guess planning of most likely outcome. It's OK for the PCs to rock a combat every once in a while, makes them feel all manly and stuff. Then you can really outclass them and see what happens (Last weekend I tossed 2 CR 12's and some trash mobs against a 6th level party to see what happened. Took like 4 hours but they won.).
I think your original idea was bang on. The whole sea battle was a single prolonged (and difficult) encounter. The only way I can see this encounter being split up into two is if the party retreated out of sight and took a prolonged rest to regain spells/abilities or if you intentionally wanted to split it into two encounters to 'go easy' on them.
I don't think there is any hard and fast rule for when exactly an encounter ends. I think it's a decision which must be made by the GM who should not feel he has to change his decision because the players are doing wonky stuff or failing to engage for 3 rounds so they can get abilities back.
Because the definition of 'encounter' is planning based and a 'game' aspect rather than a story element, I think whatever you planned must usually hold for the duration of the adventure. After the game session you can ask the players if they liked your rules calls and work something out if you wish.
The one exception I'd allow is if the encounter went far differently than you had planned. For instance, if the players cleverly avoided the battle or split it up for a prolonged time it could be treated as two separate encounters (this sounds like what happened to you).
Personally, I would have ruled the whole thing as one encounter because it makes it too easy (and less fun) if it was broken up. That's a personal call. If you feel it's integral to decide exactly when one encounter ends and another begins I think you should consult your players and agree on something reasonable. There is no written rule (that I'm aware of) that says when one encounter ends and another begins. I've always decided that in my best judgement and I hope your players would support you if you did the same.
For myself, a reasonable 'end' to one encounter would be A. one that I'd planned or B. a full retreat out of range of all attacks, danger, and an essential end to the possibility of future conflict in the near future. I.e. if the party escaped the dungeon to a place of relative safety and could then 'choose' to either re-engage or leave the area completely. If the threat pursued them to the location it could reasonably still be called one encounter (which sounds like what happened to you because the second wave would be inevitable).
I hope this helps!
In our groups we have always combined those as a single encounter in face if the fight is not considered a challenge which you first one was not. We don't even consider it an encounter.
