When the opposing party is no longer expecting a threat.
Narratively, your BBEG is fighting the party, and runs away, yelling
You'll pay for this one day!
Let's say the party, being cautious, tells you, the DM,
Ok, the Druid is downing a potion on the poor Barbarian, while the rest of us scout the remainder of his room for Mimics. Paladin is in front, I've got my Firebolt readied in case we find one.
At this point, they won't be surprised, they're still engaged in a combative attitude, calling combat actions, even if they're not strictly going by Initiative. At some point, as usually happens, everyone relaxes and you start listening to something like
I'm gonna take a short rest, get some HP back. Druid is attuning to the new magic item he just found. Wizard is trying to resurrect all the corpses around here, and the Barbarian is crying over his broken greatsword in the corner.
It is at this point that they can be Surprised again if they don't detect a threat coming. Before now they were all alert, looking around, expecting a trap (just like they behave before entering a new room, for example). Here, they've relaxed, they do not expect any further combat.
The rules aren't crystal clear on this, and Stealth (and by inference, Surprise) is often left to DM adjudication (10:50).
We very intentionally, in 5e, have put Stealth in the domain of the DM.
By RaW, once your BBEG ran away, the round would end so would combat.
Once everyone has taken a turn, the fight continues to the next round if neither side has defeated the other.
At some time later, your BBEG could come back, hidden, and re-engage to get Surprise attacks and what not. You'd compare Perception checks against your BBEG's Stealth checks and act accordingly.
The GM determines who might be surprised. [...] Any character or monster that doesn’t notice a threat is surprised at the start of the encounter.
The issue here is determining that combat actually ended/started. The PHB isn't very clear on this, but your adventurers can be in a stand-off, not rolling initiative, but alert and ready to engage if someone does something threatening. Initiative is an abstract concept for the players to organize combat, used essentially to track time, but PCs can still be alert and calling combat actions without the players moving through initiative order. There is not a clearly defined answer, and Ryan makes a good point:
One thing I like about 5e is that for the most part, the game mechanics work the same way both in and out of initiative. But mechanics like Assassinate that activate when initiative is rolled are one thing that breaks this equivalence.
It falls into the DM's hands to determine whether the PCs are surprised by the stealthed attacker, or if they are still alert enough to be considered in "combat mode" even if they're not following initiative strictly.