The character still can be surprised
Initiating a fight while being unnoticed should give an upper hand. You can't negate it just by saying "I was on high alert", without investing anything. Potential investments could be the Alert feat, the Perception proficiency, an Inspiration point being spent on the Perception roll, etc.
Adventurers always expect combat, hence Passive Perception
Chapter 8: Adventuring has the "Noticing threats" paragraph:
Use the passive Wisdom (Perception) scores of the
characters to determine whether anyone in the group
notices a hidden threat.
While traveling at a fast pace. characters take a -5
penalty to their passive Wisdom (Perception) scores
to notice hidden threats.
This implies characters are always searching for threats, by default, players don't have to announce "I'm on high alert" every time for this. You even can try to be stealthy while moving at a slow pace (and you don't have to be a rogue for that). That probably means the rogue itself can also be ambushed.
The described situation might be already an encounter
Surprised mechanic models two different things in 5e:
- The combat was unexpected for the creature — it doesn't attack anyone in the first round because it is still preparing itself and is trying to figure out the situation.
- The creature was busy doing something else when it was attacked — it doesn't attack anyone in the first round because it was busy doing this thing.
Unless Surprised is meant to trigger a feature (like rogue's Assassinate), you might just roll for initiative and let things happen naturally.
The final result depends on the situation:
Situation 1. The combatants are in place, both are aware of each other presence, and combat is imminent. This is already a combat encounter. Ask players to roll for initiative.
Situation 2. The character doesn't know, how the rogue attacks. Is he/she alone? Will they try to backstab? Will they shoot from afar? The character isn't sure if the rogue is still around. It is a normal "travelling in a hostile environment" situation — use the passive perception check.
RAW, DM determines surprise
PH doesn't say much about how surprise is determined, leaves it to the DM:
If the adventurers encounter a
hostile creature or group, the DM determines whether
the adventurers or their foes might be surprised when
combat erupts.
DMG lists surprise as a drawback:
Modifying Encounter Difficulty
Situational drawbacks include the following:
- The whole party is surprised, and the enemy isn't
It doesn't give clear instructions either, and lets the DM use common sense instead.