Summary: X winning the initiative roll doesn't mean they see or know about the attacker before the attack. It merely means that mechanically the Surprised condition isn't still in effect when an attack happens.
(Unsolved problem with this: what if A and B decide not to attack at all after rolling initiative.)
The mechanical consequence of this rule is that to actually surprise-attack someone, you also have to beat them on an initiative roll. As @CTWind says in an answer, declaring intent to attack puts the game into round/turn-based simulation mode to figure out the details of what actually happens. If B chooses to attack before A instead of Readying an action, that's what actually happens. (This sucks for B if they lose out on move+attack or Extra Attack because they happened rolled higher initiative than A: they "failed to coordinate their attack as well as they'd intended".)
I think that solves the problem of B acting before A, but if we leave out B then we still have to explain what it means for X to take their turn before A and lose the Surprised condition. (At which moment does the 'Surprised' state disappear? concludes that the only sensible interpretation of RAW is that a creature is no longer Surprised after they've had their first turn, during which they couldn't act.)
If the ambusher wins the Initiative roll, things go as they anticipated, gaining the initiative over the target. You get 2 turns of attacks before the first turn in which they can actually act, or can Attack and move away without taking an AoA.
If you (the ambusher) don't beat them, the only benefit you got was being able to attack in a round where they didn't get to do anything on their turn. So even though you lost the initiative, you've still had 1 more real turn than they have, or equal, at any point in the rest of combat.
In narration, this can be them recovering right away from the attack, quickly enough to take a reaction like Hellish Rebuke or Riposte. Or to take an AoA at a melee attacker that moved in to attack and then tried to move away.
Even though they weren't expecting it and didn't see it coming, their immediate reaction proved to be useful, and the game models that as being able to take a Reaction to anything they can perceive after their first turn that clears the surprise round.
If nothing happens until A's turn, then there's nothing for X to react to.
This is important for preserving causality in the narration: X can't actually do anything until after something is done to them. (Or they see something, e.g. if B moved past them without attacking, X could make an AoA.)
X's first turn, where they clear their Surprised status, doesn't involve them reacting to anything at all. They don't know that they're "not Surprised" because the thing that might have surprised them still hasn't happened. Just that if/when something they notice does happen, they won't still be Surprised.
One possible problem with this interpretation is that RAW, A could decide not to actually attack once it's their turn. The DM might disallow that as meta-gaming after declaring intent to attack, or might narrate "steadying yourself for the shot, you feel your timing is off; you take a breath and clear your head". I don't think you get to try again right away; you're still in initiative order and the non-surprised target is still doing whatever they were doing (e.g. walking or talking). Maybe if you wait for the target to focus on something else, you might get another chance to surprise them? But these aren't ideas I've playtested, I'm not recommending them; I'm pointing them out as problems with letting A change their mind after getting into initiative order. But in general they need to have some flexibility; what if other party members act first and their intended target is dead, or Banished.
Another possible interpretation: The turn-based system is a mechanism for simulating simultaneous events, simplified/modeled as a sequentially-consistent series of turns. There aren't real gaps of time between turns. X's losing the Surprised condition narratively happens as they're being attacked, and the surprise-attack wasn't perfectly successful so didn't get the real benefit of Surprise. (e.g. Assassin (Rogue)).
But the sequential-consistency of the turn model still applies: if the target is KOed by the attack, they don't get to take a Reaction because the attack happened first. Or during the flow of combat, if you're KOed before your turn, that means that you don't get to act; your character was probably in the middle of doing something, but it didn't end up being completed or having an effect before the knockout blow.
RAW, to fully successfully ambush someone, you have to beat them on an initiative roll. i.e. they get a roll to reduce the impact of the surprise.
This really matters a lot for an Assassin (Rogue), where a hit on a Surprised creature is automatically a crit, and they get advantage against targets that haven't taken a turn yet in this combat. (The can't-act turn for a Surprised creature counts as a turn).
Thus an Assassin (Rogue) needs to beat their victim on an initiative roll to use their class features as part of an attack from stealth, according to RAW. As CTWind and
I discussed in comments, this is sort of like the victim getting a "saving throw" against being assassinated. (Mechanically, Initiative rolls are Dex ability checks, not actual saving throws.)
At which moment does the 'Surprised' state disappear? has some examples about Assassin Rogues, and this is the RAW conclusion.
RAW, this still applies even if the Assassin is far away, in cover, and has been holding a crossbow aimed at the target for minutes. Their only movement is to pull the trigger. So physically there's nothing for the target to react to.
Narration like "saw the assassin move while they were winding up to throw a knife" doesn't work for a hidden crossbow from behind. (Assume magical Silence or whatever if you think they might hear the bolt whoosh through the air.)
But D&D doesn't run purely on physics. Luck is a real thing. Rolling high on initiative could be (if no other explanation works) a kind of "danger sense" (aka spidey sense) tipping you off to change your stride and make the bolt miss a vital area, only doing normal damage spoiling the crit Sneak Attack. High-level characters with lots of initiative bonuses are hard to assassinate in this way, as well as having lots of hit points. (Which are also a pretty nebulous concept).
D&D is (usually?) a fantasy setting where you wouldn't find a laser weapon that's impossible to react to without FTL senses, so usually there will be a plausible explanation of reacting to a projectile in flight, or for melee of an assassin moving up behind you. But regardless, "danger-sense" probably always works. The world is a dangerous place; part of having class levels is this kind of always-ready ability to make an initiative roll in any circumstance.
People that survive an assassination attempt were lucky and/or happened not to be flat-footed. (If a low-HP target does still die from an attack that turned out not to be a crit Sneak Attack, you could still describe it as a going off mostly the way the assassin planned it.)