Let's say, hypothetically, that a rogue gets trapped in a dead end of a 40 foot long hallway that is exactly 5 feet tall and wide. If someone cast Vitriolic Sphere at the end of that hallway, how does the rogues evasion work? There is nowhere to run, and it fills the whole space with acid, how can the rogue conceivably dodge it with no damage? It may just be for game balance but I was wondering if there was some scenario I overlooked that could make the rogues evasion make sense.
Yes, it would work. How? Well…
Evasion is merely a modification of the existing save-for-half convention that many spells use. How does a rogue sometimes evade the damage from an area effect that it seems like they should take full in the face? The same way that a normal, non-rogue can sometimes take only half damage from an effect that seems like they should take full in the face.
How do you picture normal save-for-half-damage working? Whether you imagine it as flinging themselves into exactly the right spot to avoid the brunt of the effect; or rolling with the effect so that they blunt its force; or ducking and covering just right so that the worst of it rolls off them like water off a duck; or some kinds of semi-magical ability to deflect area damage; or just a genre trope where they inexplicably always seem to escape unscathed from things that should not be survivable — that's how a rogue evades too, just better.
They're just extra-good at whatever that normal save-for-half technique/accident/trope is, enough that they always take only half damage when they mess up, and avoid the damage entirely when they're on their game.
(Note that vitriolic sphere isn't a literal 20′ sphere of liquid acid. It's a 1-foot sphere that explodes, splattering everything in a 20′ sphere. It's much less inexplicable to imagine someone miraculously avoiding all injury from that than from being submerged in a giant ball of acid.)
Keep in mind that game mechanics don't necessarily follow the rules of normality. Take the elemental monks ice shaping discipline for instance. You can control ice completely at will but a caveat is that you can't shape it as a prison around people. There is no explanation for this, as it happens within the span of 6 seconds like any action and can't be interrupted, but it's a rule for balance's sake.
Similarly, the rogue's evasion mechanic is a way to protect them from high damage spells at higher levels. Is there necessarily an explanation how they dodge a blue dragons lightning breath? Lightning moves faster than the sound barrier but this doesn't prevent a reaction or dodge move because it's D&D.
That being said, you are a DM and if you want him to simply fail, that's in your power, but the question you should be asking is "What's more fun for the player?"
If you really need an explanation you could say the spell is mistimed. Or maybe the rogue tried to jump past it. Maybe it just went wrong or he got lucky. The options are endless, but mechanics wise, it relies on suspension of disbelief like most game powers
It sounds to me like you're looking for an in-game/roleplaying description of a Rogue using Evasion to save against a Vitriolic Sphere while in a confined space (although not Squeezing).
When the Sphere explodes, it does not spread uniformly in all directions - its expansion is erratic in that it expands in different directions at different velocities. As such, it would create pockets of space where the Rogue could weave herself in and out of to avoid the splattering of acid.
"You watch intently as the green blob soars through the air toward you. As the blob explodes with violent force, you duck and weave among the splashes of vitriolic acid flying in the air all around you. You run up along the western wall, and you jump across toward the eastern wall to avoid acid that splashes right onto the spot you've just left. Once the explosion has spent itself, you land right back on the spot where you originally were, unfazed and unharmed, look the surprised wizard in the eye, and smile a mischievous smile. That all you got, mage?"