The goal of this question is to compare both Ranger level 11 features Whirlwind Attack & Volley and examine their comparative strengths and weaknesses for Rangers.
Whirlwind Attack
You can use your action to make a melee attack against any number of creatures within 5 feet of you, with a separate attack roll for each target.
Volley
You can use your action to make a ranged attack against any number of creatures within 10 feet of a point you can see within your weapon’s range. You must have ammunition for each target, as normal, and you make a separate attack roll for each target.
At level 11 you would have two attacks available during a normal action so you would need to have at least three enemies within 5 feet of you to make Whirlwind Attack viable from a pure damage output standpoint. The way I see it, you need to have good positioning, a number of enemies clustered, and be willing to sacrifice the utility of focused fire. I have trouble seeing this as being more useful than Volley.
Is there something that makes this choice mechanically competitive besides being in a campaign where you are frequently engaging hoards of swarming melee combatants?
As a rough surface calculation I am looking at volley as ranged AoE which can effectively hit 16 squares, while Whirlwind is a melee attack that can hit 8. The only direct advantage's to melee attacks that a melee focused ranger could have is a +2 to damage with every attack, or an additional melee attack as a bonus action. So if Whirlwind attacks and hits all 8 squares it can do 16 extra damage on a melee focused build. Or if we went with two weapon fighting one extra attack that would probably max out at slightly less than 16.
At a bare minimum if I could add one single target with volley to an ideal whirlwind I would getting almost the same exact damage spread.
If I were already surrounded by 8 opponents obviously WW is better, but chances are I don't actually want that to happen in the first place.