Setup
- Half-orc
- You start with a +4 Strength modifier at level 1
- You start with the Great Weapon Master Feat at level 1, as allowed by your DM
- Your Constitution modifier is assumed to be +3 at level 1
- ASIs go towards raising Strength to 20, the Sentinel Feat, the Savage Attacker Feat, and raising Constitution, in that order
- all attacks are made with Reckless Attack, -5/+10, and receive the rage damage bonus
- The enemies' AC is 16 for the level 12 comparison and 18 for the level 20 comparison
- Primal Path is Totem Warrior
- No magical items
- we optimize for Damage Per Round (DPR)
In the question's body you note that GWM, Savage Attacks, and Brutal Critical all profit greatly from critical hits, so we aim to incorporate the Champion's (Fighter) Improved Critical.
The first comparison is at level 12, as that is the earliest you can have Brutal Critical and Improved Critical. The second comparison is at level 20, to determine which is the desired final state.
The other PHB Fighter subclasses offer no tangible synergy (Battlemaster) or are objectively worse than pure Barbarian (Eldritch Knight; you can't cast spells while raging).
The next Fighter Feature after Improved Critical that could produce a high DPR build is probably Extra Attack 2, but that comes as late as level 11, which you specified as too late.
The question of when to use -5/+10 has been answered many times already so I won't elaborate on it, but under the given preconditions (attack modifier, enemies' AC, average damage, advantage) it is worth it. Of course, in actual play you'll have to judge situations separately.
The Great Weapon Fighting Style adds 0.833 per d12 on average. Technically you would profit more from GWF with a Great Sword, as its damage increases by 1.333, but Great Axes profit far more from Brutal Critical, so much so that this difference is more than compensated.
Sentinel offers simply more opportunities for Attacks of Opportunity and therefore more DPR. Note that Reckless Attack gives you advantage "during this turn", so if you used it during your turn, you won't have advantage on Attacks of Opportunity.
Savage Attacker provides a bonus of ~2 to a roll of 1d12+mod, and ~5 to a roll of 6d12+mod (with Savage Attacks and Brutal Critical 3 a Great Axe crit will do 6d12). This is a very minuscule bonus considering it only applies once per turn. Nevertheless, it is a damage bonus, although I personally would prioritize survivability over this Feat.
Both feats are not included in the concrete calculations, but are mentioned in the analysis.
Although you specify that survivability can be ignored while optimizing DPR, you mention in the body that you considered Barbarian 16 Fighter 4 to get that fifth ASI back, which offers no real improvement to your offense, given that you start out with 18 Strength and GWM at level 1. Therefore I will still try to mention at least your Constitution and max HP when comparing builds. The static hp-increase-per-level will be used to calculate expected max HP (7 instead of 1d12 for barbarians). Also, don't forget that a living Barbarian deals more damage than an unconscious one, which puts a real DPR gain on better survivability, albeit hard to quantify.
Barbarian 12 vs. Barbarian 9 Fighter 3
If you go full Barbarian, you get your third ASI right at level 12, which gives you a +5 Strength modifier and two more ASIs for Savage Attacker and Sentinel.
If you dip three levels into Fighter, you get only two ASIs, resulting in a +5 Strength modifier as well, but no Savage Attacker. You also have the opportunity to pick up the Great Weapon Fighting Style, which more than makes up for it.
Damage wise, B9F3 comes out ahead by a hefty margin of +14 average damage per round (59.98 over 46.14). Sentinel can't change this in favor of B12, as the average damage per attack is also higher for B9F3. Savage Attacker does at most ~5 additional damage, not enough to compensate the difference.
Barbarian 20 vs. Barbarian 17 Fighter 3 vs. Barbarian 16 Fighter 4
B20 gets you another two ASIs you could invest into Constitution, getting a score of up to 24 in both Strength and Constitution. Constitution would be your best choice for survivability, raising both your AC and your HP - which scale fantastically due to the damage resistance granted by your Rage. Indomitable Might is crazy good for stuff like grappling and shoving and Primal Champion is arguably the coolest capstone of all classes. Expected max HP: 285 (12 + 19x7 + 20x7)
B17F3 also gets you two additional ASIs, or four in total, giving you only one ASI for Constitution, getting a score of 18. But you get all Brutal Criticals (dealing 6d12 + mod on a crit) and Improved Critical, resulting in a phenomenal damage potential. Expected Max HP: 222 (12 + 16x7 + 3x6 + 20x4)
B16F4 costs you one Brutal Critical, but gets you that fifth ASI back, which we're promptly putting into Constitution, getting a score of 20. Expected Max HP: 241 (12 + 15x7 + 4x6 + 20x5)
All builds get both Sentinel and Savage Attacker, which are expected to scale the differences even further appart.
And now what we've all been waiting for. The winner is:
- B17F3 with an average 72.33 DPR
- B16F4 with an average 67.03 DPR
- B20 with an average 61.33 DPR
Magic items
If you found an item that raised your strength to, say, 24 (or higher, but we're using 24 to prove a point) , B20 would gain nothing and B17F3 would gain another immense boost, making that option even more attractive.
If you found a +1/+2/+3 Magic Weapon, both options would still gain a lot, but B17F3 would even more than B20, making it, again, more attractive.
If you found an item that raised your constitution to 20 or higher, all options would gain some, as there's still Dexterity to improve, which would raise your AC. This would be hard to quantify, however, so I'm not gonna try.
My Recommendation
The jump from 61 to 72 DPR is about a 20% increase, costing you 63 Max HP (or about 22%). If you compromise for 67 DPR you lose out on half your damage increase, gaining back merely 19 max HP, which makes this a suboptimal choice in my eyes.
If you want to be unkillable, walk the Path of the Totem Warrior, (choosing the Bear at level 3) all the way to level 20 and you'll have an insane amount of HP and still deal absolutely respectable damage.
If you enjoy the satisfaction of a build well executed and reaaaally want to regularly deal those 6d12 damage (about once every two turns, opposed to once every 5 turns), go B17F3.
Level progression
One of the largest damage increases you'll feel is Extra Attack at level 5. In order not to feel left behind I suggest - should you decide to multiclass - you go 5 Barbarian, 3 Fighter, 12 Barbarian. Action Surge, the Great Weapon Fighting Style, and Improved Critical all improve your DPR substantially, providing most of your biggest DPR increases as early as possible.
(source)
"My other recommendation"
I stood before this same dilemma a while ago and solved it as follows: I asked my DM for a specific magic item and received:
Gorehowl
Greataxe, req. att.
+2 weapon. Rolls of 19 or 20 count as critical hits. Disadvantage on Stealth.
