Yes, this should work
But it's not quite as broken as it seems at first glance, but more on that later.
Let's go over possible "failure points" for this strategy:
- Monk restrictions: some monk features are restricted based on your armor or the weapons you're wielding. The claws count as monk weapons, as they're simple melee weapons without the two-handed or heavy property (monk weapon definition, PHB p. 78). Since a barbarian wouldn't be wearing any more armor than a monk, we can thus consider these prerequisites for some monk features as fulfilled.
- Flurry of blows restrictions: you only need to have taken the Attack action, so no issue here.
- Rage restrictions: being enraged prevents you from using certain features, such as doing any kind of spellcasting or concentrating on them. Everything else about the feature is either a benefit or restrict the frequency of its use or its duration, which don't matter here, since you're attacking in round 1 (btw, you could already make 3 attacks in round 1 with your claws, not just 2).
- Action economy: everything good here, since the claw's 3rd attack happens as part of your Attack action, and the flurry of blows is a bonus action.
Thus, nothing prevents this from working.
Balancing (at level 8)
Having established that the combo works, let's have a look at the balancing aspect.
Assuming you're a barbarian3/monk5, your attacks look like this (I'm using STR so you gain the barbarian's damage bonus and can use Reckless Attack; plus using DEX for monk attacks is optional):
- attack 1, 2, and 3 (claws): 1d6 + STR + 2, for a total of 3d6 + 3*STR + 6.
- attack 4 and 5 (unarmed strikes): 1d6 (because monk lvl 5) + STR + 2, for a total of 2d6 + 2*STR + 2
In summary, you'd deal 5d6 + 5*STR + 10, assuming all attacks hit - which is fairly likely if you use Reckless Attack. With a STR modifier of +4 (assuming you started with +3 and used your ASI for STR), this equals an average damage output of 47.5, which is pretty good tbh.
60% of that damage would be slashing damage from your claws, and 40% would be bludgeoning damage. From the text you quoted, it doesn't seem like claws can be used for unarmed strikes, but that only changes the damage type anyway, and not the total damage.
An average level 6 berserker barbarian equipped with a greataxe would be able to deal 3*(1d12 + STR + 2) damage while in a frenzy rage (40.5 total, as he'd have one more ASI to spend for getting STR to +5). A non-multiclassed beast barbarian couldn't surpass this without feats, and Great Weapon Master isn't even an option since you're not wielding "great weapons".
A single-classed monk, meanwhile, would only deal 4*(1d6+5) damage (again with +5 DEX due to one more ASI), for a total of 34 damage.
Thus, at level 8, this combination is very strong. However, you need to spend ki for this to work. Without that, you can attack 4x while raging, for an average of 38 damage - just slightly below the damage output of a frenzying berserker barbarian.
Balancing (at level 19/20)
If you're going up to level 20, it doesn't matter that much damage-wise if you focus barbarian or monk when leveling up; the damage bonus from the barbarian and the monk's martial die increase are about equal as far as I can tell.
I'll just assume barbarian3/monk17 (for the necessary level to gain path of the beast, paired with maxed martial arts die and a lot of ki points) for my calculations. I'll also assume STR 20 (i.e. +5).
At best (i.e. when spending ki), you'll deal 5*(1d10 + STR + 2), for a total of 62.5 damage. Without spending ki, you'll deal 4*(same thing), for a total of 50 damage.
Compared to a frenzying level 20 barbarian, who can deal 3*(1d12 + STR + 4) or 52.5 damage (bear in mind he'd have +7 STR due to primal champion), you're still ahead, and roughly equal if you're not spending ki (although you have enough ki to last quite some time when you only use them for flurry of blows).
A level 20 monk with 4*(1d10 + STR) or 42 damage is significantly behind, even when spending ki. However, this is to be expected, as monks focus on debuffs (such as stuns) and evasiveness, not on damage. Notwithstanding quivering palm, of course, but that's a) hard to compare and b) fricking broken anyways (55 average damage on a SUCCESSFUL save (!), and insta-knockout to zero HP on a failed save).
If you're at level 19, the berserker performs comparatively worse, as he'd gain +2 STR at level 20 (times 3, i.e. +6 damage), while the monk only gains a bigger martial arts die, for a total average bonus of +4 or +5 (depending on if you use flurry of blows).
Overall, I don't think this strategy is too badly broken. It's a strong combination to be sure, but it's not that much stronger than a frenzying berserker barbarian.