Need help with balancing a homebrew monk subclass

Question

I'm new here. So I will be running a campaign and one of my players gave me a homebrew monk subclass they would like to play. I don't have a lot of experience with balancing, so I'd like to hear some of your views.

Here is the subclass in question.

3rd level ability
Go Beyond!

You channel your inner energy and put what you can without going over your limit into one strike. Make only one attack this turn, and if it hits roll normal damage and 2d8. Add this in as force damage.

As a bonus action you can go past your limit and increase your own power after expending two ki points. Outputting damage at the cost of your own health. Roll 2d6 and add that to your attack’s damage. You take half of the damage you output with this roll, your body simply can not safely handle this amount of energy. (scales up to 4d6 at 6th level, 6d6 at 11th, 8d6 at 17th.)

3rd level ability
Endurance!

Speak some encouraging words to yourself, and become boosted from your own form of personal inspiration. Spend a ki point and as a bonus action roll a martial arts die and add your wisdom modifier to it. You gain this number in temporary hit points. You can only use this ability up to three times before needing to take a long rest.

When you gain this ability, once per short rest if you go down in a combat due to extra damage that you inflicted yourself by pushing past your own limits, you do not go down. You instead end your turn at 1hp.

6th level ability
Smash!
(power increases at 11th level)

Something comes over you and you push yourself past your limit to destroy an enemy. You're able to channel your power into one powerful attack.

Once per turn as an action you can force one creature in front of you to make a Dexterity saving throw after you hit them with an attack DC = 8 + Proficiency + Dexterity Modifier. On a fail they take 6d6 Force damage and they are shot back 20 feet.

If they succeed the saving throw they take half damage and are not pushed back. You may do this a number of times equal to your Constitution modifier before needing to take a long rest.

Using this ability more than once between long rests is possible. But after the first use the user begins to take damage from exerting such a powerful blast. If used more than once between a long rest, the user receives half of the damage dealt. You spend 3 ki points using this ability.

11th level ability
This is it!

As a last effort to defeat your enemy you feel a surge of kinetic energy through your body. While you are below 1/2 your total hit points, you gain the following for 1 minute:

• Your unarmed attacks deal additional damage equal to your proficiency bonus, but you take an equal amount of damage.
• Inspiration. As a bonus action, Inspire your allies to fight and give their all! Your allies gain temporary HP equal to your level. One use
• Limit Break. you have advantage on Strength and Constitution saving throws.

11th level ability Smash!
(improved smash)

You have become more powerful overtime, and are always trying to learn how to control your power better. Once per turn, as an action you can attack one target or target up to 3 creatures in a 20 foot cone to make a Dexterity saving throw DC = 8 + Proficiency + Dexterity Modifier.

Those who fail take 6d8 Thunder damage and are shot back 20 feet If they succeed the saving throw they take half damage and aren’t knocked back

You may do this a number of times equal to your Constitution modifier before needing to take a long rest.

Using this ability more than once between long rests is possible. But after the first use the user begins to take damage from exerting such a powerful blast. The user receives half of the damage dealt. You spend 3 ki points using this ability. Add another ki for each other person hit, max of 2 more so 5 ki points.

17th level ability
Final Smash!

At 17th level, you’re able to unleash the most devastating attack in your arsenal. You cause a burst of energy originating from your fist, every creature within a 30 foot radius must make a strength saving throw or take 8d10 Force damage. Creatures within 30 feet are knocked prone, this can also be used directly on a creature which forces them to be knocked back 30 feet and they are stunned if they fail the save. Creatures who succeed the saving throw take half damage and suffer no other effects. Using this on the ground causes 15 foot radius to become difficult terrain. This deals double damage against objects. You can use this a number of times equal to half your Con mod, and gain a point of exhaustion each time you use it. After using it one time between a long rest, you take half the damage you deal back. You regain all uses at the end of a long rest.

Answer 1

This is ... Not Balanced

Assuming comparison to the core Monk, this is not a balanced Monastic Tradition. Credit to the author, the self-destruction theme is an interesting attempt at balancing the power of the abilities, reminiscent of several popular fictional power sets, but it fails in a TTRPG context.

The basic issues are: too many abilities, badly written abilities.

Too Many Abilities
This variant gets six abilities, more than the four granted to the Way of the Open Hand. While three of those abilities have similar names, implying they are upgrades of a single ability, they are completely separate abilities.

Badly Written Abilities
The abilities are not clearly written. They frequently deviate from accepted rules text. They reference rules that don't actually exist. If the reader has to guess what a rule means then the rule is badly written.
Example: Endurance! directs the user to roll "a martial arts die", which is not a thing. Presumably, the author meant "roll the correct damage die for your Martial Arts ability", but that is a guess.

1. Go Beyond! adds 2d8 Force damage to a single Attack. Not only is this a massive damage increase (average +9 damage), the bonus damage is Force damage, bypassing most Resistance and Immunity abilities for the "price" of not spending a ki point on Flurry of Blows. It does also prevent the character from using Extra Attack at higher levels, but that does not mitigate how insane this power is for a level 3 character.

• Further, it allows the Monk to spend 2 ki points to add 2-8d6, if the monk is willing to suffer half the 2-8d6 damage. Untyped damage, which lets it ignore the Resistance and Imunity rules.

2. Endurance! Grants temporary HP, at 3rd level, for a single ki point and a bonus action. The amount is dubious but, assuming my earlier guess is correct, should be 1d4+WIS to 1d10+WIS. This is a fairly strong ability, and a good level 3 tradition ability all by itself.
3. Smash! does not work. It is an action to use Smash! but it requires a successful attack. You can't use it with Flurry of Blows because FOB requires the character to take the Attack action. If it did work then it would be a 1 / turn to inflict an average 21 Force damage and a 20' push (10 Force damage and no movement on a successful save).

• It also has three different limiters on repeated use: CON / Long Rest, 3 Ki Points per activation, and self-harm on all subsequent uses. Decide if it should be Ki powered or X per Long Rest.

4. This is It! is screwy. The trigger should be 1/2 of Maximum hit points - a character can never be below half their hit points. The benefits are automatic, and last for 10 rounds. The character inflicts +4 to +6 damage, but suffers 2 to 3 damage for each successful attack. Allies gain [character level] Temporary HP (11 to 20) - possibly every one of your turns? this part is very unclear. Gain Advantage on two saving throw types.
5. Improved Smash! is a separate ability that increases the damage of Smash! by about 6, changes the damage to Thunder, and adds the option to affect two additional targets within a 20' cone of the first target. It's a confusing way to increase the number of Smash! attacks that can be made, but at least it can actually be used. Similar restriction issues as Smash!.
6. Final Smash! does not have a Saving Throw (despite repeatedly referencing one), can be a 30' radius or a single target or a 15' cone, automatically inflicts Prone, can cause Stunned, deals double damage to objects, and can be used for free once per Long Rest. While 8d10 Force damage (average 44 damage) isn't very special at level 17, the other effects make this pretty absurd.

Potential Fixes

• Clarify every ability. Remove all ambiguity, explicitly list all costs and conditions up front, make sure to list the saving throws with each ability, and put the separate uses into separate paragraphs that also state they are alternate uses of the abilities.
• Unify the limitations. Monk abilities are either X / Rest or require Ki points to activate. This mixes the two, and throws on self-harm (easily countered by allied casters, especially the spell that heals any creature that enters a specified area).
• Fewer abilities. I like Endurance! but it doesn't fit the theme and style of the rest of the tradition. This is It! is trying to do too much, removing its identity; it is also really strong compared to something like Tranquility. I recommend removing both abilities.
• Specific fixes. Go Beyond! could easily work as a Flurry of Blows alternative - spend a Ki point, use a bonus action, increase your unarmed damage by +1 die.
• Specific fixes. Smash! could be rewritten to work like Open Hand Technique, being activated with Ki after a successful hit. Then change the effect to player's choice of 'Save or be pushed back 20 feet' or 'Bonus damage. Only use this option once per Short Rest'.
• Specific fixes. Improved Smash! could add a cone area option to Smash!, pushing back all creatures in the cone. Being an actual modification of a previous ability, this opens up room in the design for a replacement of This is It!.
• Replacement ability. Replace This it It! with a self-healing, similar to Wholeness of Body.
• Specific fixes. Final Smash! needs the general rewrite recommended earlier. Additionally, clarify the player's options by using a bullet-point list of the different areas and conditions that can be imposed.

Good luck with your new campaign!