Having such numerical rules of thumb are both design decisions and design guidelines
There is no “correct” ratio of monster damage to player damage or player HP to player damage, but these kinds of ratios are well worth thinking about. They influence balance, but also influence the play and feel of your game. If HP is about 4× damage, you expect people to take four hits (or last four rounds, or whatever period you’re aggregating damage over). Deciding what ratios you want (which ratios will generate the feel you’re going for and encourage the types of play you’re expecting) is a big part of the design process, and it is, of course, very complicated.
There’s also probably not just one ratio you want for any given pair of numbers. Tough characters are supposed to last longer, so perhaps their HP:expected-damage ratio is higher. Nimble characters are expected to deal high damage but be fragile. So on and so forth. So you really need different notions of what these ratios are supposed to be for the different sorts of characters you want to support. You can make this very math-y and codified (e.g. D&D 4e) or very vague and flexible (e.g. almost any point-buy system you care to name).
And then there are issues of system-mastery: how much do you want to reward people for thinking carefully about their character’s mechanical components? How much do you want to punish people for failing to do so? Are certain archetypes going to be possible, but bad ideas? Is that a good thing, or something you want to either build real support for or prevent? If you allow them, are you going to warn players about them explicitly?
This isn’t even all of the numerical thoughts a game-designer needs to have, without even getting into the other crucial aspects of the design (setting and tone and description and art and so on and so forth).
Theory-crafting can help
Designing a balanced game requires, more than anything else, a lot of time spent fiddling with it. Theory-crafting is faster than play, so on some level you get more fiddling per unit time. This can be useful: look at your current ruleset, and try to determine the maximum possible values someone can get for various statistics. You’ll have to try going at this a lot of different ways, however; optimums are often found in surprising locations, and you can never be certain you have found one.
Having an interested community helps massively here. More minds, more eyes, more original ideas and approaches.
But it cannot replace play-testing
When theory-crafting finds game-breaking exploits, they’re worth fixing. Sometimes the builds that yield them aren’t even powerful in practice; it’s just about maintaining those design decisions you made (e.g. those ratios) and keeping the environment stable (if something achieves an anomalously-large value for some number, and you ignore it because that number isn’t very important, you now had better remember that it can get anomalously-large before you add a new thing that depends on that number!)
But theory-crafting cannot and will not catch everything that is actually a problem. Having overly-high numbers might make things too easy, but the players having abilities that you didn’t expect to see at all is often much more troublesome. If a player figures out a clever way to fly when they’re not supposed to, that might invalidate whole sections of your plan.
And flying is an easy example. The real danger are things you didn’t even think of, that never seemed relevant or important, until some clever player breaks everything. This is what play-testing uncovers.
Again, an active and involved community helps a ton here; each group may only play it once (or maybe a couple of times), but if you have many groups, that’s many tests. Especially, say, if one person can run it for multiple groups, letting each group try different tricks.
Especially if this is actually a matter of testing, and these are explicitly (volunteer or paid) testers, it is worthwhile to encourage destructive testing. This means that the testers are trying their hardest to break things; they are stressing the system, maybe trying out some of those theory-crafted builds that you found to be just within the bounds of acceptable, maybe just trying oddball things. Even abusing foreknowledge of the events of the game, for repeat players: if someone could plan for those events and be over-prepared for those specific events, someone might accidentally stumble upon the same preparations. Plus it's just good to get a gauge for what is the easiest someone could get through here?
Testing the opposite side is useful too: if someone made some truly hare-brained decisions, can they still enjoy this game? Try throwing some intentionally “awful” characters through: how much do they suffer for those poor decisions? Is that level of penalty for bad decisions appropriate to your design goals?