Frame challenge
I'd like to offer a bit of a frame challenge.
You say:
However, I think they are playing with fire by pushing ever farther beyond the guideline to break us -- at some point, the risk of a TPK becomes very real.
I think your actual problem is that the DM and the players (or at least one player) are not agreeing as to what is fun.
You might not be 90 degrees off from each other, but you're some degrees off.
The combat you described does indeed sound over-the-top, but you also say:
We somehow managed to not only survive and escape, but also kill about half of the leadership and most of the rank-and-file Hobgoblins in the process.
Sounds like a success to me. Faced with overwhelming odds, you decimated enemies that were intent on not just killing you, but destroying your very souls, and you either eliminated them as a threat, escaped their area of influence, or weakened them so much you'll be able to mop them up later.
And maybe you've bagged so much XP it's time to level up.
But there is something here that the DM has done that is not-fun to you, and that's the issue, and that's the way you should approach it.
Some tables, TPKs aren't a no-fun game
At some tables, the game is played where TPKs are possible, or even likely, or common. These 3 characters die, 3 more come in. I don't think that kind of play is probably common, but it is certainly done, and within the rules. For one table a TPK is a disaster and a horrible, no-fun session, everything is ruined; another table, it's business as usual.
Also, you can certainly play a high paranoia game where those 3 characters never make a move without scouting and scrying, always hidden, and whenever confronted with a superior force, run and hide, then regroup, strategize, and attack. History is full of examples where an inferior force defeated a superior force in just this way. (Although tbf, history is also full of TPKs.)
Handling character death
Different tables handle character death differently. Some tables, characters effectively never die permanently, or perhaps even temporarily. Other tables, characters die all the time. The DM chooses how death is handled. If you don't like it, then that's a topic to discuss with the DM.
Finding an XP recommendation won't fix your problem
Because basically what you want to say to the DM is, "you're doing it wrong, here's better XP guidelines.".
But nothing you've said suggests that the GM thinks they're doing it wrong.
XP budgets don't kill parties, DM's do. A TPK can only happen because the DM lets it happen. The DM has an infinite set of tools at their disposal to prevent it. And for many players, at least the possibility of a TPK makes the game fun.
And also an XP recommendation is likely to be imprecise
There are a lot of factors that make XP guidelines for building encounters very imprecise, especially as you level up.
They don't take into account the state of the party. The party could be well-rested, or not; buffed or not; have chosen the right spells or not; have the optimal equipment for dealing with this particular encounter, or not.
They don't take into account that as the number of participants goes up, the complexity goes up. XP becomes an extremely poor predictor of outcome given the many variables such as area-of-effect, who is aiding who, who is benefited or hindered by terrain, and even can the DM (and the players) keep track of the many participants. If the XP recommendation assumes that either the DM or the players are executing perfectly, then the extent to which it accurately depicts outcome will be heavily influenced by how well the combat is executed on either side.
XP guidelines can only average swinginess, at best. As levels and CRs go up, effects and outcomes become ever more swingy. Spells and effects like disintegrate are all-or-nothing, and scoring a hit on a leader of one side or another at the right time could turn the tide, but missing will use up a 6th-level slot and more importantly, likely a character's whole turn. And outcomes become more swingy, too.
What you can do instead
There are some tools, most of which I have used many times. I can't say there aren't too many times where I just decided not to play D&D because the game didn't seem worth the candle, but it has happened.
Talk to the DM one-on-one out of session
Hopefully you're basically friends, so somehow you should be able to talk about it. You probably can't "fix" the problem in a single discussion. However, you can become more aware of each other's viewpoints, and in my experience, that helps. I find in these conversations it's usually better to go light, and not try to get it all out all at once.
Talk with the whole table, after session
In the group I play with, there are multiple games going on, and various people are players or DMs, and we're a fairly experienced and opinionated bunch. In the games I'm in we almost always have a discussion afterwards. Usually the DM says, "Aaand that's where we'll end for tonight. Before we go, questions, comments, concerns?" Sometimes it's pretty quick, other times it's lasted quite awhile.
I'm not sure a player can effectively start this conversation, I think it probably has to come from the GM. Maybe it's something you can suggest to the GM, one-on-one.
What you can do at the end of a session as a player
What I have done as a player is two-fold. Even when I don't agree I've listened very carefully to what the DM and the other players have to say about my role in what is fun and not-fun.
Secondly, I make sure that at the end of every session I usually try to throw out a few comments about what I thought was cool about the session, whether it was a fellow player's clever maneuver or roleplay, or something cool the GM did.
I have also found in group pastimes that sometimes one person bitching and moaning becomes contagious. I try to make sure that any feedback I offer does not feed into this, at the very least it's counterproductive if my observations on something being less than perfect actually leads to things being more less than perfect.
Change the game
If you aren't having fun, don't play. Suggest a different game, or run the game yourself.
Roll with it
You can also just lean into it. Miamoto Musashi taught in The Book of Five Rings that a samurai does not fear death because they are already dead.
You could decide that this is the game, and if you're having fun, who cares if everyone dies.
No one ever likes to hear "change your attitude" as a solution, but it is true that sometimes focusing on changing yourself yields better results.
Suck it up
If you're having enough fun that you don't want to quit or play a different game, if you can't change the GM, and you can't change yourself, then just suck it up and focus on the parts you like. Again, people rarely want to hear this, but it's an option.
Calibrating encounters for higher levels
As a DM
I don't weigh everything out by XP, and the GMs I play with don't either.
I try hard to make sure character death is unlikely but possible, and that a TPK can only happen if the PCs do really dumb things on top of dumb things, or purposely decide to go down fighting. I tell the players this ahead of time. We discuss it in session 0, and along the way if we need to. I find it's more fun if the players usually triumph but not always, and sometimes have real setbacks, and feel a certain sense of risk.
I design encounters by designing an encounter I think will be fun, then add or subtract monsters until it's where I want from challenging to deadly by seat-of-the-pants, then I design in some level of early warning for the PCs and make sure they can escape. A good encounter might not be winnable or might not be escapable, but it shouldn't be unwinnable AND unescapable.
I haven't had a TPK yet, although I've had a couple that came pretty close, and some heated conversations. Externally I shrug and say, "I'll keep your concerns in mind, but we all agreed at the beginning that high danger was possible." Internally I look very carefully at what I did that resulted in something that I didn't expect, so I'll know better for next time.
As a player
As a player, I can't tell you the number of times we've gotten our butts kicked, escaping by a thread, only to re-group and either wipe out whatever it was through planning, or just avoid the encounter all together. We the players have had (good-natured (mostly)) arguments that we bugged out too soon. I'm usually of the school that as soon as it starts to go pear-shaped, you bale and hit 'em from orbit, but I tend to usually be the voice of paranoia.
This approach as players has helped, but it's mostly been after- and between-session conversations, that have allowed us to play the campaign I am currently a player in, into the top ranges of Tier 3, and soon into Tier 4.