The 3 PCs enraged the main quest giver in my campaign by the PCs threatening to kill him and doing things to try to get more money out of him. They are level 1 and could easily get TPK'ed by this NPC. The main quest giver is part of my campaign's thief guild and has no mercy, but I don't want to TPK my party in the second session. What do I do to put them back in their place?
Potentially consider non-lethal ways of demonstrating consequences
Letting it slide will diminish his credibility, and a TPK is definitely not ideal. Demonstrate his clear superiority with a cutscene moment if need be, but if they absolutely insist on fighting I'd let them.
In a current game we are playing, our party became indebted to some very bad people, essentially a thieves guild. They're forcing us to perform jobs for them. This turned an impossible fight into one of the driving factors for the story.
You don't have to have the "take a family member hostage and force them to obey thieves guild", but something that shows actions have consequences when you're low on the pecking order is a good idea.
You control the NPC and there are several reasons for a generic NPC to avoid attacking.
Even if your NPC has "no mercy" and is enraged there is any number of reasons why the NPC would not attack them. Most people have been enraged more than once in their lives without committing multiple murders. For one thing, cleaning up bodies is messy under the best of situations, and even more so if you need to do so in a way that will not get you tied to multiple murders.
Not to mention that once you step outside of the mechanics of D&D, fights are dangerous and unpredictable, especially against multiple opponents. 5e's bounded accuracy was introduced partially to help bring that back into the rules. If your NPC is a (demi-)human, they should not be all that confident, within the story, that they will walk away completely unscathed from a fight against a whole party, no matter how high level they are because combat is chaotic. You haven't said how the party enraged your NPC, but if it is something petty the danger to your NPC, at least within the story if not mechanically, should make them think twice about fighting several enemies.
Since there are multiple reasons for your NPC to not create a total party kill, the question is how to move forward and that depends on the details of the situation.
Talk to your players and consider changing course.
Depending on the situation, this could be an implicit signal from your players that they do not like this NPC or the quest he is offering at that point there are a couple of general ways to address this. One is to follow their lead, change directions, and provide a different quest giver with a different quest. This one could fade into the background for now, perhaps to bring out a more subtle vengeance later. Another is to go out of character and discuss the situation with them. Especially when the group is new there is nothing wrong with asking directly what they don't like, why, and how they would like to see it changed.