You don't have to use everything that you prepare
It's been my experience that it is much easier to leave something out than it is to make something up on the spot, so my advice is to, for lack of a better word, "over-prepare" in this particular scenario.
That is, build your encounter as if you're planning a TPK, keep that material on hand, and then leave most of it out at first! If your party starts to just handle things, bring in some of that extra material, and keep doing so until you've figured out the right balance between "challenging" and "impossible".
As to where that balance is, I can't actually offer much advice. That depends on too many factors, such as party size, party composition, players' mentality and tactical capabilities, your own tactical capabilities, and so on. So, you will have to experiment with that on your own.
This approach has the benefit that you won't find yourself underprepared, meaning you can always ramp up the challenge level! And conversely, if your party is less tactical (or more unlucky) than anticipated, then you're still reasonably covered from a TPK!
But how do you make it clear in game that they're not supposed to win?
If the players are practiced, they should be able to size up their opposition from that first encounter and decide "This is a bad idea." But if they're new, or if they're stubborn, then you can outright tell the players that their characters are fighting something out of their league, and that they will (almost) certainly all die if they continue. It's a pretty sensible conclusion that these adventurers would be able to recognize when something is just too much for them, and it also gives players the right information to make an informed decision.
A word of caution
Looking from a player's perspective, this feels like some heavy-handed railroading (that is, the players might feel like their choices and actions in this session don't matter). This can leave a poor impression on your players if they aren't expecting it, and many players don't like to have their agency taken away.
I would suggest letting your players know before-hand that this session will end in a TPK, but that they'll still be able to keep playing these characters. It might ruin the surprise, but player buy-in often helps things go more smoothly as well, in addition to building trust between player and DM. It's been my experience that players are willing to accept less agency when they know that it's part of a story's setup.