The rules don’t have an answer for us
There is no answer to this. The rules explicitly indicate that temporary hit points are lost before real hit points, but does not indicate any kind of ordering within either of those categories. In reality, you’re not supposed to have different “types” of temporary hit points; they’re all supposed to be exactly the same as one another, really just a count you track as a single number. They’re not supposed to be divided into categories.
But Drunken Brawler cares, and it isn’t alone. So now we have to worry about when those temporary hit points are lost, rather than just whether or not you have temporary hit points at all.
Suggested houserule: change Drunken Brawler et al. to eliminate this
Up front, I’m just going to say the obvious: this is poor design. Temporary hit points weren’t meant to work like this, and for good reason: trying to track it in any way quickly becomes a real pain in the ass. So I would suggest a houserule saying that all such abilities last so long as you have any temporary hit points. Nothing is allowed to care about “which” temporary hit points are left or lost. Just delete “gained by this effect” from Drunken Brawler’s description. Yeah, you can cast aid to keep your Drunken Brawler going—cool, fine. This is not going to break anything, might even lead to neat interactions, and it’s so much simpler to run.
Rules-compliant rulings: filling the gap without changing anything
Anyway, though, while that’s my recommendation, it’s not what the rules say: the rules say you have to be concerned about the temporary hit points from Drunken Brawler. The rules don’t say when temporary hit points are lost, but if we want to run Drunken Brawler as written, we need to come up with some consistent rule:
Possible ruling: defender’s choice
The person who has the temporary hit points decides which hit points they’re losing. If Drunken Brawler is the only effect that cares about “its” temporary hit points like this, this works the same as my suggested houserule: the person with that feat will (presumably) choose to save those temporary hit points for last, so they can keep the feat’s benefits, and will only lose those benefits when they run out of temporary hit points entirely.
For cases where multiple effects care about different sets of temporary hit points, you basically hold onto at least 1 from each until you’re almost out of temporary hit points. But, for example, if you have 5 such effects, and take damage so that you only have 2 temporary hit points remaining, you have to choose which 2 effects to keep and which 3 to lose.
From a complexity standpoint, this is not too bad. It’s a fairly niche case that won’t come up very often, and only really needs tracking then. Still, when it does come up, it’s going to be painful: that character might agonize of which to keep and which to lose, slowing down the game, and it’s liable to cause issues when a player realizes “oh wait, I forgot this other effect cares about its thp, I shouldn’t have had that effect,” which slows down the game even more.
This is the only other ruling I would consider, aside from my houserule. It’s not bad. But I still think it’s far from worth the effort.
Possible ruling: first in, first out (FIFO)
Here, we have to keep track of the order in which we gained temporary hit points, because we’re always going to lose the “oldest” temporary hit points first. This eliminates the decision point that might slow down the game, but it’s a lot of bookkeeping, and even more liable to allow errors to creep in (“wait, when did I take that drink? was that before, or after, the cleric’s aid?”). It also could lead to feeling a need to “re-up” one’s temporary hit point effects, to make sure they’re ordered the way you want, wasting some of the duration the character would otherwise be entitled to. Your mileage may vary on whether or not that’s a good thing, but it does feel like a deviation from the intent to me, since the character was intended to enjoy that wasted duration.
Possible ruling: last in, first out (LIFO)
The opposite of the previous ruling, where the “newest” temporary hit points are always lost first. Has the same bookkeeping issues and duration wasting issues. Whether FIFO or LIFO is better for a given character will depend on the relative durations of the various effects.
Possible ruling: attacker’s choice
The opposite of the first ruling, the attacker gets to choose which temporary hit points to remove. This is weird and doesn’t make much sense unless the attacker has some way of knowing what effects like this are active on the defender, which they would generally not. It also makes such effects weaker, as they’re easily eliminated.
Possible ruling: random
You could devise a scheme where you randomly choose X numbers from 1 to Y, where X is the amount of damage and Y is the defender’s total number of temporary hit points, and then for each number that is Z or lower, where Z is the temporary hit points granted by Drunken Brawler, you lose one of the temporary hit points from Drunken Brawler (which you’ll need to keep track of since it will change Z for future damage).
This would be absurd. Don’t do this. It would be so messy and difficult to keep track of.