Strong, maaaybe unbalanced early, but fails to scale well
As you've noted, this is a pretty hard area to judge balance in. These cantrips occupy a unique place in the design space.
- They tend to do more damage/have more impact (if their special conditons are met) than comparable ranged cantrips like firebolt or ray of frost.
- This is balanced by the fact that their use is quite niche - if you're an arcane spellcaster you probably want to avoid being in melee since your defenses will probably be weaker than a martial character, and your primary melee attack stat is unlikely to be maxed making the initial melee attack less likely to hit. If you're a martial/martial-ish character who gets access to these (Eldritch Knight, Arcane Trickster, Bladesinger, Valor Bard), the fact that they don't allow you to use Extra Attack or Two-Weapon Fighting with them will make them less attractive options, not to mention if your spellcasting stat is not maxed/high-for-your-level the cantrips that use the spellcasting stat will be less good.
At levels 1-4
In this level range, it's comparable to Green-Flame Blade:
- Both have an extra effect keyed off spellcasting modifier
- No special requirement (like GFB's "another enemy within 5 feet) to activate it's extra effect
- The temp HP is useless if you don't take damage in the next turn, while extra fire damage to another enemy has a permanent effect on their HP
I think I would say that if you're using this cantrip in melee, you're going to get hit, and make use of the temp HP. How big a deal is this temp HP, then? Well, False Life is a 1st level spell that gives you on average 6.5 temp HP for an hour, no concentration. This cantrip gives you 3 temp HP (assuming point buy & min-maxed casting stat at level 1), at an opportunity cost of not dealing 3 extra damage to an enemy who may or may not be present. At level 1, 3 temp HP per round is a lot. Goblins, a CR 1/4 enemy who are often faced at level 1, deal an average of 5.5 damage on a hit. This temp HP cuts this damage in half, assuming you only take one hit per round. And a wizard/sorcerer/warlock will have 5-9 HP at level 1, making this temp HP between 50% and 33% of their total HP pool. That's a lot.
But maybe this is balanced by the fact that spellcasters really should not be in melee if they don't want to die at this level. Characters with stronger reasons to be in melee like Eldritch Knights and Arcane Tricksters only get this cantrip at level 3, by which time the temp HP is less of a big deal.
Now the cantrip gets a lot weaker. GFB & Booming Blade get +1d8 to both their initial damage and their extra effects, while Vampiric Assault only gets +1d6 to the initial damage. Now granted, you will always be able to use Vampiric Assault's extra effect (unless you have temp HP from another source), but as levels scale up 3-5 temp HP will become less and less relevant.
The cantrip seems ok at levels 1-4. Possibly a little stronger than the other similar cantrips, but not so much I think it definitely needs changing. I think if you're using this in a game you should warn your players you may need to tweak this (perhaps with an agreement to let them switch cantrips if they're unhappy with the tweak), and watch how it does at these levels.
Beyond these levels, it needs a boost in its scaling.
I would suggest following the example of Green-Flame Blade and having matching scaling to both the initial damage and the extra effect (in this case, temp HP). My instinct is to go for d4s for this scaling, but that's just based on a gut feel of the impact of lots of temp HP every round at higher levels. I may be overestimating the power of that temp HP, in which case d6s would be appropriate.