The other answers give a good picture on why the spell is, from a mechanical perspective, too weak. I will touch on that too, but also a couple of game design points that might help yiu understand why this spell won't work very well.
From a purely mechanical point of view, "succeed on more rolls to inflict more damage" is a workable concept. The expected damage output should be very high compared to normal cantrips to offset the fact that for normal cantrips, two successes (for three castings) inflict 2/3 of the damage one would inflict on three hits, while the "all-or-nothing" approach of the three word curse would inflict no damage unless all rolls succeeded. (Or failed, from the saving creature's POV)
However, mechanical balance isn't enough to make a cantrip good. I forecast problems relating to the use of this spell in tables.
It should be used early...
Suppose you're fighting a group of nasties and choose to use TWC. Three rounds later, you may have a chance to deal 5d8 damage ...if the combat is still going on. Many encounters last for a surprisingly small number of rounds, so TWC: Nails would probably find itself used mostly as an opener, because it might not get a chance to fire later on and 5d8 against an enemy weakened by the action already would be more likely to be an overkill.
...but early turns are too valuable to waste
Removing enemies from the encounter during the first turns is very valuable in DnD, strategically, because every removed enemy shifts the action economy to favor the PCs. Removing enemies during the first turns is something TWC: Nails can't do, but since it has little use later in the combat too, it's an inadvisable spell to use regardless.
Inaction is boring
This is the worst part to me. TWC: Nails will have the player essentially skipping three turns in order to have a chance to deal relatively massive damage. Assuming the spell dealt enough damage to be mechanically appealing, you'd be promoting an extremely boring style of gameplay to your players --- don't do that. As a general rule, make fun choices strong and strong choices fun, so players don't feel tempted to min-max themselves into something boring.
Poor party synergy
Especially assuming the cantrip gets the damage buff to make it mechanically competitive, another problem sets in --- it's overkill against many types of enemies. To avoid wasting that sweet damage, it encourages the players to not target the enemy targeted with TWC. Reserving enemies for oneself gets stale fast and can cause a lot of tension among the players, and can also result in very swingy outcomes: if the spell fails, you're gonna wish you had put in some damage instead of trusting the dice.
The spell forces an additional bookkeeping need for the players or the GM. It's a minor issue, but still one more thing I'd change.
The spell seems like something that might be a workable concept in a computer game where turns can be executed in mere seconds. However, in DnD, I think even the part where you have to idle during combat to have a chance to deal damage is unacceptable. The spell's concept is neat, but ultimately just a gimmick that can't be the sole justification of its existence.
If you want to preserve the "three strikes", I recommend the following changes:
- make the spell deal some damage on every failed save
- don't require the saves to be consecutive
- the third failed save deals extra damage
- no concentration needed
It's still not my favorite kind of spell, but it's more flexible in its use than before and far less swingy.