Straight answer: Put in gatekeepers.
The easiest answer is to simply TPK them, tell them it was because they were unprepared, and let them try again. After all, that's what happens in video games all the time. It looks like you're not satisfied with that approach, though.
Therefore, we can borrow something else from video games: gatekeepers. In many games, there are frequently highly frustrating gatekeepers or locks that keep you from progressing until you have a certain ability or equipment or whatever. This forces players to do whatever quest they need to do first. Without such gatekeepers, players will blithely forge on ahead. Consider games like Zelda, Metroid, Pokemon, etc. etc. As you have discovered, without a hard stop like this, players will often ignore requirements and go straight for their goal.
In game, this could take various forms: magical barriers that can only be broken by certain items, impassable crevasses that require special wings to traverse, or even a guide that refuses to take the party somewhere before they're equipped. Importantly, it can't be something that the party can just kill through sheer force of will.
As an aside, such gatekeepers let the players know that they are indeed appropriately geared for the upcoming fight. Otherwise, they will only know that they haven't upgraded their equipment enough while they're getting murdered by the boss.
Your players might hate this tactic, and it's definitely railroading. But if you really want your vision to work, you might be forced to implement such obstacles.
Frame Challenge: You have a party that wants to play an evil campaign.
Honestly, it seems like your problem is less with the mechanics than with the style of campaign.
It looks like your players did get the message about improving their equipment, but they decided to get the monster parts through another method (plundering that caravan). They aren't totally unwilling to fight--they're just going for easier targets. Your comment that you had this same issue in Call of Cthulhu shows that this problem is totally independent of any homebrew game mechanics.
The usual suggestion here is to look at the same page tool, but your players have already definitively told you that they want to be evil. In fact, I suspect that you already have all the information you need to make a campaign that they would find entertaining.
One of my groups has had a similar experience. After some decidedly evil acts, everyone (including the DM) decided that we would start an evil campaign. In reality, it was pretty much the same structure and mechanics (kill this thing, find this item), except that our quest givers had evil aims and we would knock down the occasional village. In an evil campaign, perhaps your players can get upgrades by raiding traders and caravans, and their fights would not be against monsters in the wild but with the increasingly powerful guards that caravans would start hiring.
Most importantly, if you're not willing to run this kind of evil campaign, you are not the right DM for that party right now.