Abjure Enemy has value even if enemies get out of it
It is possible that an Abjure Enemy effect will last only one round. It still has value even if it does.
Firstly, many enemies won’t, though
It’s possible for an enemy who has fought or studied paladins with the Oath of Vengeance to be familiar with this ability. If nothing else, the Oath of Vengeance paladins themselves know, so if two fought one another, they would each know it about the other. Subjects of Abjure Enemy snap out of it whenever they take damage; that is a clear and test-able response. While it’s certainly true that characters don’t automatically “know the rules,” this is also the world they live in. Some creatures (immortal forces of evil, perhaps?) have fought paladins a lot, other creatures study the world and its magic and learn things like this, so some characters would know how Abjure Enemies works.
But it’s only some. How many depends on the setting:
In Eberron, where basically the entire continent has been at war for a hundred years and one of the major combatants is a theocracy that leans heavily on paladins for its military, you’d expect most war vets to know this, or at least those that have fought with or against Thrane. Many of the nations of Eberron also have a well-functioning education system that was geared, in large part, to fighting that war—it’s plausible that at least in some cases, the details of paladin’s abilities are close to common knowledge.
In the Forgotten Realms, where information travels slowly if at all, and knightly orders of paladins are elite organizations that must be directly sanctioned by a Good god,¹ and Oaths of Vengeance are perhaps unusual for those, only the most learned scholars and ancient evils might be familiar enough with Abjure Enemies to know this detail.
In Dark Sun, the only “paladins” may be the sorcerer-kings’ personal enforcers, the templar,² in which case it’s entirely plausible that almost no one who has been Abjured has lived to tell about it, and there it might be that no one knows this aside from the templar themselves.
So it varies, but at least some of the time, in at least some settings, yes, characters may know how this ability works and know they can “snap themselves out of it.”
More importantly, Abjure Enemies still has value even if they do
Creatures can’t just “will” damage unto themselves. They have to act to deal damage. Even if they make an attack with an unarmed strike against themselves, and can presumably choose to forgo any special bonuses to that if they’re a monk or whatever, that still burned their action for the turn. Combats only last so long, so losing an action is a big deal. True, you lost an action using Abjure Enemies in the first place, but there are many combats where losing an action to cause an opponent to lose an action is a good trade for you: say, against a boss creature, the sort of thing that is perhaps more likely to know about how Abjure Enemies works.
So against such creatures, you’re still getting value out of Abjure Enemies. Perhaps less than you would otherwise—since it’s up to the enemy to decide whether to just suffer through it or spend an action undoing it—but used well it still valuable.
Finally, talk to your DM about it
Personally, while I think it’s entirely plausible for enemies to at least sometimes know how Abjure Enemies works (and, frankly, implausible for them to not in some cases), and Abjure Enemies is still valuable even if this is the case, I’m still not sure I like it. My issue here is one of narrative—I’m not sure I buy that this response fits into the narrative that Abjure Enemies is portraying. Are frightened enemies likely to “damage” themselves? Does that even really “count”—narratively—as the sort of thing that Abjure Enemies seems to be calling out as triggers to end the effect? To me, taking damage ends the effect because they’re presented with something scarier than your sheer divine-channeling presence, a more realized threat. But if the damage is coming from themselves, it doesn’t seem like it should “count” for that.
The rules totally say it does. And smacking yourself to snap out of it is hardly an outrageous narrative. But it still doesn’t sit entirely right with me. So as a DM, I might choose to rule that someone subject to Abjure Enemy can’t injure themselves to end the effect—either the fear means they wouldn’t do that, or even if they do it doesn’t count. If you feel the same way, you might ask your DM how they feel about it. Perhaps they’ll rule that way too.
I don’t have Sword Coast Adventurer’s Guide, so I don’t know if 5e has changed this. If so, though, that is a massive change from previous editions.
There is no official Dark Sun material for 5e, so I’m guessing about how they would include the paladin. There are other plausible, perhaps even superior options for templars (cleric, as they were originally, or warlock, since the sorcerer-kings are quite like patrons). But for my money, they would be likely to go with paladin for templar simply because there isn’t any other good way to include paladins in the setting at all. The original Dark Sun implementation in 2e just left paladins out, but I don’t think they’d ever do that in modern D&D.
