A mechanic for party member discouragement?

Question

I am running a campaign in which I expect the PCs will encounter a nothic. I want to use the nothic's Weird Insight ability to have it learn the characters' fears or flaws and use that information to telepathically speak discouraging things to the minds of the PCs who failed to avoid the Weird Insight.

For example, one of our players is a sorcerer princess who fears that she will never fill the shoes of her mother, the queen. The nothic might say "You are far too naive and irresponsible to rule. Though she has erected a loving facade, your inability to control your powers brings nothing but shame and disgust to your mother. If I were to allow you to leave this place alive and you should someday take the throne, you will bring your kingdom nothing but ruin and misery. Fortunately for your subjects, I will not."

I want to then give the PCs an opportunity to make a Wisdom saving throw—if they succeed the saving throw, they are unaffected by the nothic's taunts. Otherwise, I want them to have some kind of mechanical disadvantage representing the nothic's success at discouraging them and lowering their confidence. It doesn't seem like the PHB lists any kind of condition that meets this criteria—is there such a condition in another rulebook? Are there any official rules or guidelines regarding the creation of custom conditions?

Answer 1

Fear, Horror, and Sanity

The DMG offers some options that may work with the effect you're trying to provide to your players.

Chapter 9 covers mechanics like Sanity Ability Score (pp 259) and Fear and Horror (pp 266-7) that could provide the effects you're looking for.

Much of that references the Madness table in Chapter 8 (pp 258-60) for more frightening fun.

The Obvious: Disadvantage*

KSchank specifically asked (emphasis mine):

I want them to have some kind of mechanical disadvantage representing the nothic's success at discouraging them and lowering their confidence.

They also already suggested a WIS save for this effect, so the effect of a failed save could easily be Disadvantage for certain rolls (saves, attacks, ability checks).

*Special thanks to GreySage for this!

Answer 2

Disadvantage

A simple solution would be to impose Disadvantage on rolls either for a period of time or for certain rolls. Disadvantage was designed to be a generic "something makes this hard to do" mechanic, and this seems like a perfect application.

Disadvantage means they roll the d20 twice and take the lower result, and is neutralized by Advantage.

Answer 3

At the outset, I observe that the title of your question is asking about "a mechanic" generally, and most of the body of the question follows suit -- but at the very end of the question, you ask about "conditions." As other answers (in my opinion) already cover "conditions," I'll address the more general question about "a mechanic."

Hit points.

The rules already provide a ready mechanism for measuring the sturdiness of a character's mind:

Hit points represent a combination of physical and mental durability, the will to live, and luck.

PHB p. 198 (emphasis mine). Give the nothic an ability that deals psychic damage on a failed Wisdom save. That damage can represent the nothic's mental battering of a character's confidence.