1. Understand what Challenge Rating is and is not.
You're already on the right track here, as you have observed that there are some situations where Challenge Rating magnificently fails to capture the danger a monster or group of monsters represents. The first step in recognizing monsters that can lead to problematic encounters is understanding exactly what Challenge Rating does represent. Fortunately, the Dungeon Master's Guide gives thorough guidance on this in the chapter 9 section Creating a Monster.
The short version is that Challenge Rating is based almost entirely on the raw numbers - hit points, accuracy, damage, armor class, saves - with guidance for converting some monster features that aren't explicitly numeric into numeric equivalents that you can factor into the calculations.
This is all Challenge Rating is trying to do.
As you have observed, there can be monster features and encounter designs that are not fairly represented by Challenge Rating, and this is usually because one or more monsters in an encounter have features that do not translate into numbers we can include in the calculation. We will examine a few concrete examples of this later.
To give a concrete "action item" here for getting toward a solution to your problem, I would first recommend: become very familiar with the "Creating a Monster" section in chapter 9. Study the moving parts of challenge Rating presented there, so that when you are designing encounters, you will be able to point out features that don't fit into that paradigm, because these features will be the first place to look for potentially problematic encounters.
2. Vanilla monsters are typically not going to be an issue.
This point follows naturally from the exposition above: if the Challenge Rating methodology accounts for all of a monster's features, don't worry about it. This is probably obvious, but still worth observing. When determining if an encounter may "punch above its weight class", don't spend too much time looking for issues with monsters that don't have any non-numeric features. If a monster just hits stuff, the Challenge Rating is probably accurate.
3. Know your player characters.
This one is really important: you need to know the characters really well. You need to have an understanding of what they are capable of and what their weaknesses are. One encounter might present an insurmountable obstacle for one party composition, and may be trivial for another party of the same size and level, but different composition.
For example, in one of my games, we recently traversed the Caves of Hunger beneath the frozen wastes of Icewind Dale. I won't give too many details because spoilers, but there is one encounter where the entire party may be subject to a powerful charm effect. Without a reliable method of dealing with this, this encounter can be incredibly difficult, possibly even a TPK. However, I was playing a Twilight Domain cleric, who has a handy feature called Twlight Sanctuary:
Whenever a creature (including you) ends its turn in the sphere, you can grant that creature one of these benefits:
- You grant it temporary hit points equal to 1d6 plus your cleric level.
- You end one effect on it causing it to be charmed or frightened.
My ability to just end the charm effect for free trivialized this encounter for us. Without this ability, we would have been in huge trouble. If you have a good awareness of what features the party has access to, you can keep those things in mind as you design encounters. You can ask yourself, "Does the party of any means of countering this?" If there is a monster feature the party has no counter to, then that is a feature you need to give more consideration to as you build your encounter.
4. Monsters with Paralyzing, Stunning, and Dominating features are automatically suspect.
These are features that you can assume without too much analysis may present trouble for the party. Both Paralyzed and Stunned inflict the Incapacitated condition, and will generally completely remove a character from the combat for at least one round, maybe more, especially if the effect targets a character’s weakest save. If you have designed your encounter around four characters, but one of them ends up stunned for two rounds, that alone can be enough to overwhelm the remaining characters if the encounter was already going to be difficult without considering the incapacitating effects.
With this, I would challenge something stated in the question:
For example on CR 1 the Brown Bear was much deadlier than a Ghoul.
Brown bears are simple beasts. They scratch and bite, and as long as someone in the party can eat those hits, there won't be any surprises in terms of encounter difficulty. On the other hand, the ghoul, while weaker “by-the-numbers” than the brown bear, has this nifty feature:
Claws. Melee Weapon Attack: +4 to hit, reach 5 ft., one target. Hit: 7 (2d4 + 2) slashing damage. If the target is a creature other than an elf or undead, it must succeed on a DC 10 Constitution saving throw or be paralyzed for 1 minute. The target can repeat the saving throw at the end of each of its turns, ending the effect on itself on a success.
If the ghoul manages to stick one of these to one of the characters, that character is useless for at least one round, and paralysis has the added complication:
Any attack that hits the creature is a critical hit if the attacker is within 5 feet of the creature.
Paralysis can make the party tank quite squishy once attacks against them have advantage and hit twice as hard. One failed save can turn an otherwise pedestrian encounter into a full party wipe.
Creatures with dominate X features have the same problem, but make the issue worse by giving the dominated characters actions to the enemy for at least one round.
5. Petrification should be a plot device until at least late Tier 2.
Petrification presents the same in-combat problems as Stunned and Paralyzed, but typically has the added bonus of being semi-permanent. Darth Pseudonym asks about this in their Q&A: Petrification at low levels. The problem with petrification is that the players generally won't have an immediately accessible means of curing unless they have a cleric and are at least 9th level for access to greater restoration:
You can [...] end one of the following effects on the target:
- One effect that charmed or petrified the target
Prior to 9th level, the party cannot cure petrification unless you give them a means to do so. Here's how I handled it:
In one of the games I ran, one of the members of the 5th level party was petrified by a basilisk. At the end of that session, I had a conversation with the table about what they wanted to do about it:
Thomas: Okay guys, Chris’s character is petrified, which for the moment, is basically death. Now, Tessa’s character (the cleric) would know that the temple in the capital provides the kind of care that can reverse petrification, but they usually require some work to be done. So your options here are: Chris, you can just roll a new character, or, if the rest of you want to, you can pursue getting Chris unpetrified.
The party decided to try to get Chris cured. Upon arriving at the temple and talking with the chief priest, they were tasked with accompanying one of the temple’s paladins to consecrate a wilderness temple that had been desecrated and had a necromancer squatting in it. I used this paladin as Chris’s temporary character, and upon returning from the quest, Chris’s character had been cured of petrification.
If the party has greater restoration, then petrified is just another status condition that they can cure with what they have at their disposal. If they don't have it, then it is a candidate for creating perilous situations much like Stunned and Paralyzed, with the additional potential effect of changing the direction of the campaign for many sessions.
6. So what do you do with all of this? Playtesting.
This will add a good bit of time to your prep work. If you really want to know if an encounter might present the problems you ask about in the question, you have to get your player's character sheets and run the encounter yourself. I've done this a lot, because I like to run combat-heavy games. I enjoy combat, and have generally played with tables that feel the same way. Where my game preparation may lack in world building and NPC development, I make up for it in encounter design by spending time running encounters at my kitchen table with dice, minis, my players' character sheets, and my Monster Manual. This is why the other things I've talked about are really important: understanding Challenge Rating, knowing your player characters, and watching out for debilitating effects can save you a lot of time, since those things give you an idea of what encounters need playtesting, and which ones don't.
