How, exactly, is a new creature playtest conducted? [duplicate]

Question

Many questions on this stack that are primarily about CR Calculation point to playtesting as the final step to homebrewing a creature. That, merely calculating the Challenge Rating isn't enough, the creature shall have to see play for it to be gauged more-or-less accurately. @DaleM puts it nicely in this answer:

Think of the CR calculation as trying to work out how to fly a spacecraft to the moon with calculations on the back of a napkin.

Now that's all well and good, I have gotten to the habit of painstakingly playtesting my creatures recently (and have bugged several stackfolk to do it with me). Though we've been playing, getting feedback and tweaking, I feel there is lack of process and am missing something.

Which brings me to ask: how, exactly, are playtests done- in terms of determining a creature's CR?

Specifically, I'd like to know if creatures were playtested assuming a certain number of them. For example, if I made a CR1 creature, do I pit one against a 1st level party of four? or do I pit two against an 3rd-level party of four? and so on...

Answer 1

I will assume the playtest was actually done by playing the monsters against the variety of players, classes, parties etc available during the D&D Next phase.

Playtesting provided by
over 175,000 fans of D&D. Thank you!

MM page 3

That is the only reference I found on the MM about playtesting.

I still find that information to be vague, so I would advise to test the monster with the guidelines found on the DMG about creating encounters, and see how the combats feel to different levels and number of players. An encounter build to be easy should be easy, as well as a deadly encounter should be extra difficult and put the PCs in real danger.

Testing the monster (or party of monsters) against a balanced party and seeing how it feels (Easy, Medium, Hard or Deadly) should give you a good indicator of the monster's CR.

these guidelines can be found on DMG page 82

Answer 2

In my experience, playtesting isn't about having an awesome game. But having a game that gathers data to make future games awesome. So your players will need to buy into the goal, and be willing to have after-encounter discussions about what they liked or didn't like about the creature, how they think it could be improved, how hard or easy they thought the encounter was, and so forth.

Start of with as vanilla, plain, ordinary a set of by-the-book characters as you can. These should have standard stats and abilities straight from the PHB, no special powers gained or overpowered items. No custom or add-on rules, nothing that makes your character more powerful than the typical, generic, characters that come with published modules.

Level them up to what you think the creature's CR ought to be, but again, make sure they are text-book advancements. Use only things affordable off the starting equipment at higher levels rules from DMG, for example.

Then have your test party go up against the monster(s) in fairly generic ways, more than once. With each encounter, rotate who plays what class, so your data will reflect the different playing styles each player brings to that PC class.

Track how hard it is to overcome over a number of fresh encounters with a published monster of the same CR you believe your creature should be -- full health and full spells at the start of round 1, etc. Use hard data, not just subjective questionnaires, so maybe track things like:

• Percentage of hits scored by PCs
• Percentage of hits scored by Creatures
• Damage dealt by PCs
• Damage dealt by Creatures
• Rounds to completion of encounter
• Did the PCs win or lose?
• PCs reduced to zero or lower HP during encounter (How many? How fast?)

(Or whatever metrics will help you align the creature with your expectations.) This is your baseline, so you may want to find a creature as similar mechanically speaking to yours as possible. Find similar attack styles or abilities, etc. to remove as many variables as you can. Fight this encounter with your party at least once or twice, but maybe 3 or 4 times, gathering data to use as your baseline.

Then do similar encounters with the same PCs against your monster. Again, repetition may help improve your dataset here.

If your creature went down too fast or too slow relative to your baseline, then adjust the CR up or down. Or change the monster stats. Then reset and try again.

Ask for subjective feedback, too. But your primary focus for setting the CR should be baselining your beast against published beasts.

(Not that published creatures are perfect. Every gaming group approaches encounters with different tactics...)

Answer 3

Officially, they are/were playtested in normal game scenarios

I'm trying to prove a negative here, so it's very possible my answer will be supplanted by someone who does find something.

I read a few interviews with Mearls, and looked at the playtest adventures. It seems to me that the playtested monsters were dropped into adventures, and WoTC simply solicited questions about the system overall, instead of focusing specific tests on monsters themselves. (Note that I can't access the surveys themselves, which likely had more detailed information.)

This means that the monsters were found in varied environments and varied numbers, such as bandits in their hideouts or a single ogre in a cave. I can't find any evidence that they did systematic, artificial, "monster in a featureless box" testing. Therefore, I conclude that monster playtesting was conducted alongside the rest of the playtesting as a whole.

Additionally, to address a sub-question, DMG 274 states,

A single monster with a challenge rating equal to the adventurers' level is, by itself, a fair challenge for a group of four characters.

Thus, the system assumes that a "standard" single monster fight is 1 monster against 4 adventurers.

Playtesting on a Spreadsheet

I don't have experience writing monsters for general adventures, so I won't address that area. I have found that simulating new monsters against the party I'm DMing is very useful. In particular, I made an excel sheet with player and monster statistics in order to calculate various probabilities, such as the likelihood that players will hit the monster, or that the players will make the monster's saves (or vice versa). This lets me tweak the monsters more directly, and is a useful, party-specific intermediate step between CR calculations (which are general) and actually playing out the fight. Additionally, I find that having automatically calculated probabilities is far easier than actually playing out the fight repeatedly, which is time consuming and not necessarily representative.

Answer 4

With Unearthed Arcana being as popular as it is, it generates a lot of questions, which is fortunate for you because Jeremy Crawford sometimes tweets about both the design and playtesting process:

• We have several hundred playtesters who sign NDAs and provide feedback before we publish something.
• Unearthed Arcana surveys stay open for 3–4 weeks. We get feedback from them, our playtesters who've signed NDAs, and our own play.
• UA material tests a concept. If the concept passes, we then work to get the balance right. You're seeing the natural playtest process.
• You can tweet to us, but the survey is the only way to guarantee we see the feedback.
• We test what's written. Our books don't come with a copy of me to explain them. The text needs to stand on its own as much as possible.
• In the weeks ahead, we'll avoid answering rules questions on Unearthed Arcana. We'd like feedback on the text as written.
• Telling us a rule is unclear/incomplete is one of the most important types of feedback, after telling us whether a rule is fun.
• A class should be fun to play and for a DM to witness. If a new class is a headache for most DMs in playtesting, we change or cut it.
• Class fantasy overrides almost everything else in our design. To us, a class must fulfill its archetype.
• It was an experiment—give the monk nearly every weapon at a steep cost. It hasn't tested well; it won't endure.

In short, they come up with an idea and then collect feedback from a large number of players through surveys, as well as a smaller number of players they collaborate more closely with, to see if the idea is worth refining. They also try it out themselves. When deciding whether to proceed, they take into account whether it was easy to run and fun for everyone involved.

Unfortunately none of this directly addresses playtesting monsters, and new monsters usually don't show up in Unearthed Arcana, so I assume they're relying mainly on their NDA players. It's possible some of the monsters get tested indirectly when they're playtesting specific adventures, but I doubt they'd be able to test every monster that way. I imagine at least some of the time they simply put together multiple encounters for the same monster (taking into account which other monsters it's likely to fight with) and have their players fight them using different party setups.

A DM running games as a hobby doesn't have the luxury of an army of playtesters, but the bar for success is also set much lower. You don't have to worry so much about potentially confusing text because chances are you're the only one reading the stat block. You also don't have to worry about getting the balance exactly right on the first try, because you can always fudge the numbers or have a backup plan (e.g. an NPC shows up mid-fight to help). On the other hand, paying attention to any mechanics that might've slowed down play or frustrated players remains just as important. However you have the advantage of knowing who's going to be fighting this creature, what resources are available to them and what their preferred tactics are.

The bottom line is that you should at least run through some scenarios in your head or on paper privately as a sanity check. If you can get some other people to try out your idea, even better. But you should still be prepared to make adjustments on the fly during a real game; your players may surprise you.