What can I learn from playtesters?

Question

As a roleplaying game designer, I'm interested in discovering the range of useful feedback that can be solicited from external playtesters. Assume a rule-set and playtest document is all the contact I will have with the play group.

If you've playtested, or facilitated playtests, or parsed data returned from playtests, what form did it take and how was it analyzed and implemented? What information was useful/actionable and what was not?

For example, I always ask about the rules as a procedural document - clarity, organization, and usability. I always ask about the experience of play broadly, and there are usually very specific things I want detailed feedback on which I will call out.

More broadly, what sorts of information could be returned and practically used to improve a game's design?

Answer 1

I tell playtesters to:

• Play to enjoy themselves. (This is because some people play to "break the game", which I don't like)
• Play it the way I've written, even if you think I should have written it differently. (I've had playtesters who change something I've written, run it their way, then tell me their way works. It's not helpful.)
• Be honest. I need criticism. However, if they just had a good time and enjoyed it, don't feel they must find faults.

In their reports, I like:

• A summary of what happened: both the fiction and the mechanics.
• As much detail as possible. (One of my best playtesters sent me my playtest PDF, with comments attached to each section.)
• How the players generally felt about the game.
• Any parts that felt awkward. For example, if the Investigators got to a library and just didn’t know what to do next.
• Any parts that were boring.
• Anything the person running the game just did not understand.
• Any mechanical things that didn't run well.
• Anything that didn’t feel right. For example, if the Investigators escaped on a plane, but they didn’t understand why there was a plane there.
• Anything that was factually wrong. For example, if a radio played the BBC World Service, but they know the BBC World Service wasn’t broadcasting in 1935.
• Anything else they would like to help them run the game.

I don't like:

• Numerical scores. Rating the game session, or parts of it, out of 10 doesn't help me.

I rarely like:

• Suggestions on how to improve the adventure. Certainly, I don't want anyone to rewrite sections for me. (The reality, here, is that I only like these suggestions if they're good.)

Answer 2

The single most important thing you can learn from outside blind playtesters: Is it intelligible?

There are two very good "blind test" modes:

1. hand the playtest GM the rules, and have him run a group of players and report
2. hand the playtest GM the rules, and, never once answering a rules question, play in his game with his players.

Mode 1 is less accurate at conveying information to the designer, but more playtests can be done, and what is reported is more important.

Mode 2 is more accurate (you KNOW what is being done in ways you didn't intend), but much harder to do (tendency is to look to the designer for answers in play, chopping the GM's authority), and much more time intensive for the designer.

Note that running your own game isn't really playtesting. You're not testing the rules, but the ideas you've tried to put into the rules. Best test for my own RPG designs was putting them in other's hands and letting them go try on their own. Brad running RP2 for Mark and me convinced me it wasn't ready for sharing... not even over WWIVnet. While I've used it many times for Trek games, it's just never made it to sharable, let alone salable.

Things outside playtesters are good at finding and reporting:

• omissions by the author
  • typically, things the author and his player group simply "know" get overlooked
  • dropped paragraphs and/or sentences sometimes render others less intelligible
• areas that don't do what the designer claims they do
  • in the setting especially, if the mechanics do X, and the players can't see setting issue Y because of mechanic X.
  • In rules
    • misstated modifiers (usually wrong direction)
    • dice mechanics broken
    • point costs (if applicable) over- or under-value some mechanical element.
      • note that this may be all over the map; if, with a decent number of playtesters, shows a consistent trend, it's probably true
      • if the responses are all over the map on it, it's probably close enough.
    • combat or skill systems have
      • counter-intuitive results - but real combat sometimes does
      • counter-realistic results - noting that some realistic results are counter intuitive
      • playability issues, like time to calculate, or unfun mechanics.
• areas where players need more info
  • often, this is in the mechanics
    • How do I use ?
    • Why can/can't I do ?
  • In the setting,
    • mostly about what they liked - they will want more
    • often, questions about "Why is this area doing this?" - not all need be published, but all such answers should be noted.
• areas for supplementation
  • Rules: often times, players will make not of entire chunks of rules that they want to see, but are not present.
    • Sometimes, this is a playstyle issue, revealing differences between designer and playtester
    • Sometimes, this is a rare-but-useful block, like Mass Combat or Enchanting, where it can wait for a supplement to keep the core down
  • Setting: if only some playtesters want more details on a given area or element, it's probably better left to supplements; if the majority want it, and it's not too long for the book, put it into the core.
• writing style
  • Clarity and intelligibility: can they understand it?
  • accessibility & organization. Can they find what they need when they needed?
  • arguability: if the players don't agree on interpretation, even tho each sees it as terribly clear, it's an issue that tends to get reported.
• Art
  • is art needed to illustrate some point?
  • does that needed illustration actually show what it intends to?
  • is it still current to rules?

Some advice on playtesters: always try to get a mix; some of the most litigious and some of the most story oriented, in combination, quickly show via reports where it works or doesn't. If you get lots of picayune details caught by playtester rules-lawyers, in play, the rules lawyers won't be snarking at the GM over them...

Also, it really helps to have some outside readers who don't playtest... because playtesters are notoriously bad at catching spelling and grammar issues that don't in themselves muddle rules, but consumers often will bitch and moan.

Answer 3

I think what most designers miss measuring with their play tests are some of the most important 'fuzzy' metrics:

• What was the most fun (event or mechanic)?
• What was the most complicated?
• What did you think was the most clever or surprising thing?
• What was the most boring?
• What do you remember most from the session?
• If you could change one thing, what would it be?

I like to watch body-language and facial expressions for signs of boredom, confusion, distraction, surprise, delight, and especially for game events that cause a radical shift in a players emotional state (either direction).

Sure, there are those who think they know a better way to generate random numbers, or how to fix a broken combat or auction mechanic - but the true sign of success is how the game engages them personally.

Answer 4

I can't find the original list, but there is a set of questions every RPG designer should know the answers to. From memory: 1. What is your game about? 2. What do the PCs do? and 3. How to do the rules support 1 and 2? You should know the answers to these. In playtesting you want learn how true #3 is. What you claim your game is about is nice, but if the rules disagree, the players will quickly detect this (probably subconsciously) and begin playing the game you actually designed.

For example, maybe you designed an exciting game of pulp adventure in the 20s. You're looking forward to feats of daring do! It starts off well enough, but after a bit your players grow cautious, avoiding the very daring do you wanted. Oops, turned out that your system actually discourages taking exciting risks, making the cautious route more effective.

So, in your play tests keep in mind your answers to the above. Compare what you want and expect the players to do with what they actually do. This really requires watching or at least listening to the players, it's hard to see in post-session reports. If you can't be physically present, a good audio recording will likely help. Finally, be ruthless with yourself. It is very easy to be defensive, to blame the playertesters for somehow overlooking something. Maybe for one or two players in a group. Maybe even one group out of several playtest groups that group missed something key. But if it happens more than once you probably have a disconnect between #1/2 and #3.

Answer 5

I've never play-tested systems, but I have play-tested some modules ("adventures", "scenarios" or whatever you want to call them). This has basically been the author (or publisher) giving me a print-out of the module in question (this was back in the 80s, so "printer at home" or indeed just "email" was not really something you'd expect to be common) and me reading through the text.

While reading, I'd make notes on things I found lacking (things like "you have an underground military base, but there are no toilets and no showers, what is the reasoning behind this?" and "there's no real incentive for the characters to leave the public house in the middle of the forest, to set of in search for the McGuffin"), then gather a group of players and have a go at the scenario.

While playing through, I'd (again) make notes as to what seemed to work or not. At the end of the playtest, I'd make notes in the scenario and a debrief for things that don't fit neatly into a specific location. I tended to also have a verbal debrief with the designer or line editor.

Answer 6

What are you trying to playtest, in particular?

Start with knowing what you are looking to test, in this particular playtest run. Obviously, other things may come up, but know what you're looking for as the primary thing to be tested - core mechanics, strategies, polishing edges of rules, flow, etc.

You may want to tell the group what to focus on, as well, since that will help them be more mindful during play, and take notes on that.

You mention having only the rules sent to the group as the form of contact. This would either be late in the playtest setup or with people you know. Early on, you want to test core rules, etc. and you don't want to spend a lot of time dealing with miscommunication or pedantry.

Fun and Flow

First, I look for groups to report what was fun, and how things flowed. This isn't a particularly deep or specific kind of thing, but you're looking for general signs to see if their experiences are matching up to your goals.

It gives you a bit of an angle to how you read/listen to their playtest report - if it matches up, then you're looking at technical solutions and polishing of bits. If it doesn't match up, you either have serious design problem or serious problem in communication about the design - so you have to step back from looking at details of mechanical assessment and more towards a broader picture.

Challenges

What rules were unclear? What parts did the group get lost as to "what to do next"? What felt hard to run? Did anything seem unfun?

These again may be design solutions, or communication solutions. This is also where you end up throwing a good amount of feedback out. "Your game about tactical fighting combat didn't have enough rules for romance!!!" and other such things come up, where the only thing you can do is make sure you're communicating the goal of your game well. (Vincent Baker talks about this as the Object of the game.)

Nitty Gritty

The way to pick apart specific mechanics issues is to follow this line of thought:

1) "The thing I wanted to do in the game is X. I tried to do Y to make that happen." 2) "The mechanics to do Y make it (easy, hard, a tough choice, are too complicated, no choice at all, etc.) as a process, and those mechanics make me want to do Z."

Your questions/discussions with your playtesters basically looks to pick this apart. Understand that, unfortunately a good number of gamers are not particularly adept at this.

Example: "I wanted to have a strong fighter-character. I tried to take this set of Special Sword Skills to make that happen. The mechanics to use Special Sword Skills use up my Hit Points, which makes me actually not want to get into fights after using it."

Once you have that, you need to think for yourself if Goal X fits with what your game is supposed to be about in the first place, if Method Y even fits with Goal X, and if result Z is also in line with any of the aforementioned concepts.

Sometimes this means better communication, sometimes you just have gamers who want your game to be something else, and you're not going to get much value from that communication.