Take the 2-minute tour ×
Role-playing Games Stack Exchange is a question and answer site for gamemasters and players of tabletop, paper-and-pencil role-playing games. It's 100% free, no registration required.

As a roleplaying game designer, I'm interested in discovering the range of useful feedback that can be solicited from external playtesters. Assume a rule-set and playtest document is all the contact I will have with the play group.

If you've playtested, or facilitated playtests, or parsed data returned from playtests, what form did it take and how was it analyzed and implemented? What information was useful/actionable and what was not?

For example, I always ask about the rules as a procedural document - clarity, organization, and usability. I always ask about the experience of play broadly, and there are usually very specific things I want detailed feedback on which I will call out.

More broadly, what sorts of information could be returned and practically used to improve a game's design?

share|improve this question
add comment

5 Answers

up vote 12 down vote accepted

I tell playtesters to:

  • Play to enjoy themselves. (This is because some people play to "break the game", which I don't like)
  • Play it the way I've written, even if you think I should have written it differently. (I've had playtesters who change something I've written, run it their way, then tell me their way works. It's not helpful.)
  • Be honest. I need criticism. However, if they just had a good time and enjoyed it, don't feel they must find faults.

In their reports, I like:

  • A summary of what happened: both the fiction and the mechanics.
  • As much detail as possible. (One of my best playtesters sent me my playtest PDF, with comments attached to each section.)
  • How the players generally felt about the game.
  • Any parts that felt awkward. For example, if the Investigators got to a library and just didn’t know what to do next.
  • Any parts that were boring.
  • Anything the person running the game just did not understand.
  • Any mechanical things that didn't run well.
  • Anything that didn’t feel right. For example, if the Investigators escaped on a plane, but they didn’t understand why there was a plane there.
  • Anything that was factually wrong. For example, if a radio played the BBC World Service, but they know the BBC World Service wasn’t broadcasting in 1935.
  • Anything else they would like to help them run the game.

I don't like:

  • Numerical scores. Rating the game session, or parts of it, out of 10 doesn't help me.

I rarely like:

  • Suggestions on how to improve the adventure. Certainly, I don't want anyone to rewrite sections for me. (The reality, here, is that I only like these suggestions if they're good.)
share|improve this answer
    
If they're not playing to break the game, then how are they being honest? People will play to break the game eventually, you might as well face that from the start. –  Lokathor Oct 20 '10 at 22:19
    
We're probably talking about "playing to break the game" in different senses. Some playtesters think it's helpful to find holes in the rules. I would prefer they played to enjoy themselves and merely told me about any rules problems that came up. –  Graham Oct 21 '10 at 0:50
1  
I want the basic engine to work well before I turn it loose on strangers, so my plan is to pre-break anything that is fragile or dumb with my local playtesters. If external testers find broken stuff, that's awesome, but it isn't a goal because I've already done my due diligence there. Asking someone to play to failure is asking them not to pay attention to the experience of play as well. –  Jmstar Oct 22 '10 at 13:39
1  
@jmstar: While I've not playtested your stuff, I've heard the same line from several I have playtested for... and in all cases, they've missed something. –  aramis Nov 2 '10 at 20:48
1  
@Graham - I think that both questions need to be examined: are the regular players having fun, and are the power-gamers finding holes to exploit? –  Allen Gould Jun 20 '11 at 15:04
show 4 more comments

I've never play-tested systems, but I have play-tested some modules ("adventures", "scenarios" or whatever you want to call them). This has basically been the author (or publisher) giving me a print-out of the module in question (this was back in the 80s, so "printer at home" or indeed just "email" was not really something you'd expect to be common) and me reading through the text.

While reading, I'd make notes on things I found lacking (things like "you have an underground military base, but there are no toilets and no showers, what is the reasoning behind this?" and "there's no real incentive for the characters to leave the public house in the middle of the forest, to set of in search for the McGuffin"), then gather a group of players and have a go at the scenario.

While playing through, I'd (again) make notes as to what seemed to work or not. At the end of the playtest, I'd make notes in the scenario and a debrief for things that don't fit neatly into a specific location. I tended to also have a verbal debrief with the designer or line editor.

share|improve this answer
add comment

I can't find the original list, but there is a set of questions every RPG designer should know the answers to. From memory: 1. What is your game about? 2. What do the PCs do? and 3. How to do the rules support 1 and 2? You should know the answers to these. In playtesting you want learn how true #3 is. What you claim your game is about is nice, but if the rules disagree, the players will quickly detect this (probably subconsciously) and begin playing the game you actually designed.

For example, maybe you designed an exciting game of pulp adventure in the 20s. You're looking forward to feats of daring do! It starts off well enough, but after a bit your players grow cautious, avoiding the very daring do you wanted. Oops, turned out that your system actually discourages taking exciting risks, making the cautious route more effective.

So, in your play tests keep in mind your answers to the above. Compare what you want and expect the players to do with what they actually do. This really requires watching or at least listening to the players, it's hard to see in post-session reports. If you can't be physically present, a good audio recording will likely help. Finally, be ruthless with yourself. It is very easy to be defensive, to blame the playertesters for somehow overlooking something. Maybe for one or two players in a group. Maybe even one group out of several playtest groups that group missed something key. But if it happens more than once you probably have a disconnect between #1/2 and #3.

share|improve this answer
3  
When using a playtest model like this, it is also worth asking what RPGs the playtesters are used to or have played most recently. It may be that your rules produce the play, or it may be the playtesters bending the game into a more familiar shape. I've seen the latter happen a lot with even tested-and-done published RPGs. Knowing where your playtesters are coming from is invaluable in analysing playtest reports where they've diverged from the game you thought you gave them. –  SevenSidedDie Nov 1 '10 at 21:18
2  
    
Thanks for the cite, SevenSidedDie. –  Jmstar Nov 4 '10 at 13:26
add comment

I think what most designers miss measuring with their play tests are some of the most important 'fuzzy' metrics:

  • What was the most fun (event or mechanic)?
  • What was the most complicated?
  • What did you think was the most clever or surprising thing?
  • What was the most boring?
  • What do you remember most from the session?
  • If you could change one thing, what would it be?

I like to watch body-language and facial expressions for signs of boredom, confusion, distraction, surprise, delight, and especially for game events that cause a radical shift in a players emotional state (either direction).

Sure, there are those who think they know a better way to generate random numbers, or how to fix a broken combat or auction mechanic - but the true sign of success is how the game engages them personally.

share|improve this answer
add comment

The single most important thing you can learn from outside blind playtesters: Is it intelligible?

There are two very good "blind test" modes:

  1. hand the playtest GM the rules, and have him run a group of players and report
  2. hand the playtest GM the rules, and, never once answering a rules question, play in his game with his players.

Mode 1 is less accurate at conveying information to the designer, but more playtests can be done, and what is reported is more important.

Mode 2 is more accurate (you KNOW what is being done in ways you didn't intend), but much harder to do (tendency is to look to the designer for answers in play, chopping the GM's authority), and much more time intensive for the designer.

Note that running your own game isn't really playtesting. You're not testing the rules, but the ideas you've tried to put into the rules. Best test for my own RPG designs was putting them in other's hands and letting them go try on their own. Brad running RP2 for Mark and me convinced me it wasn't ready for sharing... not even over WWIVnet. While I've used it many times for Trek games, it's just never made it to sharable, let alone salable.

Things outside playtesters are good at finding and reporting:

  • omissions by the author
    • typically, things the author and his player group simply "know" get overlooked
    • dropped paragraphs and/or sentences sometimes render others less intelligible
  • areas that don't do what the designer claims they do
    • in the setting especially, if the mechanics do X, and the players can't see setting issue Y because of mechanic X.
    • In rules
      • misstated modifiers (usually wrong direction)
      • dice mechanics broken
      • point costs (if applicable) over- or under-value some mechanical element.
        • note that this may be all over the map; if, with a decent number of playtesters, shows a consistent trend, it's probably true
        • if the responses are all over the map on it, it's probably close enough.
      • combat or skill systems have
        • counter-intuitive results - but real combat sometimes does
        • counter-realistic results - noting that some realistic results are counter intuitive
        • playability issues, like time to calculate, or unfun mechanics.
  • areas where players need more info
    • often, this is in the mechanics
      • How do I use ?
      • Why can/can't I do ?
    • In the setting,
      • mostly about what they liked - they will want more
      • often, questions about "Why is this area doing this?" - not all need be published, but all such answers should be noted.
  • areas for supplementation
    • Rules: often times, players will make not of entire chunks of rules that they want to see, but are not present.
      • Sometimes, this is a playstyle issue, revealing differences between designer and playtester
      • Sometimes, this is a rare-but-useful block, like Mass Combat or Enchanting, where it can wait for a supplement to keep the core down
    • Setting: if only some playtesters want more details on a given area or element, it's probably better left to supplements; if the majority want it, and it's not too long for the book, put it into the core.
  • writing style
    • Clarity and intelligibility: can they understand it?
    • accessibility & organization. Can they find what they need when they needed?
    • arguability: if the players don't agree on interpretation, even tho each sees it as terribly clear, it's an issue that tends to get reported.
  • Art
    • is art needed to illustrate some point?
    • does that needed illustration actually show what it intends to?
    • is it still current to rules?

Some advice on playtesters: always try to get a mix; some of the most litigious and some of the most story oriented, in combination, quickly show via reports where it works or doesn't. If you get lots of picayune details caught by playtester rules-lawyers, in play, the rules lawyers won't be snarking at the GM over them...

Also, it really helps to have some outside readers who don't playtest... because playtesters are notoriously bad at catching spelling and grammar issues that don't in themselves muddle rules, but consumers often will bitch and moan.

share|improve this answer
    
The advantages of external playtesting are immeasurable ... provided that you actually use that feedback (unlike, say, Steve Jackson back when he first churned out GURPS). –  JUST MY correct OPINION Nov 2 '10 at 14:17
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.