The single most important thing you can learn from outside blind playtesters: Is it intelligible?
There are two very good "blind test" modes:
- hand the playtest GM the rules, and have him run a group of players and report
- hand the playtest GM the rules, and, never once answering a rules question, play in his game with his players.
Mode 1 is less accurate at conveying information to the designer, but more playtests can be done, and what is reported is more important.
Mode 2 is more accurate (you KNOW what is being done in ways you didn't intend), but much harder to do (tendency is to look to the designer for answers in play, chopping the GM's authority), and much more time intensive for the designer.
Note that running your own game isn't really playtesting. You're not testing the rules, but the ideas you've tried to put into the rules. Best test for my own RPG designs was putting them in other's hands and letting them go try on their own. Brad running RP2 for Mark and me convinced me it wasn't ready for sharing... not even over WWIVnet. While I've used it many times for Trek games, it's just never made it to sharable, let alone salable.
Things outside playtesters are good at finding and reporting:
- omissions by the author
- typically, things the author and his player group simply "know" get overlooked
- dropped paragraphs and/or sentences sometimes render others less intelligible
- areas that don't do what the designer claims they do
- in the setting especially, if the mechanics do X, and the players can't see setting issue Y because of mechanic X.
- In rules
- misstated modifiers (usually wrong direction)
- dice mechanics broken
- point costs (if applicable) over- or under-value some mechanical element.
- note that this may be all over the map; if, with a decent number of playtesters, shows a consistent trend, it's probably true
- if the responses are all over the map on it, it's probably close enough.
- combat or skill systems have
- counter-intuitive results - but real combat sometimes does
- counter-realistic results - noting that some realistic results are counter intuitive
- playability issues, like time to calculate, or unfun mechanics.
- areas where players need more info
- often, this is in the mechanics
- How do I use ?
- Why can/can't I do ?
- In the setting,
- mostly about what they liked - they will want more
- often, questions about "Why is this area doing this?" - not all need be published, but all such answers should be noted.
- areas for supplementation
- Rules: often times, players will make not of entire chunks of rules that they want to see, but are not present.
- Sometimes, this is a playstyle issue, revealing differences between designer and playtester
- Sometimes, this is a rare-but-useful block, like Mass Combat or Enchanting, where it can wait for a supplement to keep the core down
- Setting: if only some playtesters want more details on a given area or element, it's probably better left to supplements; if the majority want it, and it's not too long for the book, put it into the core.
- writing style
- Clarity and intelligibility: can they understand it?
- accessibility & organization. Can they find what they need when they needed?
- arguability: if the players don't agree on interpretation, even tho each sees it as terribly clear, it's an issue that tends to get reported.
- Art
- is art needed to illustrate some point?
- does that needed illustration actually show what it intends to?
- is it still current to rules?
Some advice on playtesters: always try to get a mix; some of the most litigious and some of the most story oriented, in combination, quickly show via reports where it works or doesn't. If you get lots of picayune details caught by playtester rules-lawyers, in play, the rules lawyers won't be snarking at the GM over them...
Also, it really helps to have some outside readers who don't playtest... because playtesters are notoriously bad at catching spelling and grammar issues that don't in themselves muddle rules, but consumers often will bitch and moan.
