Playtest LiveGUIDE
Reading playtest findings
A useful finding keeps the observed event, the researcher’s interpretation and the product decision connected without pretending they are the same thing.
Keep three layers visible
- Observation
- What happened in the session record: an action, hesitation, route, quote, timing or repeated sequence.
- Interpretation
- Why the team believes it happened, connected to the observation and open to a competing explanation.
- Decision
- What the product team will change, preserve, investigate or deliberately leave alone.
“Players did not understand the objective” compresses all three layers into one sentence. A better record shows when players missed it, what they did instead, what they later said they believed, and why that pattern matters to the pending design choice.
Read confidence and consistency separately
Confidence describes how well the evidence supports an interpretation. Consistency describes how often the pattern appeared under the tested conditions. A rare event can carry high confidence if the causal sequence is clear. A common comment can carry low confidence if participants were all responding to a leading question.
Show disagreement. If one squad recovers immediately while three do not, that exception may reveal the missing cue or the teammate behaviour that changes the outcome. Averaging it away throws out the mechanism the team needs.
Preserve the tested conditions
Every finding belongs to a build, cohort, session format, region, route and moment in the product. “New OCE players in build 0.8 did not find the recovery path during their first losing round” is narrower than “players cannot recover”, and far more useful.
Conditions are not caveats to hide in an appendix. They are the boundary that tells the team where the finding can travel. A later build, experienced cohort or different latency profile may produce another answer without making the earlier observation false.
Absence of an issue is not proof of absence
A session that does not surface a problem has shown that the problem was not observed in that room. It has not proven that the problem cannot occur. Ask whether participants had a fair chance to encounter it, whether an expert teammate routed around it, and whether the method could detect it if it happened.
Positive evidence needs the same discipline. Smooth completion can come from clear design, prior genre knowledge, social coaching or a moderator intervention. The record should let the reader distinguish them.
Triangulate without turning votes into truth
Combine the live record, participant explanation, build or route evidence and relevant telemetry when they answer different parts of the same question. Agreement across sources increases confidence. Disagreement is not a nuisance: it shows where the current explanation is incomplete.
Participant preference matters when preference is the question. It does not replace observed usability, and the loudest person in a debrief does not become the representative of the room. Preserve individual sequences before summarising themes.
End with the next decision, not a list of issues
A decision-ready finding should answer five questions:
- What happened, and where is the source event?
- Who experienced it, under which build and room conditions?
- What interpretation best fits, and what alternative remains?
- How confident and consistent is the pattern?
- What decision does it inform, and what evidence would change that call?
Not every observation deserves a fix. Some support the current design, some expose a tradeoff, and some simply define the next test. Keeping those outcomes visible is how a study stays useful even when the evidence does not support the change someone expected.