How to analyse playtest findings

Start with the decision the findings must inform

Before analysing a playtest result, write down the product decision it is meant to inform. That keeps a session from becoming a list of interesting moments with no way to decide what matters. For a multiplayer game, the decision might concern onboarding, squad recovery, match flow or a server route; it should be specific enough that an observation could strengthen, weaken or redirect it.

A finding is not a vote and it is not a bug report. It is a traceable explanation of what happened under stated conditions, how certain the team can be, and what should be investigated or changed next. Keep the raw session record beside the analysis so a teammate can challenge the interpretation without losing the event it came from.

Keep three layers visible

Observation

What happened in the session record: an action, hesitation, route, quote, timing or repeated sequence.

Interpretation

Why the team believes it happened, connected to the observation and open to a competing explanation.

Decision

What the product team will change, preserve, investigate or deliberately leave alone.

“Players did not understand the objective” compresses all three layers into one sentence. A better record shows when players missed it, what they did instead, what they later said they believed, and why that pattern matters to the pending design choice.

Read confidence and consistency separately

Confidence describes how well the evidence supports an interpretation. Consistency describes how often the pattern appeared under the tested conditions. A rare event can carry high confidence if the causal sequence is clear. A common comment can carry low confidence if participants were all responding to a leading question.

Show disagreement. If one squad recovers immediately while three do not, that exception may reveal the missing cue or the teammate behaviour that changes the outcome. Averaging it away throws out the mechanism the team needs.

Preserve the tested conditions

Every finding belongs to a build, cohort, session format, region, route and moment in the product. “New OCE players in build 0.8 did not find the recovery path during their first losing round” is narrower than “players cannot recover”, and far more useful.

Conditions are not caveats to hide in an appendix. They are the boundary that tells the team where the finding can travel. A later build, experienced cohort or different latency profile may produce another answer without making the earlier observation false.

Absence of an issue is not proof of absence

A session that does not surface a problem has shown that the problem was not observed in that room. It has not proven that the problem cannot occur. Ask whether participants had a fair chance to encounter it, whether an expert teammate routed around it, and whether the method could detect it if it happened.

Positive evidence needs the same discipline. Smooth completion can come from clear design, prior genre knowledge, social coaching or a moderator intervention. The record should let the reader distinguish them.

Triangulate without turning votes into truth

Combine the live record, participant explanation, build or route evidence and relevant telemetry when they answer different parts of the same question. Agreement across sources increases confidence. Disagreement is not a nuisance: it shows where the current explanation is incomplete.

Participant preference matters when preference is the question. It does not replace observed usability, and the loudest person in a debrief does not become the representative of the room. Preserve individual sequences before summarising themes.

End with the next decision, not a list of issues

A decision-ready finding should answer five questions:

• What happened, and where is the source event?
• Who experienced it, under which build and room conditions?
• What interpretation best fits, and what alternative remains?
• How confident and consistent is the pattern?
• What decision does it inform, and what evidence would change that call?

Not every observation deserves a fix. Some support the current design, some expose a tradeoff, and some simply define the next test. Keeping those outcomes visible is how a study stays useful even when the evidence does not support the change someone expected.

A worked example: one observation, written three ways

The example below is illustrative. It is not a delivered study, and no Studio, build or participant in it is real. The same underlying event is written three times, because the gap between these versions is where most of a study’s value is lost.

Written as a verdict: “The revive mechanic is confusing.” This is unusable. It names no event, no participant, no condition and no alternative explanation, and it cannot be checked by anyone who was not in the room. It is also unfalsifiable, so a team that disagrees with it has nothing to argue with except the researcher.

Written as raw record: “Four participants pressed the interact key repeatedly next to a downed teammate.” This is checkable but inert. It reports keystrokes without saying what the players were trying to do, whether they succeeded, or what the team should now consider. A decision cannot be built on it without someone silently supplying the interpretation later.

Written to be decision-ready: in build 0.8, three of five first-time squads had at least one player stand over a downed teammate pressing interact for several seconds without starting a revive. In debrief those players said they believed the revive had begun. The two squads that recovered contained a returning player who called the hold duration aloud. The best-fitting interpretation is that the revive gives no visible progress feedback and the behaviour is learned socially rather than from the interface; a competing explanation, that the prompt was legible but the key binding was unclear, is weakened because the same players used interact correctly on doors in the same round.

Only the third version carries the observation, the interpretation, the competing explanation and the conditions together. It also tells the team what would change the call: if a squad with no returning player recovers reliably once progress feedback is added, the interpretation holds. If they still fail, the problem is not feedback and the next test needs to look at the binding or the timing instead.