Before commissioning a new variation, establish which problem it is meant to solve and why you expect this particular change to help. A test hypothesis makes that reasoning explicit by connecting your observation, proposed explanation and planned change. The experiment plan then defines how you will assess the effect and which adverse outcomes would count against adoption.
Keep observation separate from explanation
In a hypothetical usability test, several participants only find a replacement part’s compatibility information after the facilitator asks about it. This leads to the proposed explanation that its current location causes people to overlook it before selecting.
If location is the suspected cause, the design could show the same substantive information beside the selector. Keeping the content constant lets you investigate placement more directly. Changing price, included parts and images at the same time would instead test a larger package whose effect cannot be attributed to that single explanation.
An initial usability test can establish whether the new presentation is understood. Statistically comparing an outcome is a further question.
Complete a short hypothesis card
Complete the card for the specific change you want to investigate. If the outcome is still undefined, for instance, part of the experiment plan is missing.
| Field | Planning content |
|---|---|
| Starting point | Observation, location and source |
| Explanation | Why might the task break down here? |
| Change | What exactly differs between B and A? |
| Affected group | Who reaches the change, under which conditions? |
| Outcome | Which confirmed action should change? |
| Guardrails | Which unwanted effects need monitoring? |
| Decision | Which evaluation procedure will be planned in advance? |
Microsoft’s experimentation team describes connecting hypotheses with success and guardrail metrics. Your card needs to match the product and available measurement.
Connect the metric to the business question
More clicks on compatibility information could mean it is more noticeable. They could also reflect greater uncertainty. If the business question concerns suitable purchases, clicks alone are not a sufficient outcome.
The measurement you need depends on the confirmed action and its observation window. For orders, it must reflect actual completion; for enquiries, whether they can be acted on may also matter. Because these outcomes can arrive later, the plan needs to explain how they will be linked to the appropriate group without collecting unnecessary personal data.
Guardrails might concern incorrect selections or technical faults. Define which deterioration is critical and who will act on it. If the evaluation question is still unresolved, an impact analysis can address it as part of our CRO work.
Leave room for an unresolved result
Your eventual decision needs to allow for more than “B wins”. The result may be inconclusive, or a fault may require the experiment to stop. In those cases the question can remain open: an inconclusive comparison does not establish equality, and choosing a more convenient metric afterwards does not confirm the original explanation.
Sample size, duration and evaluation method need to fit together. Use the guide to multiple test variations to check whether the comparison fits your traffic budget. If not, the hypothesis can remain an open question in the backlog. A finished design is not by itself a reason to launch it.
