Your test is meant to split visitors evenly between A and B. Some time later, the report contains 700 units in A and 300 in B. B has a higher order rate. Before adopting the new page, find out why the groups are so uneven.
This fictional example illustrates a possible sample ratio mismatch, or SRM: the observed group sizes are unexpectedly far from the planned allocation. Microsoft Research describes SRM as a warning sign for several kinds of data problem. It does not identify the cause or establish which variation performs better.
Decide what receives an assignment
Before launch, record the unit being randomised. A browser, a signed-in account and a session are different units. In a browser-based experiment, someone using two devices may appear twice. Calling the recorded browsers “unique people” would conceal that limitation.
For a product-page experiment, each browser could keep the same variation throughout the experiment. Reloading the page should then show the same variation without creating another experimental unit. Ask the developers to check whether the implementation actually follows that rule. Stable assignment is a technical property, rather than evidence of complete measurement.
Optimizely documents how an experiment maps an identifier to a range and therefore to a variation. The practical implication for your check is that the identifier, experiment version and allocation rules need to agree. Changing the identifier or modifying the experiment while it runs can change the original assignment.
Put three counts beside each other
Ask the developers for a short breakdown for each variation:
| Count | Implementation question |
|---|---|
| Assigned units | When is a valid unit first randomised? |
| Recorded entry into the experiment | At what point does the unit enter the analysis? |
| Confirmed outcome events | Which business action counts within the defined window? |
These counts serve different purposes. A unit can be assigned to a variation that subsequently fails to render. If only B loses those cases from the analysis, the remaining groups have been selected differently. Define the inclusion rule in advance; do not remove inconvenient cases afterwards to restore a 50/50 split.
Conversely, approximately equal counts in a small sample do not establish that assignment works correctly. Random allocation produces variation. An appropriate SRM check therefore considers both the sample size and the planned proportions, rather than applying a universal tolerance.
A/B/C needs its own expected proportions
Three equally sized arms should each receive about one third of the units. A deliberately different design might use 50/25/25. Do not check every experiment against 50/50. Also record what share of all eligible units is enrolled at all.
If the breakdown looks unusual, locate a specific change: did B launch later? Does switching variations create a second record? Is entry missing for one page type? A breakdown by date or browser may narrow the investigation. It does not automatically make a damaged experiment usable again.
Record the fault before planning a restart
Keep the expected allocation, observed counts and known changes in the experiment log. An unresolved SRM leaves the result undecided. If a restart is necessary, give it a traceable new version; do not silently pool the old and new observations.
The existing guide to differences between analytics and shop reports explains how to distinguish business counts. The Analytics data review can examine existing key events and measurement gaps. Repairing experiment assignment requires a separately agreed scope.