The overall report has no clear winner. You filter by device, then browser, then traffic source. B looks promising among mobile visitors. That observation gives you a new question. It is not yet a sound rule for rolling B out only on mobile.
Optimizely documents filters by device, browser and traffic source. When those slices are chosen after inspecting the results, we treat the search as exploratory. Keep the interesting finding together with the search that produced it. Do not present the final slice as the original experiment objective.
Smaller groups, more chances to find a pattern
Each filter often reduces the number of observed units. Meanwhile, you have more opportunities to find an apparently striking difference by chance. “Mobile” may sit alongside three browser groups, several traffic sources and different outcome events. One attractive percentage does not reveal that search space.
Record which analyses you actually inspected. Handling multiple comparisons belongs in the statistical plan. The Office of Evaluation Sciences describes how it controls false alarms across planned confirmatory hypotheses and distinguishes these from exploratory questions. An adjustment for the two originally planned comparisons does not automatically cover every later search.
Two effects do not establish a difference between groups
Fictional example using confirmed outcomes: On mobile, A has 25 successes among 500 units and B has 35 among 500: 5% and 7%. On desktop, A has 30 out of 500 and B has 31 out of 500: 6% and 6.2%.
The observed difference is 2 percentage points on mobile and 0.2 percentage points on desktop. The difference between those effects is 1.8 percentage points. So far, that is just arithmetic using the example data.
Claiming that the change works differently on mobile needs an appropriate analysis of the difference between effects, such as a preplanned interaction test. “Significant on mobile, not significant on desktop” would not establish that on its own. Show uncertainty and group sizes as well. These illustrative figures deliberately name no statistical winner.
Was the segment defined before the change?
Device type recorded at entry is different from “reached the cart”. Reaching the cart may itself be affected by the page being tested. Comparing only those people afterwards may select different kinds of units in A and B.
Microsoft Research recommends stable characteristics unaffected by the treatment for ordinary segment comparisons. A practical implication for a small shop is to define the characteristic and when it is recorded before launch. Still check its distribution and known assignment gaps. Switching devices during the experiment can complicate classification.
Behavioural groups formed after entry can remain useful descriptions. Label them accordingly. Without additional assumptions, they do not answer the causal question of how the same target group would have responded to A or B.
Turn the finding into a question you can investigate
Write a specific hypothesis: “The longer delivery explanation pushes the selector down on small screens; a shorter version might affect completion.” The proposed explanation should be observable in the interface. A contrary observation matters too.
Before a new experiment starts, define the mobile population, primary outcome, relevant effect size and analysis plan. Use new observations; data that suggested the idea cannot simultaneously provide independent confirmation. If too little traffic is available, qualitative research can investigate the suspected usability obstacle, but cannot establish an increase in purchases.
Keep the original overall experiment and its result in the report. Add the segment finding as an exploratory observation and state the unresolved follow-up question. The Analytics data review helps examine existing events and measurement gaps; full statistical experiment consulting requires a separate scope. The guide to duplicate analytics events covers an earlier technical check.