Three versions of a product page are ready. The plan is to compare A, B and C simultaneously. For a small shop, the first question is whether each arm can collect the required number of analysable units within a useful period.
Adding an arm divides the available traffic into more groups. It does not create more visitors. The decision you need to make, the comparisons you plan and the sensitivity of the analysis determine whether the experiment is practical before it starts.
Count traffic reaching the actual experiment
Do not use all shop visits as your starting point. Count units that can actually enter this experiment under its inclusion rules, such as browsers receiving their first assignment. With stable browser assignment, returning browsers are not new independent units every week.
Fictional capacity calculation: 600 new, eligible and analysable units enter each week. An equal split gives roughly 200 per arm. If prior specialist planning required 8,000 units per arm, enrolment would take about 40 weeks. With two equally sized arms and the same traffic, it would take around 27 weeks.
The 8,000 is an assumed input for this arithmetic, rather than a calculated sample-size recommendation. Actual duration will also change if traffic varies. If the offer will change substantially in twelve weeks, the capacity calculation already shows a conflict with the experiment plan.
Sample size depends on what you need to detect
For a binary outcome such as “at least one confirmed purchase within the window”, planning needs a baseline rate, the smallest relevant change, a significance level and a power target, among other inputs. Minitab documents these relationships for two proportions. Three arms also need an appropriate comparison plan.
Write relative changes out explicitly: moving from 2% to 2.2% is a 10% relative change and a 0.2 percentage-point absolute change. Confusing them means planning a different experiment. The smallest relevant change is a planning input; it does not promise that the new page will produce that effect.
For A/B/C, decide whether B versus A and C versus A are sufficient, or whether B must also be compared with C. Two improvements over A do not establish a difference between B and C. NIST explains why repeating ordinary individual comparisons changes the combined error control. Plan the method, comparisons and sample size together.
Enrolment and analysis end at different times
The date you stop enrolling units is not necessarily the analysis date. If a confirmed purchase within seven days of entry is the outcome, the last unit needs those seven days too. This is a fictional window; choose one that fits the actual buying or enquiry process.
Also define a calendar period representing normal use. One unusually strong promotional day cannot answer a question about the ordinary range. A blanket “finished after two weeks” rule ignores both sample size and delayed outcomes.
For an analysis with a fixed endpoint, wait until that endpoint before making the statistical success decision. A sequential method suitable for repeated interim decisions needs to be planned accordingly. Optimizely’s current documentation distinguishes these approaches. You can still switch off a broken checkout immediately; record a safety stop separately from a statistical winner.
When the experiment cannot fit your timetable
Remove an arm for a substantive reason before launch. Alternatively, observe a specific purchasing obstacle or investigate a prototype qualitatively first. Those approaches answer different questions and should not later be presented as evidence of increased purchases. Extending an experiment until an attractive number appears does not solve the planning problem.
The Shop diagnosis helps turn a scoped buying journey and existing data into a next investigation. It does not promise delivery of any A/B/C experiment. For an earlier choice of method, read which decision UX research should inform.