The useful method depends on what you still need to know. If it is unclear where people struggle with a task, a usability test can make that journey visible. If you want to compare the effects of defined variations on a measurable action instead, you need an appropriately planned A/B test.
When you do not know why people leave
Visitors reach the variant selector, but you do not know why they go back. A moderated usability test can reveal what information a person seeks, what they misunderstand and where they need help.
To observe the participant’s own selection, give them a realistic task, such as finding a replacement part for a described device. The facilitator does not name the correct filter or route to the compatibility list. Otherwise, you would mainly see whether instructions can be followed rather than how selection works independently.
The result is a set of traceable observations. You can use them to develop a design and investigate it again. A small qualitative study cannot calculate a percentage purchase increase for the whole shop. NN/g maps research methods to the questions and data they address.
When you need to compare variations
An A/B test randomly assigns eligible units to variations. Beforehand, define the outcome, who enters the experiment and when their observation is complete. A variation can then be compared with the current product page, for example.
For the comparison to answer the intended question, delivery and measurement must be reliable. If confirmed purchases are the outcome, a click on “Buy” is insufficient; returning browsers must not switch arbitrarily between variations either. Traffic requirements and the evaluation plan therefore belong before the launch.
The question is not whether participants like a new explanation. It concerns a defined difference in behaviour under the experiment’s conditions. Why a person made a particular choice often remains unanswered.
Repair technical faults first
If the variant selector demonstrably puts the wrong item in the cart, reproduce and repair the fault. That does not need an A/B test. If you only know that the exit rate is high, the explanation is still missing: observing the selection task can help you investigate it.
Understand the task, then test the remaining question
The methods can also follow one another: observing the replacement-part search may lead to clearer compatibility information, whose use another usability test examines. The remaining question may then be whether a shop metric changes. That needs a separately planned comparison because the earlier observations do not yet measure that effect.
Our guide to traffic requirements for test variations helps assess whether an experiment fits your timeframe. It treats sample size as a planning question, not a universal minimum for every shop.
For our usability test, we first agree on the task to observe. An impact analysis addresses the later evaluation of a change; both are part of our CRO services.
