You remove two fields from an enquiry form. Variation B receives more submissions. Sales also reports more requests outside the service offered. Both observations can be correct. The experiment needs a predefined outcome and a view of subsequent quality before you can make a decision.
An enquiry is initially an opportunity for contact. Assignment to an experiment does not make it a purchase or revenue. Google Analytics lists generate_lead and purchase as separate events. Your own implementation still needs to reflect the actual business state.
Define quality before launch
Agree with the responsible person what makes an enquiry suitable. For example: you offer the requested service, the project is in an area you serve, and the contact details allow a response. Use criteria relevant to the business that can be applied equally to both variations.
Where possible, the person assessing a request should not know its variation. Otherwise, expectations about the new form may influence classification. Keep the rules stable during the experiment. If the meaning of “qualified” changes later, record that change in the log.
Distinguish these measures:
| Measure | Question it answers |
|---|---|
| Received enquiries per assigned unit | How often does a stored contact arise? |
| Qualified enquiries per assigned unit | How often does a contact meet the same suitability rule? |
| Qualified share of enquiries | What is the composition of the contacts received? |
| Confirmed business outcomes per assigned unit | How often does the defined later outcome occur? |
The qualified share alone is insufficient. A variation could receive fewer enquiries and produce a higher qualified share without generating more suitable contacts. Microsoft Research describes the risk of interpreting local metrics without the wider context.
When the counts move in different directions
Fictional groups whose follow-up periods have all closed: Each arm has 1,000 assigned units. A produces 40 enquiries, including 20 qualified contacts and ten later completed deals. B produces 60 enquiries, including 18 qualified contacts and eight completed deals.
The enquiry rate rises from 4% to 6%. Qualified-enquiry rates are 2% and 1.8%; completed-deal rates are 1% and 0.8%. These are descriptive figures. On their own, they establish neither a statistically reliable difference nor a lasting decline.
If qualified contacts were the predefined primary outcome, the greater number of submissions cannot retrospectively make the experiment a success. If the outcomes were unclear beforehand, record that limitation and write a new comparison plan.
Give late outcomes equal time
A contact arriving on the first day has had longer to progress than one arriving on the last. Define the same follow-up period from entry for each unit, then wait for outstanding windows to close. Whether 30 days is enough depends on the real sales cycle; that number is not a general recommendation.
Attribute a later outcome to the original assigned unit and variation where a permitted, reliable link exists. Without that link, outcomes by variation remain unknown. Do not replace missing attribution with the average order value across all customers. For repeated enquiries from one unit, also decide in advance whether the metric counts at least one contact or the total number of contacts.
Money needs a confirmed business state
Binding orders, payments, cancellations and refunds are separate states. Specify which of them a report includes. The quoted price in a non-binding enquiry is not realised revenue. Revenue analysis in the Analytics & A/B tests tool currently envisages manual order imports confirmed by staff; this does not promise a finished automatic Shopify or WooCommerce order integration.
If an offer page attracts unsuitable requests, the Enquiry review can examine clarity and the enquiry journey. For a separate scenario calculation using an explicitly assumed closing rate, read what a website enquiry is worth. That calculation does not replace analysis of your experiment.