A/B Testing: Frequentist, Bayesian and Sequential in Practice

Published · 25 min read

An overview of how each approach works, where each one breaks, and what separates a test you can trust from a test that looks great until it stops looking great.

Let’s learn everything A/B has to offer in the shortest time possible

Let’s learn everything A/B has to offer in the shortest time possible

In 2012, a Bing engineer had an idea. Move the ad’s secondary text up into the headline line and make the headline longer. The idea sat in the product backlog for six months, deprioritized, until someone implemented it in three days and ran an A/B test just to satisfy their curiosity. Within hours the system fired a “too good to be true” alert because revenue was 12% above control. It was not a bug. The change became the best revenue-generating idea in Bing’s history, worth more than US$ 100 million a year in the US alone, as reported by Kohavi and Thomke in the Harvard Business Review.

That is the good side. The ugly side is that at Microsoft itself, across more than a decade of running tests at scale, roughly one third of ideas improve the key metric, one third are neutral, and one third make it worse. At Bing specifically, the success rate is even lower. Booking runs 25,000 A/B tests a year. If their statistics were poorly calibrated, they would be making thousands of wrong decisions with high confidence.

The choice between Frequentist, Bayesian and Sequential is not an academic debate. It is the difference between trusting the one third that actually works and fooling yourself with false positives for the rest of the year.

When you realize you spent all your time on something that doesn’t work

When you realize you spent all your time on something that doesn’t work

Frequentist, the default and its gotchas

Frequentist is what practically every online A/B calculator uses under the hood. The logic comes from the Neyman-Pearson framework of the 1930s. You define a null hypothesis H₀ (the variants are equal), an alternative H₁ (they are different), compute the test statistic, and from that the p-value, the probability of observing the result you observed (or a more extreme one) if H₀ were true.

If p < 0.05, you reject H₀. If p ≥ 0.05, you fail to reject. That is not “accept H₀”, it is “I could not knock it down”. Subtle, but important.

You compute the sample size beforehand, as a function of four things, the metric’s variance, the minimum lift you want to detect (MDE), the significance level (α, typically 0.05) and the power (1-β, typically 0.8). Run until the planned N, look at the result, decide.

It works. It has decades of use, solid theoretical grounding, and any paper or product reviewer accepts it. The problem is that almost nobody follows the protocol in practice.

Gotcha 1, peeking

The rule is you do not look before the planned N. Looking early corrupts the p-value in a specific way, the more often you look, the higher the cumulative false positive rate gets. Optimizely simulated this explicitly. In an A/A test (the same variant against itself, so the correct result is zero effect) with 5,000 visitors:

Read that again. More than half of A/A tests with random peeking would declare a winner that does not exist. In a 1 million visitor test with peeking, that number goes above 70%.

Why does this happen? Because the p-value is not a stable value converging to “the” true value. It oscillates. In an A/A, if you plot the p-value over time, it crosses 0.05 several times even when the real effect is zero. You only need to wait for it to cross again, and stop at that point, to “find” significance.

Trap 2, multiple testing

You run a test with 1 variant and 1 metric, error rate 5%. Run it with 4 variants and 5 metrics, and that is 20 independent comparisons. The probability of at least one false positive rises to about 64% (1–0,9⁵²⁰).

The simplest correction is Bonferroni, divide α by the number of tests. Too conservative when metrics are correlated. Modern alternatives use Benjamini-Hochberg to control FDR (False Discovery Rate) instead of FWER (Family-Wise Error Rate). Optimizely controls FDR directly in its Stats Engine.

Trap 3, low power

Power is the probability that your test detects a real effect. Run a test with 1,000 visitors to detect a 0.5% lift and your power is something like 15%. Even if the variant is better, in 85% of cases your test cannot see it. That is not “it was a tie”, it is that the test is blind to the effect you want to measure.

And here lies the compound problem, low-powered tests do not just fail to detect true effects, they also inflate effects when they do come out significant. The phenomenon is known as the winner’s curse. If an underpowered test gives p < 0.05, the estimated effect size tends to be much larger than the real one, because only the big noise manages to cross the significance barrier. The result is you ship, measure in production later, and the effect is gone.

Trap 4, p-hacking

A refined variation of the previous three. You run the test, no significance in the aggregate. Slice by mobile, nothing. Slice by new vs returning, there it is! Publish.

That “there it is” is purely an artifact of how many slices you tried. If you make 20 slices in an A/A, the probability that at least one gives p < 0.05 by pure chance is high. Kohavi, Tang and Xu, in the book Trustworthy Online Controlled Experiments, call this a manifestation of Twyman’s Law: any figure that looks interesting or different is usually wrong. An apparently positive result in a specific segment deserves double checking.

When you believe every result just because it has numbers in it

When you believe every result just because it has numbers in it

Frequentist works, with methodology

You computed the sample size beforehand. You locked the metrics beforehand. You did not look midway. You did not run multiple slices hunting for a story. Then it is robust, transparent and cheap.

The issue is that product environments have pressure to “see how it’s going”, multiple stakeholders staring at the same dashboard, and the obvious human temptation to declare victory early. The methodology is solid, the environment is hostile to it.

Bayesian, the alternative that looks intuitive (and what it hides)

Bayesian flips the question. Instead of “what is the probability of seeing this if H₀ is true?”, it answers directly, “what is the probability that B is better than A?”.

The mechanics use Bayes’ theorem. You start with a prior (an initial belief about the metric’s distribution), observe the data, and compute the posterior, the updated distribution. For conversion tests, the standard prior is a Beta distribution, which is the natural conjugate of the Binomial distribution of the data, and that means the posterior is also Beta and has a closed-form formula. No MCMC, no expensive sampling, it is straight arithmetic.

Bayesian outputs are qualitatively different from Frequentist ones:

That sounds far more aligned with business decisions than a raw p-value. That is why VWO built SmartStats on top of Bayes-Beta-Binomial and Google Optimize (discontinued in 2023) also used Bayesian.

Where Bayesian does well

Three points where it genuinely beats classic Frequentist:

  1. The question it answers is the question you want to ask. “What is the probability that B is better than A?” is what the stakeholder wants to hear. The p-value does not deliver that. It lets you incorporate prior knowledge formally. If you have already run 200 landing page tests and know typical lifts sit between -5% and +5% with a mean near zero, you can put that information in the prior instead of assuming uniformity. The “expected loss + threshold of caring” interface forces you to think about how much loss risk you are willing to accept. To be declared the winner in VWO’s SmartStats, a variant needs expected loss below a threshold AND a chance to beat all above 95%. That is more robust than just “p < 0.05”.

But not everything that glows in the dark is gold.

Hidden prior

Commercial vendors frequently hard-code priors or utility functions without documenting them, which makes it impossible for the user to assess or adjust the assumptions. Georgi Georgiev, of Analytics Toolkit, documented multiple cases of platforms presenting Bayesian outputs without exposing the priors used. If the prior is Beta(1,1) (uniform), results start at 50%/50% and take a while to move. If it is Beta(50, 950) (informative, based on a historical 5% conversion), results start concentrated near 5% and require much more data to shift.

The practical consequence, two Bayesians with different priors, looking at the same data, can give different numbers. And if you do not even know which prior is running, you have no way to judge whether it is too conservative or too aggressive.

“Probability to be best” does not mean what it seems

Read this slowly. On a standard Bayesian platform with a non-informative prior, the probability that B is the best starts at 50% (with 2 variants) and in most A/As stays around 50% even with hundreds of thousands of visitors. That makes sense. If A = B, the metric oscillates symmetrically around 50%.

Georgiev showed in 60 real A/As run on Google Optimize that the “probability to be best” was distributed around 50% as expected, but in several individual tests it crossed 95% by pure chance and stayed there long enough for an untrained operator to declare a winner.

The number “B has an 87% probability of being the best” is frequently read as “I am 87% sure I will gain revenue by shipping B”. It is not. It is the probability that B has a higher conversion than A, conditional on the prior used, with no guarantee of how much higher or for how long. Those two things are not the same.

Stopping rules still matter

This is the most common mistake in the market. Many people migrate to Bayesian thinking they can look at the test whenever they like and stop whenever they please because “Bayesian doesn’t suffer from peeking”. Not exactly.

Under strict assumptions and specific priors, some Bayes factors are invariant to the stopping rule. But in practice, commercial A/B testing uses default priors and composite hypotheses, and in that scenario the theoretical robustness falls apart. The posterior can end up poorly calibrated and lead to biased decisions.

The intuitive explanation is that the stopping rule is part of the data generating process. If you decide to stop when “probability to be best > 95%”, you are conditioning the experiment to stop on the trajectories where that threshold is crossed, which biases the final estimate upward. Serious Bayesians stress this. Bayesian without a disciplined stopping rule has the same problem as Frequentist with peeking. The interface just hides it better.

Sequential, monitoring continuously without burning α

Sequential is what people usually want from the start, to look at the test continuously without corrupting the inference. The idea is old. It comes from Wald’s Sequential Probability Ratio Test (SPRT) in the 1940s, originally developed for quality control in World War II. The modern application to A/B is more recent.

There are two living lineages, always-valid (mSPRT) and Group Sequential with alpha spending. They solve the same problem by different routes.

Always-Valid and mSPRT

Optimizely uses the mixture sequential probability ratio test (mSPRT), described in the paper by Johari, Pekelis, Walsh and Koomen as the foundation of its Stats Engine. The construction produces “always valid” p-values in the formal sense that P(p_T ≤ x) ≤ x for any stopping rule T, meaning the user can look whenever they want, stop whenever they want, and the inference remains correct in the frequentist sense.

The intuition behind it is that mSPRT mixes over a family of alternative hypotheses (the “m” in mixture) using a prior τ. That solves the problem of classic SPRT, which required you to fix an effect size up front to compute the boundaries, which nobody does in practice in product.

In simulations where classic Frequentist adjusted for daily peeking declared a false winner in 57% of tests, sequential drops to 3%, as reported by Optimizely.

It has a cost. Sequential needs more sample to reach equivalent power when the effect is small. You gain stopping flexibility in exchange for statistical efficiency in the worst case. The magnitude depends on the prior τ used and the true effect size.

Group Sequential and Alpha Spending

An older alternative, coming from clinical trials. The idea is that instead of looking continuously, you define N scheduled peeks (for example every 25% of the planned sample) and use an alpha spending function to split the 5% error among them.

The seminal paper is Lan and DeMets (1983), which generalized earlier work by Pocock (1977) and O’Brien and Fleming (1979). The two classic functions:

There are intermediate variants (Wang-Tsiatis, linear alpha spending) and the flexibility to use custom functions, in which the look timings do not need to be fixed beforehand, only the spending function does.

Take a little break before the conclusions

Take a little break before the conclusions

Who uses what

Booking, Microsoft and Netflix use variations of sequential testing combined with alpha spending or group sequential in their internal platforms. Optimizely offers always-valid as the default mode. ABsmartly (founded by engineers who built Booking’s platform) offers Group Sequential as the default and fixed-horizon Frequentist as an alternative for teams that want high reliability or have strong weekly seasonality.

In practice, sequential has become the de facto standard in mature experimentation programs. Online A/B calculators are still classic Frequentist or simple Bayesian in most cases.

Summary of the three approaches

Frequentist answers “how unlikely is this result if the variants were equal”, requires a pre-computed sample size, is sensitive to peeking, and works if the operator follows the protocol. Low learning curve, potentially high gotchas.

Bayesian answers “what is the probability that B is better”, accepts prior knowledge formally, expresses uncertainty as distributions, and has outputs aligned with business decisions. But it depends on priors that are often opaque, and in practice it is still sensitive to poorly designed stopping rules.

Sequential answers “is this difference robust to continued looking”, allows flexible stopping by design, is what companies running tens of thousands of tests a year use, and charges a sample overhead in exchange for the flexibility.

There is no right paradigm. There is the paradigm that fits the context. A small team with an online calculator is probably running poorly disciplined Frequentist and would gain by migrating to a Bayesian platform with expected loss + threshold. A platform with a data science team embedded in the product and meaningful volume almost certainly wants sequential.

Variance reduction, a shortcut for everyone

Regardless of paradigm, there is a technique that reduces the required sample size without changing the methodology, variance reduction. The most popular is CUPED (Controlled-experiment Using Pre-Experiment Data), introduced by Deng, Xu, Kohavi and Walker in a 2013 Microsoft paper.

The intuition is simple. Most of the variability in a user’s metric during the experiment comes from that user’s own variability before the experiment. Heavy users were heavy before. Churning users were churning before. If you use pre-experiment behavior as a covariate and adjust the experiment metric by what was expected given the history, the residual variance drops dramatically.

In the original paper, in a Bing experiment, using queries-per-user from the pre-experiment period as a covariate reduced variance by more than 45%. Combined with other covariates, it got close to 50%. 50% less variance means a narrower confidence interval for the same N, which means half the time to detect the same effect.

With CUPED, Microsoft managed to turn tests that took 8 weeks into tests of 5 to 6 weeks. For a company running tens of thousands of tests a year, that is doubling iteration speed.

CUPED became standard. Statsig, Eppo, Amplitude, LaunchDarkly and Kameleoon implement variants. Netflix published a paper in 2016 (Xie and Aurisset) with adaptations for their context. Meta has machine learning variants (Guo et al., 2021). The modern version of CUPED, in a 2023 paper by Deng himself, generalizes to ratio and percentile metrics, and to using in-experiment data, not just pre-experiment.

If you are on a modern commercial platform, CUPED is probably already running under the hood. If you are running A/B tests in homegrown SQL, it is worth reading the original paper and applying it.

Sampling validation

Before analyzing results, validate the setup. This step is usually forgotten in immature environments and automated in mature ones. Without it, any of the three methodologies above delivers wrong numbers with high confidence.

A/A, the test of the test

A/A is exactly an A/B, but with the same variant on both sides. The real effect is zero. If the result shows a “winner”, the problem is not the variant, it is the infrastructure.

Optimizely used simulated A/A to calibrate its Stats Engine. It is the reference experiment for demonstrating the peeking problem. Microsoft, Booking and Netflix run A/A periodically in production, against real traffic. The purpose is to catch subtle platform bugs that only show up at volume, biased bucketing hash, exposure logging that fires at different moments between variants, bot exclusion applied in different windows, asymmetric cache invalidation.

If a production A/A declares a winner with p < 0.05 in more than ~5% of runs, something is broken. The expected false positive rate needs to match α exactly. If it does not, there is systematic bias.

In the end it’s all about how you approach things

In the end it’s all about how you approach things

Sample Ratio Mismatch (SRM)

You designed the test 50/50, and in the logs control ended up with 50.2% and treatment with 49.8%, which looks irrelevant at first glance, but at high volume a chi-squared test can flag something like a 1 in 500,000 chance of that difference happening by pure chance. Small in percentage, large in evidence.

SRM with a p-value that low is strong evidence of a bug or bias. The common causes:

Microsoft’s SRM case, documented by Kohavi, illustrates the subtlety. In one experiment, engagement increased so much in the treatment variant that heavy users ended up above the bot classification threshold and were removed from the analysis, distorting the split. When the bot threshold was adjusted, the SRM disappeared, and the treatment turned out to be even better than the initial reading. SRM is not just a technical error, it is also a symptom that something real is happening and needs to be understood.

Microsoft monitors SRM automatically on every experiment. If it fires, the result is considered invalid until the root cause is identified. It does not matter how good the main lift looks. Booking runs automated SRM checks as part of the platform, blocking variant promotion in case of mismatch.

Pre-experiment balance

Before the treatment is applied, the two groups should be statistically indistinguishable on any pre-experiment metric. If they are not, the randomization is biased or the randomization unit has temporal bias.

The check is simple, run the same statistical test you would use post-experiment (Frequentist, Bayesian, whichever) on metrics from the period before exposure. If a statistically significant difference shows up in pre-experiment metrics, any post-experiment difference is confounded by the pre-existing difference.

CUPED, mentioned in the previous section, partially mitigates this problem by adjusting for history, but the balance check is worth doing as a sanity check independent of the analysis method. If balance fails systematically, the problem is in the randomization, not in the analysis.

Hash quality and bucketing

The function that decides which user lands in which variant needs to be uniform. Naive implementations like user_id % 2 appear to work but have problems. Sequential IDs correlate with signup time, so the two groups can have different distributions of new vs old users. In systems with machine-generated IDs (Snowflake, for example), the modulo can pick up the machine_id bit and split by datacenter instead of randomly.

Robust implementations use a cryptographic hash of the user_id concatenated with an experiment salt, something like SHA-256 of experiment_id:user_id. That guarantees three things, deterministic assignment (the same user always lands in the same variant of that experiment), independence between experiments (assignment in one experiment does not correlate with another, so a user can be in 15 experiments simultaneously without cross bias), and uniformity (each bucket receives the expected fraction).

Periodic hash validation includes running the function over millions of IDs and checking uniformity globally and also across slices, geo, account age, plan, hour of day. Non-uniformity in some slice signals that specific cohorts are under or over-represented in some variant.

Trigger analysis and Intent-to-Treat

When the treatment only affects a subset of users (for example a feature visible only on a specific page, or only on mobile, or only for users with a balance above X), the analysis has two paths:

The practical rule is, if the trigger is symmetric (same definition in control and treatment, counted the same way), the analysis restricted to the triggered is valid. If it is asymmetric (treatment triggers by a different criterion than control, or only logs the trigger in treatment), the analysis must be ITT.

CUPED has specific extensions for one-sided triggering, where the trigger is only observed in one of the variants. It is a common scenario in feature flags where the exposure event only fires when the feature is rendered, meaning only in the treatment variant.

Trouble in all three

Sampling validation takes care of the setup. But even with a perfect setup, there are phenomena that destroy the inference of any paradigm. These are the problems that survive every automated check and still break the reading of the test.

Now that we’re here, let’s go all the way to the end

Now that we’re here, let’s go all the way to the end

Outliers and dirty data

In an Amazon experiment documented by Kohavi, with 100,000 users, a 2% conversion and an average order of US$ 30 gave US$ 60,000 in total revenue. A 2% lift would be US$ 1,200. Occasionally a “user” bought ~US$ 2,500 in a single transaction. That single user, dropped into control or treatment, could inflate the result by more than the effect expected from the real lift.

The discovery was that libraries buy books in irregular batches, generating those giant purchases. The solution, cap any individual user’s value at the 99th percentile of the distribution (winsorization). Robust metrics, percentiles instead of means, help.

Bots

More than 50% of Bing’s traffic is classified as non-human, according to Kohavi. In e-commerce and media the number is also high. Without bot filtering, you are testing whether the red or blue variant pleases Google’s crawler and scraping robots more, which is probably not your KPI.

Bot filtering has to be consistent between control and treatment, otherwise you generate SRM artificially. And it needs to be re-evaluated when the test changes user behavior, as in the Microsoft case above.

Novelty effect and long-term effect

The “new” variant causes unfamiliarity or curiosity in the first weeks, and the effect decays. Or the opposite, a dark pattern that gives short-term lift and burns retention in 3 months.

Kohavi recommends two approaches. Long-term experiments run with the explicit goal of learning. At Bing they do this by increasing and decreasing the number of ads to understand the impact on key long-term metrics. And predictive models with historical data to estimate future impact from short-term signals. In Amazon’s case, the email team discovered that sending more email increased short-term revenue but raised unsubscribes, losing future revenue. They built a model to predict that loss and use it as a counter-metric.

Ad clickbait is the archetypal example. CTR goes up in the short term, retention and satisfaction fall in the long term. Without a long-term metric, the test declares victory and the company loses money in silence.

Heterogeneous Treatment Effects (HTE)

The test’s average effect is positive. But maybe it is very positive on mobile and slightly negative on desktop, and the aggregate masks that. HTE is the analysis of how the effect varies across subpopulations.

The Netflix Tech Blog described how their platform supports automated HTE analysis as part of standard reporting. Microsoft, Google and LinkedIn do the same. Without HTE, a global launch decision based on the average can hurt important segments.

The opposite risk also exists, p-hacking via segmentation. If you slice by 30 segments looking for “where it worked”, the chance of finding significance by accident is high. That is why good platforms fix the segments of interest before the test runs.

OEC, the metric that decides

The last piece of the chain, the Overall Evaluation Criterion (OEC). It is the single metric (or function of metrics) that will decide the test, defined before the test starts.

Without a declared OEC, any team finds some metric to justify the decision it already wanted. Conversion went up 1% but time on page dropped 8%? If the OEC was conversion, ship it. If the OEC was composite engagement, rethink. If there was no OEC, it becomes politics.

Bing’s typical scorecard has more than 2,000 metrics, according to Kohavi. Most of them exist to detect unexpected consequences, not to decide the test. Some of these metrics are guardrails. Any significant drop in them blocks the launch regardless of what happened to the OEC. Latency, error rate, refund rate, accessibility, fraud, customer service contact rate.

Booking maintains explicit guardrails like cancellation rate, payment failures, refund friction, and load times. A 3% conversion lift that comes with a 1% increase in cancellations can be zero net gain after the customer cancels and logistics absorbs the cost.

Conclusion

Frequentist, Bayesian and Sequential solve the same problem with different priorities. Frequentist is robust if the protocol is followed, and most practice violates the protocol. Bayesian gives outputs that are more readable for the business but hides modeling decisions (priors) that change the result, and it is still vulnerable to poorly designed stopping rules. Sequential solves the operational problem of peeking and for that reason became the standard in mature programs, at the cost of statistical overhead in the worst case.

The common denominator of the three is Twyman’s Law. Any result that looks too good is probably wrong. It is worth checking SRM, worth running A/A periodically, worth double-checking the analysis of any unexpectedly high lift. Bing found US$ 100M in a trivial change, but most of Bing’s growth came from increments of 0.1% to 0.2% per iteration, adding up to ~2% a year. Airbnb got ~6% revenue uplift by accumulating 250 small experiments in search relevance.

A mature experimentation culture is not a culture of genius tests. It is a culture of many tests, with rigor, with automated guardrails, with a platform that blocks the wrong decision before it happens, and with the humility to accept that two thirds of ideas will not work even if they look great on the whiteboard.

The choice of statistical paradigm is one layer of that. On its own it solves nothing. Combined with monitored SRM, an OEC declared up front, locked guardrails, variance reduction applied where possible, and stopping rule discipline, it becomes the operating system of anyone who grows through A/B.

References

#ab-testing#statistics#experimentation