Bayesian A/B Testing Framework

Project roadmap

1. 01

   Compute the posteriors

   ~2h

   Write a function posterior_beta(conversions, visitors, prior_alpha = 1, prior_beta = 1) that validates its inputs (non-negative whole numbers, and conversions not more than visitors) and returns the posterior Beta parameters (alpha + conversions and beta + failures), with a short explanation in a comment of why the Beta prior makes this exact. Add a function that returns each variant's posterior mean and 95% credible interval with qbeta. Done when: results for sample data match hand calculations.

2. 02

   Estimate the probability that B beats A

   ~1.5h

   Draw many samples (for example 100,000, with a set seed) from each posterior with rbeta, and estimate the probability that B's rate is higher than A's, the distribution of the difference and the relative lift, with their credible intervals. Check the estimate against the exact numerical answer (by integrating) for one example. Done when: the simulated probability matches the exact value closely.

3. 03

   Compute expected loss

   ~1.5h

   Compute the expected loss of choosing each variant: the average amount by which the chosen variant is worse, over the samples where it is worse. Write a decision function that recommends a variant only when its expected loss is below a threshold you set (such as 0.1 percentage points), and otherwise says to keep collecting data. Done when: the decision function gives sensible answers for clear, close and tiny-sample cases.

4. 04

   Visualize the results

   ~1.5h

   Write plotting functions with ggplot2: both variants' posterior densities overlapping, with the probability that B beats A in the subtitle, and a histogram of the difference with its credible interval and zero marked. Make them readable with clear labels and colors. Done when: the charts clearly show the results for sample data.

5. 05

   Study priors and peeking

   ~2h

   Run simulations: compare results with a flat prior and an informative prior based on past conversion rates, and simulate many A/A tests (no real difference) checked every day, counting how often the decision rule wrongly picks a winner. Summarize what you learned in README.md. Done when: README.md shows the effect of priors and of checking early, with numbers.

6. 06

   Package the functions

   ~2h

   Organize the functions as a small R package (with usethis): documented with roxygen2 (parameters, return values and examples), a vignette walking through a complete analysis, and a single summary function that prints a plain-language report. Done when: devtools::check() passes, and the vignette builds.

7. 07

   Test it

   ~1.5h

   Write testthat tests: input validation, posterior parameters, probability estimates against exact values, expected loss in obvious cases (a clearly better variant has near-zero loss), and reproducibility with a seed. Done when: devtools::test() passes.

Resources

• ArticleBayesian A/B testing (Evan Miller: formulas for Bayesian A/B testing)
• ArticleBeta distribution (Wikipedia)
• DocsBayes Rules! An Introduction to Applied Bayesian Modeling (free book)
• DocsR Packages (free book)
• Docsggplot2 documentation