Bayesian A/B Testing Framework

Project roadmap

1. 01

   Model conversions with Beta-Binomial priors

   ~2h

   Write a function that takes conversion counts for two variants and computes posterior Beta distributions.

2. 02

   Simulate posterior samples

   ~1.5h

   Draw random samples from each posterior and estimate the probability that variant B outperforms A.

3. 03

   Compute expected loss and credible intervals

   ~2h

   Calculate expected loss for choosing each variant and 95% credible intervals for the conversion rate difference.

4. 04

   Visualize posterior distributions

   ~1.5h

   Plot overlapping posterior density curves for both variants with the probability-of-improvement annotated.

5. 05

   Package as reusable functions

   ~1.5h

   Organize the functions into a small package-like structure with documentation and example usage.

Resources

• DocsR Manuals