A/B Test Calculator
Build an A/B test calculator in R with Shiny: upload the results of an experiment (for example visitors and conversions for version A and version B of a web page, or a numeric measure such as time on page per user) and get an honest answer about whether B is really better. The app shows the difference between the groups with a confidence interval, the p-value explained in plain words, and the effect size, checks for common mistakes such as groups of very different sizes, and includes a sample size planner that tells you how many users you need before starting a test. It uses the infer package from tidymodels for tests by simulation. You'll learn hypothesis testing, confidence intervals, effect sizes and statistical power, simulation-based inference, and explaining statistics clearly.
- Estimate
- ~11h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
Shiny · tidymodels
Project roadmap
- 01
Define the inputs and outputs
~1.5hWrite README.md: the two kinds of test (conversion rates from counts, and a numeric measure from one row per user with a group column), the uploaded file formats, the outputs (each group's rate or mean, the difference with a 95% confidence interval, the p-value, the effect size and a plain-language conclusion), and the checks the app makes. Prepare sample files for both kinds. Done when: README.md defines the inputs, outputs and checks.
- 02
Compute the statistics
~2.5hWrite pure R functions: for conversions, the difference in proportions with a confidence interval and a two-proportion test (prop.test), plus the relative lift; for numeric measures, the difference in means with Welch's t-test and Cohen's d. Add a simulation-based version with infer (specify, hypothesize, generate and calculate) and compare the results. Done when: the functions give correct results on examples checked against online calculators.
- 03
Check for common mistakes
~1.5hAdd checks that warn the user: a sample ratio mismatch (group sizes very different from the planned split, tested with a chi-square test), very small samples, a test stopped before reaching the planned size, and outliers in numeric measures. Done when: sample files with each problem produce the right warnings.
- 04
Plan the sample size
~1.5hWrite a planner: given the current conversion rate, the smallest improvement worth detecting, the significance level and the power, calculate the users needed per group (with power.prop.test, or power.t.test for numeric measures) and how many days that takes at a given daily traffic. Show a chart of sample size against the detectable effect. Done when: the planner's numbers match standard power calculators.
- 05
Build the Shiny app
~2.5hBuild tabs for analyzing results (upload or type in counts, then see the numbers, a chart of each group's estimate with its interval, the warnings and a plain-language conclusion such as B's conversion rate is between 0.3 and 1.9 points higher) and for planning. Explain what a p-value does and doesn't mean in a short note. Validate inputs and show clear errors. Done when: both tabs work with the sample files and typed-in numbers.
- 06
Test it
~1.5hWrite testthat tests for the statistics functions against known results, the mistake checks, the planner, and input validation (negative counts or conversions larger than visitors). Add a shinytest2 test for the analysis tab. Done when: all tests pass.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack