Model Comparison Service
Build a model comparison tool in R: you upload a dataset and choose what to predict, and it trains several machine learning models on it (for example logistic regression, a random forest and gradient boosting), evaluates them fairly with the same cross-validation folds, and shows their scores side by side with diagnostic charts, so you can pick the best one with confidence. Built with tidymodels, it follows good practice: preprocessing learned only from training data, a final test set never used for choosing, and results that can be reproduced. A Shiny app runs the comparisons in the background, keeps a history of past runs, and never freezes while models train. You'll learn tidymodels workflows, resampling and tuning, avoiding data leakage, and building a responsive Shiny app.
- Estimate
- ~15h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
Shiny · tidymodels
Project roadmap
- 01
Design the tool
~2hWrite DESIGN.md: the input (a CSV and the target column, detected as classification or regression), the fixed test split and cross-validation folds (with a recorded random seed), the candidate models, the metrics (for example ROC AUC and accuracy, or RMSE and R-squared), what's saved for each run (settings, seed, metrics, fitted final model), and how long-running training is handled in the app. Done when: DESIGN.md covers data, models, evaluation and storage.
- 02
Build the comparison pipeline
~3hWrite functions (in plain R, separate from Shiny): validate the dataset (target present, enough rows, not too many missing values), split it with initial_split (stratified for classification), build a recipe (impute, one-hot encode and normalize), and combine the models into a workflow_set. Evaluate them all with workflow_map using the same vfold_cv folds, tuning each model's key settings with a small grid. Done when: the pipeline compares all models on a sample dataset and returns a metrics table.
- 03
Finalize and diagnose the best model
~2hPick the best model by the chosen metric, fit it on the full training set with last_fit, and evaluate it once on the test set. Produce diagnostics: a ROC curve and confusion matrix for classification, or predicted against actual and residual plots for regression, plus variable importance with the vip package. Done when: the final test scores and diagnostic charts are produced for the best model.
- 04
Save runs and make them reproducible
~1.5hSave each run to a folder with its id: the settings, seed, package versions (from sessionInfo), metrics, charts and the fitted model (as an RDS file). Rerunning with the same settings and seed must give the same results. Done when: two runs with the same seed produce identical metrics.
- 05
Build the Shiny app
~3hBuild a Shiny app to upload a CSV, choose the target and models, and start a comparison. Run training in the background with ExtendedTask (or future and promises) so the app stays responsive and shows progress, allowing only one run at a time per user. Show the ranked metrics table, charts, and a list of past runs that can be opened again. Done when: the app stays usable during training and shows results when it finishes.
- 06
Handle bad input and limits
~1.5hHandle datasets that are too large (a row and size limit), targets with only one class or too many missing values, text columns with thousands of distinct values, and models that fail to train (record the failure and continue with the others). Show clear messages in the app for each. Done when: each of these cases gives a clear message instead of an error.
- 07
Test it
~2hWrite testthat tests for the pipeline on small built-in datasets (such as mtcars for regression and a two-class dataset for classification): validation errors, the same folds used for every model, no test data used before the final evaluation, reproducibility with a seed, and handling a failing model. Add a shinytest2 test for the main flow. Done when: all tests pass.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack