Help us measure site usage?

With your permission, we use self-hosted analytics to understand page usage. It is optional and does not affect the Service. Learn more.

codeseed
Sign in
Explore Projects
PythonintermediateaiAI generated

Data Quality Profiler

Build a Python data quality profiler: give it a CSV or Parquet file and it examines every column, reporting the share of missing values, the number of unique values, the likely type (and values that don't fit it, such as text in a number column), outliers, suspicious patterns such as a single value repeated everywhere or dates in the future, and duplicate rows. The results are a readable HTML report with charts and a JSON file other tools can use, and rules you define (such as email must match a pattern) are checked too. It works on large files by reading them in chunks. You'll learn profiling data with pandas, type inference, outlier detection, writing data quality rules, producing reports with charts, and processing files larger than memory.

Estimate
~12h
Steps
6
Completed by
0
Proposed by
codeseed.app

pandas · matplotlib · Jinja2

Sign in to claim this project

Project roadmap

  1. 01

    Define the profile

    ~1.5h

    Write DESIGN.md: for each column, the metrics to compute (count, missing share, unique count, inferred type, share of values that don't match the type, minimum, maximum, mean and quartiles for numbers, most common values for text, and date ranges), the warnings (high missing share, constant columns, likely ids, outliers by the IQR rule, and future dates), the optional rules file (YAML with per-column rules), and the outputs (report.html and profile.json). Done when: DESIGN.md defines every metric, warning and output.

  2. 02

    Load data and infer types

    ~2h

    Load CSV (reading every column as text first) or Parquet with pandas. For each column, infer the most likely type (integer, decimal, boolean, date or text) by trying to convert values with pd.to_numeric and pd.to_datetime (with errors="coerce"), and count the values that don't fit. Done when: a sample file with mixed and messy columns gets sensible inferred types and mismatch counts.

  3. 03

    Compute metrics and warnings

    ~2.5h

    Compute every metric from DESIGN.md per column, find duplicate rows, and generate warnings with a severity and a short explanation (such as 38% of values are missing). Check the optional rules (required, range, pattern, allowed values and unique) and count violations with examples. Done when: the profile of a test file with planted problems reports each one.

  4. 04

    Handle large files

    ~2h

    For files larger than a threshold, read in chunks (pd.read_csv with chunksize) and combine the metrics across chunks: counts and sums add up, unique counts use a set up to a limit and then an estimate, and quantiles use a sample. Show progress. Measure memory use on a large generated file and note it in README.md. Done when: a file larger than your memory limit is profiled with steady memory use.

  5. 05

    Write the reports

    ~2h

    Write profile.json with every metric and warning, and an HTML report (with a Jinja2 template) showing a summary, the warnings sorted by severity, and a section per column with its metrics and a small chart (a histogram or top values bar chart made with matplotlib and embedded as an image). Done when: the HTML report is readable and matches the JSON.

  6. 06

    Test it

    ~2h

    Write pytest tests with small data frames and files: type inference for each type, missing and unique counts, each warning, rules, duplicate detection, and that chunked processing gives the same results as reading the whole file for a medium file. Done when: pytest passes.

Resources

Ready to build this?

Get a GitHub repo and start building. Your AI reviewer checks each step as you go.

~12h · 6 steps

Tech stack

pandasmatplotlibJinja2