CSV Batch Transformer
Build a Bash tool that cleans CSV files in bulk: it checks selected columns against rules (required, numbers within a range, dates in a given format, values from an allowed list), normalizes values (trimming spaces, fixing case, converting dates), writes a clean output file, and puts every rejected row in a separate file with the reason. The heavy lifting is done by GNU Awk, which handles large files quickly, and input files can also be downloaded from a URL with curl. It can process a whole folder of files, skip files it already processed, and never leaves half-written output. You'll learn awk programming, handling quoted CSV fields, validation rules, safe file handling, and testing shell tools.
- Estimate
- ~10.5h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
curl · GNU Awk
Project roadmap
- 01
Define the rules and outputs
~1.5hWrite README.md: a rules file listing, per column name, rules such as required, int:0:120, date:YYYY-MM-DD, enum:active|inactive, email, and normalizations such as trim, lower and title; the outputs (FILE.clean.csv, FILE.rejects.csv with the line number and reason, and a summary); and the command (csvclean --rules rules.conf input.csv, or a URL or a folder). Done when: README.md describes the rules format and outputs with examples.
- 02
Parse CSV with awk
~2hUse GNU Awk's FPAT to split fields correctly when they contain commas inside double quotes (FPAT = "([^,]*)|(\"([^\"]|\"\")+\")"). Read the header to map column names to positions, and stop with a clear message when a column in the rules doesn't exist. Handle Windows line endings. Done when: a test file with quoted commas and quotes inside fields is split correctly.
- 03
Validate and normalize
~2.5hImplement each rule and normalization as an awk function. For each row, apply the normalizations, then the rules; write valid rows to the clean output and invalid ones to the rejects file with the line number and every failed rule. Keep the output quoted correctly. Done when: each rule catches its errors on a test file, and clean rows are normalized.
- 04
Download inputs with curl
~1hWhen the input is a URL, download it with curl --fail --silent --show-error, with retries and a timeout, to a temporary file, and stop with a clear message on HTTP errors. Clean up temporary files with a trap on EXIT. Done when: a CSV served by a local test server (such as python3 -m http.server) is processed, and a 404 gives a clear error.
- 05
Process folders safely
~1.5hWhen given a folder, process every .csv file in it. Write outputs to temporary files and rename them only after success, so a failure never leaves a partial clean file. Record a checksum of each processed input in a state file, and skip files already processed unless --force is given. Print a summary per file: rows read, kept and rejected. Done when: re-running on the same folder skips everything, and an interrupted run leaves no partial outputs.
- 06
Test it
~2hWrite tests with bats-core: each rule and normalization, quoted fields, missing columns, Windows line endings, an empty file with only a header, URL input with a local server, folder processing and skipping, and output quoting. Run shellcheck on the script. Done when: the tests pass, and shellcheck reports no warnings.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack