High-Performance CSV Query Engine

Project roadmap

1. 01

   Set up the project and test data

   ~1.5h

   Create a crate with csv, rayon, clap and anyhow. Write a generator (a small binary or script) that creates a sales CSV with columns such as date, region, product, quantity and amount, with a configurable number of rows, and generate a 5-million-row file for benchmarks (don't commit it). The CLI takes a file and a query string. Done when: the generator creates the file, and csvq --help works.

2. 02

   Read the CSV and detect types

   ~2h

   Read the header with the csv crate, then sample the first 1,000 rows to decide each column's type: integer, float or text. Read rows as ByteRecord and convert only the fields a query needs, to save time. Report rows that don't match the detected types as warnings, with their line numbers. Done when: the tool prints each column with its detected type.

3. 03

   Define and parse the query language

   ~3h

   Write down the grammar in the README: an optional where clause with comparisons (=, !=, <, <=, >, >=) joined by and and or, an optional group by with column names, and select with count, sum(col), avg(col), min(col) and max(col). Write a tokenizer and a recursive descent parser that produces an AST, with error messages showing where the problem is, such as expected a column name at position 23. Done when: valid queries parse into ASTs, and invalid ones give helpful errors.

4. 04

   Filter rows

   ~2h

   Compile the where clause into a function that tests one row, checking that the columns exist and the types match (comparing a text column with a number is an error). Apply it while streaming through the file. Handle empty fields as null, which never match a comparison. Done when: queries with filters return exactly the rows you expect on a small hand-made CSV.

5. 05

   Aggregate with and without groups

   ~2.5h

   Implement an accumulator for each aggregate (count, sum, min, max, and avg as sum plus count) that can be merged with another accumulator of the same kind. Without group by, produce one row; with group by, keep a HashMap from the group key to accumulators. Print the result as an aligned table, sorted by the group key. Done when: aggregate results match values calculated another way, for example with a spreadsheet, on the test file.

6. 06

   Process in parallel with rayon

   ~2.5h

   Read the file in chunks of about 64,000 rows and process the chunks in parallel with rayon: each chunk filters its rows and builds its own partial aggregates, then the partial results are merged. Make sure the results are identical to single-threaded processing, sorting groups the same way. Add --threads to control rayon's thread pool. Done when: parallel and single-threaded runs give identical results.

7. 07

   Benchmark and test

   ~2h

   Measure the time for three typical queries on the 5-million-row file with 1, 2, 4 and all threads, running each several times, and write the results and your explanation (where the time goes, and why the speedup isn't perfect) in BENCHMARK.md. Write tests for the parser, the filters, each aggregate and group by. Done when: cargo test passes, and BENCHMARK.md contains the measurements.

Resources

• Docscsv crate tutorial
• Docsrayon crate
• DocsCrafting Interpreters: parsing expressions
• DocsThe Rust Performance Book
• Docsclap documentation