Embedded Query Engine
Build a small analytical query engine in C++ that runs queries such as "total sales per region for 2025, where the amount is over 100" over large local datasets, like a tiny version of DuckDB. Data is imported from CSV into a column-oriented format stored on disk (each column in its own file), so a query only reads the columns it needs. You'll implement the core operations yourself, filters, projections, group-by with aggregations, and sorting, as operators that process data in batches of values for speed, and check that results are always correct against a reference. You'll learn columnar storage, binary file formats, query operators and pipelines, hash aggregation, performance measurement, and testing correctness.
- Estimate
- ~16h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
CMake · Boost.Program_options · mmap
Project roadmap
- 01
Design the engine
~2.5hWrite DESIGN.md: supported column types (64-bit integers, doubles, strings and dates), the on-disk format (a folder per table with a schema file and one binary file per column, strings stored as offsets plus data, and a checksum per file), the query interface (built in code, or a tiny query language such as SELECT region, SUM(amount) FROM sales WHERE year = 2025 GROUP BY region), and the operator model (each operator returns batches of about 1,024 rows). Done when: DESIGN.md covers storage, queries and operators.
- 02
Import CSV into columns
~2.5hWrite an importer that reads a CSV with a given schema, parses and validates every value (reporting the row and column of bad values), and writes each column to its own file. Write files to a temporary folder and rename it when complete, so a failed import never leaves a half-written table. Use Boost.Program_options for the command-line options. Done when: importing a sample sales CSV creates the table files, and bad values are reported.
- 03
Scan and filter
~2.5hWrite a Scan operator that reads only the requested columns in batches (with memory-mapped files or buffered reads), and a Filter operator that evaluates comparisons (=, <, >, between, and and or) on a batch and passes on only matching rows, using a selection vector. Verify checksums when opening a table. Done when: a filter query over a million rows returns exactly the right rows.
- 04
Group and aggregate
~2.5hWrite a HashAggregate operator that groups rows by one or more columns in a hash table and computes COUNT, SUM, MIN, MAX and AVG. Then add Sort and Limit operators, and a Project operator for choosing and computing columns. Connect operators into a pipeline from a simple query plan. Done when: the example query from DESIGN.md returns correct totals per region.
- 05
Add a query interface
~2hWrite a small parser for your query language (or a builder API), with clear errors for unknown tables, unknown columns and type mismatches (such as comparing a string to a number). Add an interactive prompt that prints results as a table with the query time. Done when: typed queries run and show results, and mistakes give clear messages.
- 06
Measure and optimize
~2hGenerate a dataset with tens of millions of rows and time a set of benchmark queries. Try at least two optimizations, such as skipping blocks using stored min and max values per block, or running scans and aggregation on several threads, and record the timings before and after in README.md. Done when: README.md shows measured speedups for each optimization.
- 07
Test correctness
~2hWrite tests comparing your engine's results with the same queries run in SQLite on the same data, for many random queries (random filters, groupings and aggregates), plus tests for import errors, checksum failures and empty results. Run the tests with AddressSanitizer. Done when: all results match SQLite, with no sanitizer errors.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack