Distributed Key-Value Store with Replication

Project roadmap

1. 01

   Build a single-node store

   ~2h

   Create a project with tokio, serde, bincode (or serde_json) and tokio-util. Define request and response message types with Serde (Get, Set, Delete and their results), sent as length-prefixed frames (with LengthDelimitedCodec). Build a server that keeps data in a HashMap behind a lock and handles many clients at once, plus a small client command. Done when: several clients can set and get keys on one node.

2. 02

   Stream writes to followers

   ~2.5h

   Give every write on the primary a sequence number and append it to an in-memory log (and optionally a file). Followers connect to a separate replication port and send the last sequence number they have; the primary sends the log entries after it and then streams new writes as they happen, using a broadcast channel. Followers apply entries in order. Done when: writes on the primary appear on connected followers within milliseconds.

3. 03

   Catch up after reconnecting

   ~2.5h

   Keep the log bounded: when it grows too large, take a snapshot of the data at a sequence number and drop older log entries. A follower asking for entries older than the log's start receives the snapshot first, then the log after it. Followers reconnect with increasing waits after a disconnect and save their data and last sequence number to disk so a restart continues from there. Done when: a follower stopped during many writes catches up completely when it returns, whether from the log or a snapshot.

4. 04

   Route reads and writes

   ~1.5h

   Followers reject writes with a message pointing to the primary's address, and serve reads from their copy; responses include the node's last applied sequence number, so clients can tell how up to date the answer is. Add an option for clients to require reads at least as recent as a given sequence number (waiting briefly or failing). Done when: writes to followers are redirected, and reads report their freshness.

5. 05

   Test failure scenarios

   ~2.5h

   Write integration tests that start a primary and two followers on random ports inside the test: replication of many writes, a follower disconnected mid-stream and reconnected, a follower restarted from its saved state, catch-up through a snapshot, a slow follower that falls behind, and redirected writes. After each scenario, check every node has exactly the same data. Done when: cargo test passes reliably when run several times.

6. 06

   Add observability

   ~1h

   Log (with tracing) each follower's connection, catch-up method and lag in sequence numbers, and add a STATUS command showing the node's role, last sequence number, and for the primary each follower's lag. Document the design and its limits in README.md (for example, what happens if the primary crashes, which this project doesn't handle automatically). Done when: STATUS shows each follower's lag, and README.md explains the design and limits.

Resources

• DocsTokio tutorial
• Docstokio-util: LengthDelimitedCodec
• DocsSerde documentation
• ArticleDesigning Data-Intensive Applications: replication (book site)
• DocsRedis replication (a design to compare with)