Distributed Word Count with Akka Actors
Build a word counting system in Scala with the actor model, using Akka: a master actor splits a large collection of texts into chunks, sends them to a pool of worker actors that count words in parallel, and merges their partial counts into the final result, such as the 100 most common words across hundreds of books. Actors communicate only by sending messages, never by sharing memory, and a supervision strategy restarts a worker that crashes and resends its chunk, so the total is still correct. You'll compare the speed with a simple single-threaded version. Note that Akka's newer versions use a commercial license; Apache Pekko is a free, compatible alternative. You'll learn actors, message protocols, supervision, parallel work distribution and testing with TestKit.
- Estimate
- ~11h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
Akka/Pekko Actors · Akka/Pekko TestKit
Project roadmap
- 01
Design the message protocol
~1.5hCreate an sbt project with Akka Typed (or Pekko) and its TestKit. Define the messages as sealed traits: the master receives Start(files) and ChunkCounted(chunkId, counts) and ChunkFailed(chunkId); workers receive CountChunk(chunkId, text, replyTo). Write DESIGN.md explaining the flow, how results are merged, and what happens when a worker fails. Done when: the message types compile, and DESIGN.md explains the flow.
- 02
Write the worker actor
~1.5hWrite a worker behavior that counts words in its chunk: lowercase, split on anything that isn't a letter, skip empty words and optional stop words, and count with a Map. It replies with the counts. Keep the counting in a pure function that can be tested alone. Done when: unit tests of the counting function pass, and a worker replies with correct counts for a sample chunk.
- 03
Write the master
~2.5hWrite a master behavior that reads the input files, splits them into chunks of about 1 MB (without cutting words in half), creates a pool of N workers (with a Router or by spawning them), sends out chunks while tracking which are pending, and merges each reply into the running totals. When every chunk is counted, it reports the top K words and the total word count, then stops. Done when: counting a set of books gives the same result as a simple sequential count.
- 04
Handle worker failures
~2hMake workers fail sometimes on purpose (for example on a special marker in the text, or randomly in a test mode). Supervise them with a restart strategy, have the master watch for failed or timed-out chunks (using ask with a timeout or a timer) and resend them, with a limit on retries per chunk. Make sure a chunk is never counted twice if a late reply arrives. Done when: with failures injected, the final counts are still exactly correct.
- 05
Benchmark against a sequential version
~1.5hWrite a plain single-threaded version (and optionally a version with parallel collections) and time all versions on a large corpus (for example 200 MB of Project Gutenberg books) with different numbers of workers. Record the results in README.md and explain where the speedup stops growing and why. Done when: README.md shows the timings and explains them.
- 06
Write actor tests
~2hWrite tests with ActorTestKit and TestProbe: a worker replies with correct counts, the master sends every chunk and merges replies, failed chunks are resent, late duplicate replies are ignored, and the final result matches the sequential version on a small corpus. Done when: sbt test passes.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack