Streaming Log Processor

Project roadmap

1. 01

   Define the records and outputs

   ~1.5h

   Write README.md: the input (newline-delimited JSON records with a timestamp, service, level, path, status and duration in ms, sent in batches to POST /ingest), the window size (1 minute, based on the record's own timestamp), how long to wait for late records (for example 30 seconds), and the output (one summary per service per window: count, error rate, and p50, p95 and p99 durations, written as JSON lines to a file). Done when: README.md defines input, windows and output with examples.

2. 02

   Receive logs with http4s

   ~2h

   Create an sbt project with http4s (Ember), Circe, fs2 and Cats Effect. Add POST /ingest, which reads the body as a stream of lines, decodes each with Circe, counts invalid lines and returns them in the response, and puts valid records on a bounded fs2 Queue. When the queue is full, the endpoint waits (slowing the sender down) and eventually returns 503. Done when: sending a batch of records returns counts of accepted and invalid records.

3. 03

   Aggregate in windows

   ~3h

   Consume the queue as an fs2 Stream and group records into windows by service and minute, keeping running counts, errors and durations. Close a window once the stream's latest timestamp has passed the window's end plus the lateness allowance, then emit its summary. Count records that arrive after their window closed as dropped. Done when: windows are emitted with correct counts, including records that arrive a bit out of order.

4. 04

   Compute percentiles

   ~1.5h

   Compute p50, p95 and p99 durations per window. Start by sorting the stored durations, then limit memory for busy windows with a fixed-size histogram of buckets or a sketch algorithm, and document how accurate it is. Done when: percentiles on a test dataset match the exact values within the documented error.

5. 05

   Write summaries reliably

   ~1.5h

   Write summaries as JSON lines to a file, rotating it daily, using fs2's file writing. On shutdown, flush and emit all open windows before exiting (with a graceful shutdown of the stream). Expose GET /stats with the queue size, records accepted, invalid and late, and the last window emitted. Done when: stopping the service writes all pending summaries, and /stats shows live numbers.

6. 06

   Test backpressure and load

   ~2h

   Write a small load generator that sends records faster than the processor can handle (add an artificial delay in processing), and check that memory stays bounded, senders are slowed down, and no accepted record is lost. Record the throughput in README.md. Done when: memory stays flat under overload, and every accepted record is counted.

7. 07

   Test the stream logic

   ~2h

   Write munit-cats-effect tests using TestControl to control time: records assigned to the right window, late records within and beyond the allowance, error rate and percentile calculations, flushing on shutdown, and the ingest endpoint's handling of invalid lines. Done when: sbt test passes.

Resources

• Docsfs2 guide
• Docsfs2 concurrency primitives (Queue)
• Docshttp4s documentation
• DocsCats Effect: TestControl
• ArticleStreaming 101 (windows and lateness)