Server Log Analyzer with Cats Effect

Project roadmap

1. 01

   Set up the project and a sample log

   ~1.5h

   Create an sbt project with Scala 3 and add cats-effect, fs2-core, fs2-io and MUnit (with munit-cats-effect). Write a small generator, or collect real logs, to produce a sample access.log with at least 10,000 lines in the combined log format, including a few malformed lines. Make the app an IOApp that takes a file path. Done when: sbt run -- sample/access.log prints the file path.

2. 02

   Model and parse log lines

   ~2h

   Define a case class LogEntry(ip, time: Instant, method, path, status: Int, bytes: Long, durationMs: Option[Long]). Write parse(line): Either[String, LogEntry] using a regex for the combined format, parsing the date with a DateTimeFormatter. Strip query strings from paths so /api/items?id=1 counts as /api/items. Done when: parse turns valid lines into entries and returns Left with a reason for broken ones.

3. 03

   Stream the file with fs2

   ~2h

   Read the file with fs2.io.file.Files[IO].readUtf8Lines, parse each line, and count broken lines instead of failing. The stream must never hold the whole file in memory. Print the number of parsed and skipped lines. Done when: a large generated log (for example 2 million lines) is processed with steady, low memory use.

4. 04

   Compute the statistics

   ~2.5h

   Fold the stream into a Stats value: request counts per path, counts per status class (2xx, 3xx, 4xx, 5xx), total and count of durations per path to compute averages, and request counts per minute. Make Stats combinable (a Monoid, or a combine method) so parts of the file could be analyzed separately and merged. Done when: the stats for the sample log are correct when compared with counts done by hand on a small file.

5. 05

   Find traffic spikes

   ~1.5h

   From the per-minute counts, compute the average and standard deviation, and flag minutes whose count is more than 3 standard deviations above the average. Make the threshold a command-line option. Done when: a sample log with an artificial burst of traffic reports that minute as a spike.

6. 06

   Print the report

   ~1.5h

   Print a formatted report: totals, the 10 busiest paths, the status class breakdown with percentages, the 5 slowest paths by average duration (only paths with at least 20 requests), and the spike minutes. Add a --json option for machine-readable output. Done when: the report is readable and --json produces valid JSON with the same numbers.

7. 07

   Write tests

   ~2h

   Write MUnit tests for the parser (valid lines, missing fields, bad dates, query strings), for combining two Stats values, for spike detection, and one test that runs the whole stream on a small fixture file and checks the final numbers. Done when: sbt test passes.

Resources

• DocsCats Effect documentation
• Docsfs2 guide
• DocsApache combined log format
• DocsScala 3 book
• Docsmunit-cats-effect