Log Aggregation & Anomaly Alerting Tool
Build a Java tool for operations teams that follows several application log files at once as they grow (like tail -f), understands each line, and raises an alert when errors suddenly spike, for example when the error rate jumps to five times its normal level within a few minutes. Each file is followed by its own thread, correctly handling log rotation (when a log file is renamed and a new one started), lines are parsed with regular expressions, and alerts are sent to a webhook without spamming the same alert again and again. A console summary shows volume and error rates per file. You'll learn concurrency with thread pools, WatchService and file reading, regular expressions, sliding window statistics, alert design, and testing time-based code.
- Estimate
- ~13.5h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
Java Threads · Regex · java.nio.file.WatchService
Project roadmap
- 01
Follow a log file
~2.5hWrite a Tailer class that opens a file, starts at the end, and reads new lines as they're appended, using WatchService (or polling when WatchService is unreliable, for example on network drives) to know when to read. Keep the read position, and handle partial lines (wait for the newline). Detect rotation (the file got smaller, or it's a different file) and reopen from the start. Done when: lines appended to a file are printed as they arrive, even across a rotation.
- 02
Parse log lines
~1.5hDefine a LogEntry record (timestamp, level, logger, message). Write a parser with configurable regular expressions per file (for example one for a Logback format and one for JSON logs), precompiled once. Attach lines that don't match, such as stack trace lines, to the previous entry. Done when: unit tests parse sample lines from both formats correctly, including stack traces.
- 03
Follow many files at once
~2hRead the list of files and their formats from a config file. Run one tailer per file on an ExecutorService, sending parsed entries to a shared BlockingQueue that a single processor thread reads. Stop everything cleanly on shutdown with a shutdown hook. A file that disappears is retried, not fatal. Done when: entries from 5 files are processed together without lost or mixed-up lines.
- 04
Detect error spikes
~2.5hKeep counts per file in one-second buckets over a sliding window (for example 5 minutes, stored in a ring buffer). Flag an anomaly when the error rate in the last minute is above a minimum count and more than N times the average of the previous window (or more than 3 standard deviations above it). Use an injectable Clock so this can be tested. Done when: a simulated burst of errors is flagged, and normal fluctuations are not.
- 05
Send alerts
~1.5hSend an alert to a webhook URL (with java.net.http.HttpClient, with a timeout and one retry) containing the file, the current and normal error rates, and a few example error messages. Don't send the same alert again for 10 minutes (a cooldown), and send a resolved message when the rate returns to normal. Done when: a spike sends one alert, and its end sends one resolved message.
- 06
Show a console summary
~1.5hEvery 30 seconds, print a table per file: lines per minute, errors per minute, the current state (normal or alerting) and time since the last line. Warn when a file has been silent for too long, which can also mean a problem. Done when: the summary shows accurate numbers for every file.
- 07
Test it
~2hWrite JUnit 5 tests: the parser, partial lines and rotation with temporary files, the sliding window and anomaly detection with a fake clock, alert cooldown and resolution with a fake webhook (for example a local HttpServer), and the processor with several tailers. Done when: all tests pass reliably.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack