Log Tail Aggregator
Build a log aggregator in Go that follows several log files at once (like tail -f), tags every line with its source, and streams the merged output live to anyone watching: over HTTP in the browser, and over gRPC for other programs. It handles log rotation, filters by source or text, and remembers how far it read each file in PostgreSQL, so after a restart it continues exactly where it stopped without duplicating or losing lines. Recent lines are also kept in PostgreSQL for a quick search of the last hours. Slow viewers never slow down reading. You'll learn following files and detecting rotation, fan-in and fan-out with goroutines and channels, gRPC server streaming, server-sent events, checkpointing, and testing concurrent code.
- Estimate
- ~12.5h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
gRPC · PostgreSQL
Project roadmap
- 01
Define the system
~1.5hWrite DESIGN.md: the config (sources with a name and file path or pattern), the line format (source, file, offset, time received and text), how offsets are checkpointed in PostgreSQL (per file, identified by path plus inode), how rotation is detected, the streaming interfaces (a gRPC Tail method with filters, and GET /stream as server-sent events), retention for stored lines, and what happens to slow subscribers. Done when: DESIGN.md covers sources, streaming, checkpoints and retention.
- 02
Follow files
~2.5hWrite a tailer per file that reads new lines as they're appended (using fsnotify, with polling as a fallback), handles partial lines, and detects rotation (the inode changes or the file shrinks) by finishing the old file and reopening the new one. Each tailer sends lines to a shared channel. Done when: lines appended to several files arrive on the channel, including across a rotation.
- 03
Checkpoint offsets in PostgreSQL
~2hStore each file's last processed offset in PostgreSQL (with pgx), updated every few seconds or every N lines, and store lines in a table in batches (with COPY) in the same transaction as the offset update, so both always agree. On startup, resume each file from its saved offset (or from the end for new files). Done when: restarting the aggregator neither repeats nor skips lines.
- 04
Stream over HTTP and gRPC
~2.5hBuild a hub that fans out lines to subscribers, each with a bounded buffer: when a subscriber falls behind, drop its oldest lines and tell it how many were skipped, so reading never slows down. Expose GET /stream?source=&contains= as server-sent events, and a gRPC Tail method with the same filters that streams lines to the client. Done when: a browser and a gRPC client both receive live, filtered lines.
- 05
Search recent lines
~1.5hAdd GET /search?source=&q=&since= querying the stored lines (with an index on time and source, and a trigram or full-text index for text search), and delete lines older than the retention period on a schedule. Done when: recent lines can be searched, and old ones are removed.
- 06
Test it
~2.5hWrite tests with temporary files and PostgreSQL in Testcontainers: following and rotation, resuming from checkpoints after a simulated crash (no duplicates or gaps), filters, a slow subscriber not blocking others, and the gRPC stream with an in-process server (bufconn). Run with -race. Done when: go test -race ./... passes.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack