Pluggable Log Router
Build a log router in Perl that collects logs in different formats (Apache access logs, syslog, JSON lines and your application's own format), turns every line into one common event structure, and sends each event to the right destinations based on rules in a config file: for example errors to a database table, everything to daily files, and critical events to a webhook. Parsers and destinations are plugins, so adding a new format or destination means adding one module. Delivery is reliable: events for a destination that's down are kept in a spool and retried, and the router remembers how far it read each input, so a restart never loses or duplicates events. You'll learn plugin design in Perl, parsing, routing rules, DBI, buffering and retries, and testing.
- Estimate
- ~15h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
CPAN · DBI
Project roadmap
- 01
Design the router
~2.5hWrite DESIGN.md: the common event (timestamp, host, source, severity, message and extra fields), the parser plugin interface (can_parse and parse), the sink plugin interface (write a batch of events, reporting success or failure), the config file format (inputs, rules matching on source, severity or a field pattern, and destinations), and how read positions and the spool work. Done when: DESIGN.md covers the plugins, rules, positions and retries.
- 02
Write the parser plugins
~2.5hImplement parsers as modules under Router::Parser:: (Apache combined, syslog, JSON lines with JSON::PP, and a configurable regex parser), loaded by name from the config (with Module::Runtime or require). Each turns a line into a common event, normalizing times to UTC and severities to one scale. Count lines that no parser understands. Done when: unit tests parse sample lines of every format into correct events.
- 03
Route events
~2hLoad the config (YAML or JSON) and validate it at startup, stopping with clear errors. Evaluate rules for each event in order, sending it to every matching destination, with a rule option to stop after the first match. Read inputs line by line, following files as they grow. Done when: sample logs are routed to the destinations the rules say.
- 04
Write the sink plugins
~2hImplement sinks under Router::Sink::: daily rotating files, a database table through DBI (inserting in batches inside a transaction with prepared statements), and an HTTP webhook with LWP::UserAgent and a timeout. Each sink receives batches of events and reports success or failure. Done when: events reach files, the database and a test webhook.
- 05
Make delivery reliable
~2.5hWhen a sink fails, write its events to a spool file per sink and retry with increasing waits, delivering spooled events before new ones so order is kept. Save each input's read position (file and offset) only after its events are delivered or spooled, so a restart continues exactly where it stopped. Detect log rotation. Done when: stopping the database for a while and restarting the router loses and duplicates nothing.
- 06
Add limits and statistics
~1.5hLimit spool size per sink (warning loudly when full), batch size and webhook rate. Keep counters per input, rule and sink (events processed, unparsed, delivered, spooled and failed), and print them periodically or on a signal. Done when: the counters describe what the router did.
- 07
Test it
~2hWrite Test::More tests: each parser, rule matching, each sink (SQLite for DBI, a local HTTP::Daemon for the webhook), spooling and retrying, restarting from saved positions, and log rotation. Done when: prove -l t/ passes.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack