Nginx Access Log Analyzer

Project roadmap

1. 01

   Define the input and report

   ~1.5h

   Write README.md: the expected log format (Nginx's combined format plus $request_time and $upstream_response_time, with the log_format line to add to nginx.conf), the command (logan.lua [--top N] [--path-prefix P] [--since TIME] FILES...), and the report sections: status codes, top paths by count and by 95th percentile time, and the overall latency distribution. Done when: README.md shows the log format and example output.

2. 02

   Parse log lines

   ~2h

   Write a parser using Lua patterns (string.match with captures) or LPeg that extracts the IP, time, method, path (without the query string), status, bytes, request time and upstream time, handling - for missing values. Return nil for lines that don't match and count them. Done when: busted tests parse sample lines correctly and reject malformed ones.

3. 03

   Aggregate statistics

   ~2h

   Count requests per status code and status class, and per path; collect request times per path and overall. Because storing every time for huge logs uses too much memory, keep times in a fixed set of buckets (a histogram, for example from 1 ms to 60 s on a logarithmic scale), and compute the median and percentiles from the buckets. Done when: percentiles from the histogram are close to exact values computed on a small log.

4. 04

   Read files and compressed logs

   ~1h

   Read files given as arguments line by line with io.lines, and .gz files through io.popen("gzip -dc FILE"), or standard input when no file is given. Add --follow to keep reading a live log as it grows and print an updated summary every few seconds. Report missing files and continue. Done when: plain and compressed logs are analyzed together, and --follow updates as lines are added.

5. 05

   Print the report

   ~1.5h

   Print aligned tables with string.format: status codes with counts and shares, the top N paths by count with their median and 95th percentile times, the top N slowest paths (with at least a minimum number of requests), and a text histogram of response times. Add --json output. Done when: the report is readable and matches the statistics.

6. 06

   Test it

   ~1.5h

   Write busted tests: parsing (including missing times and malformed lines), aggregation on a small fixture log with known counts, percentile accuracy of the histogram, path filtering, and JSON output. Done when: busted passes.

Resources

• DocsNginx: log_format and access_log
• DocsProgramming in Lua: patterns (first edition)
• DocsLPeg documentation
• DocsLua 5.4 reference manual
• Docsbusted testing framework