Apache Access Analyzer

Project roadmap

1. 01

   Define inputs, outputs and storage

   ~1.5h

   Write README.md: the input is Apache logs in combined format (plain or .gz), the reports (hourly traffic, status codes, top pages, slowest pages if the log has a response time field, top error sources), and the SQLite tables (hourly totals per path and status, plus the imported files with their size and checksum). Set up a cpanfile with DBI, DBD::SQLite and LWP::UserAgent. Done when: README.md shows example input and output.

2. 02

   Parse log lines

   ~2h

   Write a parser module with a regular expression for the combined format that extracts the IP, time, method, path, status, size, referrer and user agent. Convert the time to UTC and truncate it to the hour. Remove query strings from paths so /page?id=1 and /page?id=2 count as one. Count lines that don't match instead of failing, and show a few examples. Done when: a sample log parses with the right fields, and malformed lines are counted.

3. 03

   Aggregate and store with DBI

   ~2.5h

   Aggregate in memory by hour, path and status (count, bytes, total response time), then write the totals to SQLite with prepared statements inside one transaction per file. Record the file's checksum in the imported files table in the same transaction, and skip any file already imported, so running the import twice counts nothing twice. Done when: importing the same log twice leaves the totals unchanged.

4. 04

   Produce reports

   ~2h

   Add a report command with --from and --to dates that runs SQL queries on the stored totals and prints: requests per hour as a simple text bar chart, status codes with percentages, the 10 busiest paths, the slowest paths by average time, and the top IPs producing errors. Add --format json. Done when: the reports match counts made by hand on a small log.

5. 05

   Post a summary with LWP

   ~1h

   Add --webhook URL, which posts a short daily summary (total requests, error rate, busiest page) as JSON using LWP::UserAgent with a timeout, retrying once on a network error. A failed post prints a warning but never loses the imported data. Done when: the summary reaches a test webhook, and an unreachable URL only causes a warning.

6. 06

   Test it

   ~2h

   Write tests in t/ with Test::More: parsing valid and malformed lines, removing query strings, hour grouping across midnight, importing twice (the second import is skipped), and report numbers on a small fixture log with a temporary SQLite database. Done when: prove -l t/ passes.

Resources

• DocsApache log files (combined log format)
• DocsDBI on MetaCPAN
• DocsDBD::SQLite on MetaCPAN
• DocsLWP::UserAgent on MetaCPAN
• Docsperlretut: Perl regular expressions tutorial