Help us measure site usage?

With your permission, we use self-hosted analytics to understand page usage. It is optional and does not affect the Service. Learn more.

codeseed
Sign in
Explore Projects
PerlintermediatecliAI generated

Website Link Auditor

Build a Perl tool that checks a website for broken links: it starts at a page, follows links within the same site up to a chosen depth, and checks every link it finds, including links to other sites (checked but not followed). The report lists broken links (404s, server errors, timeouts) with the pages they appear on, and redirect chains, such as http to https to a new address, that should be updated. Results are stored in SQLite through DBI, so you can compare runs and see what broke since last time. It's polite: it respects robots.txt and waits between requests. Only audit websites you own or have permission to check. You'll learn LWP::UserAgent, parsing HTML for links, URL handling, crawling with limits, DBI, and reporting.

Estimate
~10.5h
Steps
6
Completed by
0
Proposed by
codeseed.app

DBI · LWP::UserAgent

Sign in to claim this project

Project roadmap

  1. 01

    Define the audit

    ~1.5h

    Write README.md: the command (linkaudit.pl START_URL with --depth, --max-pages, --delay and --timeout), what's followed (same host only) and what's only checked (other hosts, images and scripts), what counts as a problem (4xx and 5xx statuses, network errors and redirect chains longer than one step), and the report. Set up a cpanfile with LWP::UserAgent, HTML::LinkExtor (or Mojo::DOM), URI, WWW::RobotRules and DBD::SQLite. Done when: README.md defines the rules and outputs.

  2. 02

    Fetch pages and extract links

    ~2h

    Fetch pages with LWP::UserAgent (with a timeout, a descriptive agent name, and max_redirect set to 0 so you can follow redirects yourself and record each step). Extract links from a, img, script and link tags, resolve them against the page URL with URI, and remove fragments. Done when: a test page's links are extracted as absolute URLs.

  3. 03

    Crawl the site

    ~2h

    Crawl breadth-first with a queue of URLs and depths, a hash of seen URLs, and limits on depth and pages. Follow only same-host HTML pages, and check external links with a HEAD request (falling back to GET when HEAD isn't allowed). Respect robots.txt and wait --delay seconds between requests to the same host. Record for every link the page it was found on. Done when: crawling a local test site checks every link once and stays within the limits.

  4. 04

    Record redirects and errors

    ~1.5h

    Follow redirects manually up to 10 steps, recording each step, and flag chains with more than one step and loops. Record timeouts and DNS errors separately from HTTP errors. Retry a failed request once after a pause before marking it broken. Done when: redirect chains and different error types are recorded correctly.

  5. 05

    Store results and compare runs

    ~1.5h

    Store each run and its results (URL, status, final URL, redirect steps, error and the pages linking to it) in SQLite through DBI, inside a transaction. Add a --compare option that shows links broken now that worked in the previous run, and links that were fixed. Done when: two runs of a site with a newly broken link report it as new.

  6. 06

    Report and test

    ~2h

    Print a report grouped by problem type with the linking pages, and optionally write CSV. Write Test::More tests with a local test site (served by HTTP::Daemon or Plack) containing working links, broken links, redirect chains, a redirect loop, a slow page and a robots.txt rule. Done when: the report is correct for the test site, and prove -l t/ passes.

Resources

Ready to build this?

Get a GitHub repo and start building. Your AI reviewer checks each step as you go.

~10.5h · 6 steps

Tech stack

DBILWP::UserAgent