Help us measure site usage?

With your permission, we use self-hosted analytics to understand page usage. It is optional and does not affect the Service. Learn more.

codeseed
Sign in
Explore Projects
PerlbeginnercliAI generated

Text Statistics Tool

Build a Perl tool that analyzes any text file and prints interesting statistics: the number of words, lines and sentences, the most common words (ignoring common words such as the and and), the average and longest line and sentence, and phrases of two or three words that are repeated often, which is handy for spotting repetition in your own writing. Perl is famous for text processing, and this project shows why: you'll learn regular expressions for finding words and sentences, hashes for counting, sorting by value, reading files line by line, handling Unicode text, command-line options with Getopt::Long, and testing with Test::More.

Estimate
~6.5h
Steps
5
Completed by
0
Proposed by
codeseed.app

Perl core modules · Getopt::Long · Test::More

Sign in to claim this project

Project roadmap

  1. 01

    Define the report

    ~1h

    Write README.md: the command (textstats.pl FILE, with --top N, --min-length and --phrases 2 or 3), what counts as a word (letters, including accented ones, and apostrophes inside words), what ends a sentence, and an example of the full report. Pick a public-domain book from Project Gutenberg to test with. Done when: README.md defines the rules and shows example output.

  2. 02

    Read the text

    ~1h

    Open the file with UTF-8 decoding (open with <:encoding(UTF-8)) and read it line by line, counting lines and characters and tracking the longest line. Read from standard input when no file is given. Stop with a clear message when the file can't be opened. Done when: line and character counts match wc on a test file.

  3. 03

    Count words

    ~1.5h

    Extract words with a regular expression such as /(\p{L}+(?:'\p{L}+)*)/g, lowercase them, and count them in a hash. Skip stop words from a list (in a file, so it can be changed) and words shorter than --min-length. Sort by count and then alphabetically, and print the top N with their counts and percentages. Done when: the top words for a book make sense and exclude stop words.

  4. 04

    Find sentences and repeated phrases

    ~1.5h

    Split the text into sentences at ., ! and ? followed by a space and a capital letter or the end (knowing common abbreviations like Mr. can fool this, and noting it). Report the number of sentences and the average and longest sentence in words. Count two- and three-word phrases (n-grams) within sentences, and list those repeated at least 3 times. Done when: sentence counts are reasonable, and repeated phrases are listed.

  5. 05

    Test it

    ~1.5h

    Move the counting into subroutines and write tests with Test::More on small texts with known answers: word extraction (apostrophes, accents and numbers), stop words, sentence splitting, n-gram counting, and an empty file. Done when: prove -l t/ passes.

Resources

Ready to build this?

Get a GitHub repo and start building. Your AI reviewer checks each step as you go.

~6.5h · 5 steps

Tech stack

Perl core modulesGetopt::LongTest::More