Sentiment Analysis of Tweets with tidytext
Build an R analysis that measures the mood of a collection of tweets (or other short posts) over time. Using the tidytext package, you'll split the text into words, remove noise such as links, mentions and common words, score each word with a sentiment dictionary (positive or negative, or a score from -5 to +5), and add the scores up per tweet and per day. The result is a set of ggplot2 charts showing how sentiment rises and falls, which words drive it, and where the notable spikes are. You'll learn text mining the tidy way, joining data with dplyr, the strengths and limits of dictionary-based sentiment analysis, time-based aggregation with rolling averages, and presenting findings in a report.
- Estimate
- ~10h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
tidytext · dplyr · ggplot2
Project roadmap
- 01
Load the data and split it into words
~1.5hChoose a dataset of short posts with timestamps (for example a public tweets dataset from Kaggle, or product reviews with dates). Load it with readr, keep an id, the time and the text, and parse the time with lubridate. Split the text into one word per row with unnest_tokens. Done when: you have a table with one row per word, linked to its post and time.
- 02
Clean the words
~1.5hBefore splitting, remove URLs, @mentions and HTML entities with stringr. After splitting, remove stop words with anti_join(stop_words), numbers, and very short words. Count the most common remaining words and check that they make sense, adding extra stop words specific to your data when needed. Done when: the 20 most common words are meaningful words about the topic.
- 03
Score sentiment
~2hJoin the words with the bing dictionary (positive or negative) and the AFINN dictionary (scores from -5 to +5) using get_sentiments and inner_join. Calculate a score per post for each dictionary, and keep posts with no matching words as neutral. In comments, note the limits of this method, such as not understanding negation (not good) or sarcasm. Done when: every post has a score, and a few examples read by hand look right.
- 04
Aggregate over time
~1.5hGroup the scores by day (or hour, for short periods), calculating the average score, the number of posts and the share of positive posts. Add a 7-day rolling average (for example with slider or zoo), and find the days most above or below the average. Done when: you have a daily table with averages, counts and a rolling average.
- 05
Visualize the results
~2hMake ggplot2 charts: sentiment over time with the rolling average and the unusual days labeled, the words contributing most to positive and negative sentiment (a bar chart per side), and a comparison of the bing and AFINN results. Give every chart a clear title and axis labels. Done when: the charts are saved as PNG files, and each one tells a clear story.
- 06
Write the report
~1.5hPut the analysis in an R Markdown (or Quarto) report: the data and cleaning, the method, the charts with a short explanation each, a look at the posts behind the biggest spikes, and the limits of the method. Done when: rendering the report produces an HTML file that someone who didn't write the code can understand.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack