Basic Info and Quality Check

The data has 12904319 rows and 51 columns in total, and the most interesting colunmns are author, author_created_utc, author_premium, author_flair_text, author_fullname, body, collapsed, collapsed_reason, controversiality, created_utc, distinguished, gilded, no_follow, quarantined, removal_reason, score, send_replies, top_awarded_type, total_awards_received.

Most of the columns are either string type or boolean type, and the string columns are always descriptive variables, while the boolean columns are some features about the authors and the posts.

In agreement with the table in the right, 8 columns have NaN values, 5 of which have nearly or more than 50% of NaN values, therefore, these 5 columns are removed. Besides, those rows that contain NaN values are also dropped.

To check the length of the comments, each post body was split into a list of words and the average length of the comments is 22. Then those posts whose length is larger than 5 and less than 100 were filtered. So far we have 7753637 row left in the data.

↓

Exploratory Data Analysis

Mainly Short Posts

The subject of the subreddit is #PoliticalCompassMemes, which is about memes and pictures, so most of the posts (over 60%) are really short, with less than 20 words.

Premium Author Does Not Make a Difference

The post length between the premium authors and the non-premium authors does not a significant difference.

Drastic Increase of Number of Posts in Pandemic

After the breakout of the COVID (2020-01), the number of posts every month started to increase drastically. However, it can be recognized that this trend started to descend after the end of 2020.

Post Score Concentrate at Original Point

The length of posts and the score of posts are not likely to have a significant relationship. Nevertheless, most of the posts have a length of less than 20 and a score close to 0.

Right Gains a Higher Post Score

The posts from right related author_flair type tend to have a higher score, no matter whether the posts are from authors of Authoritarian or Libertarian.

Summary Tables

Create Dummy Variables

Regex was utilized here to figure out whether a post contains some key words or not, including "COVID", "Pandemic", "Election", etc. and these 6 dummy variables were created based on the matching results.

External Data

COVID cases data and Nasdaq index data were collected from CDC, then a regression plot was made for all these important variables. According to the plot, the number of posts and Nasdaq close price, the average post length and Nasdaq close price, and new COVID cases and the number of posts are likely to have an apparent correlation.

Summary

1. Most of the posts (over 60%) are less than 20 words.
2. There is not a significant difference of post length between the premium authors and the non-premium authors.
3. After the COVID (2020-01), the number of posts every month started to increase fast.
4. The length of posts and the score of posts are not likely to have a significan relationship.
5. The posts from right related author flair type tend to have a higher score.
6. According to the Figure 6, it can be recognized that number of posts and stock price, new COVID cases and number of posts are related.

Resources