Data Analysis and Interpretation Capstone - Milestone Assignment 3: Preliminary Results
Project title: Predicting the popularity of online content before it is published.
This weeks assignment was to post a draft of our preliminary results section from our final report. The research question can be found in my prior post here.
LASSO Regression Analysis
Figure 2 below includes all 58 explanatory variables. This figure shows that as each explanatory variable is added to the model the Mean Square Error (MSE) declines until additional predictors don’t further reduce the MSE.
Fig 2. LASSO Mean squared error on each fold
Table 1 below provides definitions for the top explanatory variables that were chosen by LASSO above.
Table 1 - definitions for the top explanatory variables.
Determining the ‘average’ and ‘worst’ keywords was determined by Fernandes, et al. [2] using the following process: “we rank all article keyword average shares (known before publication), in order to get the worst, average and best keywords. For each of these keywords, we extract the minimum, average and maximum number of shares.”. Keywords shared an average number of times are considered ‘average’ keywords. Keywords shared the least number of times are considered the “worst” keywords.
Table 2 below shows the LASSO mean squared errors across 58 explanatory variables. The small difference between the training and test data sets below shows that the model is good since the test result MSE is close to the training result MSE.
Table 2. LASSO MSE and R-squared.
Fig 3 below shows that ‘self reference avg shares’ and ‘avg shares of avg keywords’ were most strongly associated with sharing a Mashable article. When coupled with additional top explanatory variables chosen by LASSO (LDA_topic_0 and avg_shares_of_worst_keywords) implies that articles about a certain topic where particular keywords are present tend to be shared more often than other articles.
Fig 3. LASSO regression coefficients
Table 3 below shows the top chosen explanatory variables. These variables will be the focus for the remainder of this study.
Across all variables below the Pearson correlation coefficient ( r) and coefficient of determination (r^2) are quite low.
Table 3 - The top chosen explanatory variables.
Table 4 shows descriptive statistics for the top explanatory variables and the online shares response variable. These variables have already been converted via the log transform as mentioned above. Prior to the log transform there were 3,374 average shares per article with a minimum of 1 share for 1 article and a maximum of 690,400 shares for another article. There were 39,644 unique articles in this study.
Below are histograms for the top explanatory variables and the response variable. All variables below show a steep positive or negative skew except for number of hrefs and article subjectivity which appear more normally distributed.
<Note to reviewers: My final report will have more histograms. This is just a sample. >
Fig 4. Distribution of top explanatory variables and the online shares response variable.
Overall, the scatter plots below (Figure 5) don’t show a strong correlation among the top explanatory variables and the shares response variable. The articles with ‘avg shares of avg keywords’ had the strongest correlation to online shares (Pearson r=0.18, p< 0.0001). The ‘number of hrefs’ is likely a spurious correlation since the number of hrefs within the primary article shouldn’t sway a reader to share that article more or less.
<Note to reviewers: My final report will have more scatter plots. This is just a sample. >
Fig 5. Association between the top explanatory variables and online shares response variable.
It is generally accepted that multiple regression models are used to try to find one that can best predict the response variable. Across all models below the overall p-value was significant. 24 regression models were run each varying the top explanatory variables. The best overall results are included below. The best performing model is marked below in bold.
Table 5. Regression analysis results.
The model above in bold is the best because:
All coefficient p-values are significant (p-values = 0.000 < 0.05 alpha)
The overall model p-value is significant (p-value = 0.000 < 0.05 alpha)
This model has the highest overall adjusted r-squared (r^2 = 0.072) which is a measure of variance.
We can conclude that the chosen regression model can predict online shares fairly well.
The quantile-to-quantile plot in figure 6 below was created for the chosen model above. If the residuals (blue dots) were normally distributed they would follow the red line. However, the residuals generally follow the red line at the middle quantiles but slightly deviate at the lower and higher quantile. This deviation may have several causes:
There may be a curvilinear association between the variables included in this model.
The response variable has several large values well beyond the mean and 3 standard deviations from the mean.
There might be other explanatory variables that could be included in our model, beyond those in the original data set, that could improve estimation.
Fig 6. Quantile to quantile plot for the chosen model: Multiple regression with top explanatory variables.
To predict the popularity of online articles prior to publication this project used LASSO regression and other regression models. The data set included N = 39,644 samples that describe articles published on Mashable.com from January 7, 2013 to January 7, 2015.
The first step used relaxed LASSO and Pearson correlation to find the best subset of explanatory variables that could predict whether online articles would be ‘shared’ by readers when published. The strongest predictors of online shares ‘self reference avg shares’ and ‘avg shares of avg keywords’ and whether the article was published on the weekend. On one hand this implies that articles about a certain topic have a higher likelihood of being shared.
The best performing model had a low LASSO MSE between the training and test data sets as demonstrated in table 2 which means the model fits the training data well. The chosen model has an overall p-value lower than 0.05 alpha and an adjusted r-squared of .072 which indicates that the model has some good predictive ability. However, the quantile-to-quantile plot in fig 6 shows that this model had a fair number of residuals in the lower and upper quantiles which suggests a curvilinear relationship between the top variables or possibly other issues.
In summary, when comparing the overall results above to prior research:
Article subjectivity was one of the top ten predictors of online shares as seen in table 3. The subjectivity histogram in figure 4 above also shows articles that are slightly less subjective (more objective) is one indicator to determine whether it will be shared. However, according to Bandari, et al. [1] article subjectivity was not a good predictor towards sharing news blogs.
If a Mashable article was published on a weekday was somewhat important to predict subsequent shares. Table 3 shows the variable ‘is weekend’ was ranked #3 by LASSO regression coefficients. Also, a tentative relationship can be seen in the scatter plot in figure 4. If a Mashable article was published on a weekday it had a slightly better chance of being shared. This result seems to match the results from Szabo, et. al [4] in which sharing articles, via Digg, occurred more frequently on weekdays than on weekends.
Predicting human behavior is tricky such as predicting whether a user will share an online article. At the outset the ‘right’ variables may be present in the data, however they may not be sufficient to predict human behavior.
In addition, the results may show that a particular audience may prefer articles about a certain topic and thus those articles are more likely to be shared.
The chosen data set was from a single website, Mashable.com, which may draw a particular audience. The results described above may not generalize well to other websites or audiences.
Pearson’s correlation across all 58 explanatory variables and subsequent scatter plots of the top explanatory variables showed that there was no strong correlation between the explanatory variables and the response variable. This of course affects the ability to predict an outcome.
Based on prior research [1], [2], [3], [4], it does seem possible to accurately predict the popularity of online articles before publication. More advanced algorithms may be needed to make accurate predictions, algorithms such as Adaptive Boosting, Support Vector Machines, and Naive Bayes.
It would also be interesting to determine how much influential people acting as intermediaries increase an articles “share-ability”.
[1] Roja Bandari (University of California, Los Angeles), Sitaram Asur (HP Labs), Bernardo A. Huberman (HP Labs). The Pulse of News in Social Media: Forecasting Popularity 2012, International Conference on Weblogs and Social Media. http://arxiv.org/pdf/1202.0332.pdf
[2] K. Fernandes, P. Vinagre and P. Cortez. A Proactive Intelligent Decision Support System for Predicting the Popularity of Online News. Proceedings of the 17th EPIA 2015 - Portuguese Conference on Artificial Intelligence, September, Coimbra, Portugal. http://repositorium.sdum.uminho.pt/bitstream/1822/39169/1/main.pdf
[3] Sasa Petrovic, Miles Osborne, Victor Lavrenko. RT to Win! Predicting Message Propagation in Twitter. 2011, National Conference on Artificial Intelligence. http://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/2754/3209
[4] Gabor Szabo (Hewlett-Packard), Bernardo A. Huberman (Hewlett-Packard). Predicting the popularity of online content. 2010, Communications of The ACM, volume 53, issue 8, pp 80-88. https://www.researchgate.net/profile/Gabor_Szabo10/publication/23417017_Predicting_the_Popularity_of_Online_Content/links/00463529e2169e339e000000.pdf?disableCoverPage=true
[5] Chen, Edwin. Introduction to Latent Dirichlet Allocation. The link below was last retrieved on August 18, 2016. http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/
[6] “Advantages of doing “double lasso” or performing lasso twice?” (aka Relaxed LASSO). Last accessed the following URL on August 20, 2016. http://stats.stackexchange.com/questions/37989/advantages-of-doing-double-lasso-or-performing-lasso-twice
[7] Meinshausen, Nicolai. Relaxed Lasso. http://stat.ethz.ch/~nicolai/relaxo.pdf
