Feedback Rates on FFN and AO3: Significance Testing
Introduction
In January 2018, we began collecting data to determine whether feedback rates differ between fanfiction.net (FFN) and archiveofourown.org (AO3). This is a follow-up to our initial data characterization, in which we will determine whether or not the results we found are statistically significant.
Please read the initial data characterization for a discussion of study design and limitations, and the project overview for goals and other sections of our analysis.
Note: this study focuses only on comments and reviews, and does not take into account other forms of feedback, including kudos, favorites, bookmarks, or recommendations.
Results
The data we collected include a total of 521 viable fics on FFN and 522 viable fics on AO3. FFN results show means of 1,008.2 views, 5.01 reviews per fic, and 9.00 reviews for every thousand views. AO3 results show means of 1,339.5 hits, 4.72 comment threads per fic, and 7.13 comment threads for every thousand hits.
Overall, our data show that FFN has fewer views, but more reviews per fic and per thousand hits than AO3.
In order to determine the statistical significance of the differences, we subtracted the AO3 feedback rate (comments divided by hits) from the FFN feedback rate. Outliers (1.5 IQR above Q3 or below Q1) were removed, resulting in a sample of 419 values. We confirmed that the resulting distribution was approximately normal. Then, we performed a t-test to show whether the resulting mean of 0.002 (ie, FFN has a mean of 2 reviews per thousand hits than AO3) was significant.
Figure 1. Frequency distribution of AO3 feedback rate (comments/hits) subtracted from FFN feedback rate (reviews/views), with outliers 1.5 IQR above Q3 or below Q1 removed.
Using H0 = 0, n= 419, s.d. = 0.00463463 we get a t-stat of 8.987783 which translates into a p-value of 8.790e-18. In other words, p < 0.001, which is well below the generally accepted threshold of p = 0.05 (a 95% probability that results are not due to random chance). Therefore, we conclude that there is a statistically significant difference between the feedback rates on FFN and AO3 as measured by comments/hits.
When data are normally distributed, approximately 68% of differences will fall within one standard deviation of the mean (0.002 ∓ 0.0046), 95% will fall within two standard deviations (0.002 ∓ 0.0093), and 99.7% will fall within three standard deviations (0.002 ∓ 0.0139). However, due to the fact that removing outliers took out nearly 20% of our data pairs, we will need to explore this further to determine whether this accurately describes fics - a brief glance suggests that outliers primarily consist of stories that have zero comments or reviews on one site and a nonzero amount of feedback on the other site, indicating that these results do apply to fics that have at least one comment or review on both sites.
We also ran our data through the Wilcoxon signed-rank test. Following this methodology, we got n = 497, T = 41,964.5, mean = 61,876.5, and var = 10,261,186.3 resulting in a Z-score of 6.2161 with a p-value of 5.0975e-10. Once again, we find that these results are highly significant.
Relationship Between Feedback Rates
Correlation between FFN reviews-to-views ratio and AO3 comment threads-to-hits rate is 0.2567 for all viable fics (n = 521), which means that there is not a strong relationship between a fic’s feedback rate on AO3 and its feedback rate on FFN. This indicates that a story’s level of popularity and audience engagement is not determined solely by the fic itself, but is instead influenced by a number of other factors.
Figure 2. Graph showing the relationship between the FFN reviews-to-views ratio and the AO3 reviews-to-views ratio of the 521 viable data pairs. Not all data points are shown.
Conclusions
Our results indicate that the difference in feedback rates (in this case, comments or reviews per hit) between FFN and AO3 are highly significant (p < 0.001). We also found that FFN has a mean of 2 additional reviews per thousand hits as compared to the same fic on AO3. Note that this does not indicate that every fic will receive a higher feedback rate on FFN: in our initial characterization, we found that out of stories that have at least one review or comment on each site, 29% have a higher feedback rate on AO3. Furthermore, we have shown that there is not a strong correlation between the feedback rates on both sites. This indicates that the fic itself is only one factor determining popularity and audience engagement.
We caution that these results must be interpreted within the context of our assumptions and limitations - that is, we cannot say whether or not they apply to fics as a whole. Our sample is presumably biased towards authors who primarily use AO3, and thus may have a higher feedback rate as discussed in our initial data characterization. Additionally, we only looked at single-chapter works, and it is unclear if, or how, these results may apply to multichapter fics.
We do not intend to discourage any authors from posting on AO3, but instead investigate feedback rates with the goal of using this information to determine what may be responsible for the differences, and whether we can use our findings to propose changes to increase feedback rates on AO3. Furthermore, this only takes into account one form of feedback, and neglects others - most significantly, AO3’s kudos feature. In fact, if any conclusions about which platform to use can be drawn from these data, our suggestion is… both. If stories are permitted on both sites, as FFN does not allow explicit fics, we recommend that authors crosspost their works.
In our next section, we intend to discuss the potential causes for FFN’s higher feedback rate based on what we know about how readers interact with fics and commenting, why people choose to comment (or not to comment), and, in particular, what role we believe kudos might play. See you then!
~ dragonling and Rose <3
PS: if you see errors in our math or methodology, please point them out so we can correct them.
Shoutout to @helloamhere for guidance regarding statistical tests and data analysis!
[project overview]
[initial data characterization]
fnord888 replied to your post: I feel bad repeatedly picking on Scott, but...
I’m pretty sure the numbers in parentheses aren’t p-values (I think they’re effect sizes).
Oh, definitely. Because he starts out by saying “all results are significant at the p ≤ 0.001 level.” Which is what I was objecting to.
I suppose it’s possible that that is an exhaustive list of all variables he tested, but I doubt it. And even if they are, there’s all sorts of weirdnesses with covariance and correlation and multiple comparisons going on---if you want to look at all of those things you’d need to fit some sort of complex model, and I don’t know enough statistics to be able to tell you what sort of model you’d need.
But you definitely can’t run a bunch of comparisons and see which ones give you p ≤ 0.001.
People were happier with their decision to have children if they were (all results are binomial correlations significant at the p ≤ 0.001 level): more gender-conforming (0.14), had fewer thoughts about maybe being transgender (0.20), were more right-wing (0.10), considered themselves more moral people (0.15), were less autistic (0.12), were less extraverted (0.10), were more emotionally stable (0.15), and were more agreeable (0.13)
An interesting article from Accident Analysis and Prevention from 2004 goes over three case studies where Null Hypothesis Significance Testing may have cost lives.
Case 1: Right Turns on Red
Looking at the data in Table 1, persons without training in statistics would think that after RTOR was allowed, these intersections were somewhat less safe. However, the consultant concluded, quite correctly, that the change was not statistically significant. The Commissioner of the Virginia Department of Highways and Transportation sent the consultant’s report to the Governor and in the letter of transmittal wrote: “we can discern no significant hazard to motorists or pedestrians from implementation of the general permissive rule (i.e. of RTOR). No significant increase in traffic crashes has been noted following adoption of right-turn-on-red in any state including Virginia”. Obviously, there was miscommunication. In English ‘significant’ means ‘having or likely to have considerable influence or effect’; the synonym of ‘significant’ is ‘important’. In statistics ‘not’ significant’ means that the data is insufficient to reject the (null) hypothesis of ‘no effect’. Thus, the consultant said one thing and the Commissioner transmitted something entirely different.
... And so the sequence of small studies all pointing in the same direction but with statistically not significant results continued to accumulate, till that last study which I followed was published in 1983. While 287 crashes to right turning vehicles were expected, 313 were counted. The authors concluded, once again, that there was no significant difference in vehicular crashes.
...The problem is clear. Researchers obtain real data which, while noisy, time and again point in a certain direction. However, instead of saying: “here is my estimate of the safety effect, here is its precision, and this is how what I found relates to previous findings”, the data is processed by NHST, and the researcher says, correctly but pointlessly: “I cannot be sure that the safety effect is not zero”. Occasionally, the researcher adds, this time incorrectly and unjustifiably, a statement to the effect that: “since the result is not statistically significant, it is best to assume the safety effect to be zero”. In this manner, good data are drained of real content, the direction of empirical conclusions reversed, and ordinary human and scientific reasoning is turned on its head for the sake of a venerable ritual.
Case 2: Paved shoulders on rural roads
Once again common sense and statistical ritual point in opposite directions. The figures show that, e.g. after a two-foot paved shoulder has been added, the crash rate has declined for all crash types and all severities. Therefore, ordinary reasoning would lead to the conclusions that paving shoulders has reduced crashes. And yet, because of the paucity of the data, none of these reductions proved statistically significant. But quasi-science wins again; and so, in their Conclusion section the authors write:
The study could not discern any statistically significant differences in either crash rate or severity rate between two- and four-foot shoulder installations. Unless (other) benefits … are considered important to practitioners, this study does not show the increased construction cost of four-foot shoulders on state routes to be justified by an increase in traffic safety (p. 37).
Case 3: Speed Limit Increases
The two above cases could be seen as researchers failing to appropriately communicate their findings to lawmakers. In Case 3, we see researchers themselves misusing NHST to deadly effect:
Table 3. Predicted percentage increase in the number of fatal crashes attributed to the speed-limit increases on rural interstates (from Balkin and Ord, p. 10, Table 3)
State First % (1987) Second % (1995)
Alabama 0.0 24.8
Arizona 41.0 0.0
………
Missouri 13.0 42.2
Nebraska 35.5 0.0
………
West Virginia 46.2 0.0
Wisconsin 24.3 0.0
It is obvious that 0.0 is not the best estimate of the change in fatal crashes in all these instances. Why the authors decided to enter 0.0 can perhaps be understood from the numerical example by which they explain their method. In their paper there is a graph of the monthly time series of fatal crashes from 1975 to 1998 for rural interstates in Arizona and, referring to this graph, the authors say (p. 6) that:
“We see a significant increase in the level around 1987 but none around 1995. … Statistically it is estimated that the 1987 speed-limit increase resulted in a 41% increase in rural interstate crashes an Arizona. There is no statistical evidence that the 1995 speed-limit increase has any additional effect on the number of crashes.”
That is, failure to reject the null hypothesis of zero effect at the 10% level of significance was equated with the absence of statistical evidence for an increase in the expected number of crashes. In all these cases, 0.0 was entered in the table. Thus, the table contains two kinds of entries: either estimates of percentage change when the increase was statistically significant, or 0.0 by NHST convention but unsupported by either data or prior-knowledge when the increase was not statistically significant.
The article is behind a paywall. Feel free to message me for a copy of it.
In 1997, I compiled a list of articles, books, and book chapters that questioned the widespread use of null hypothesis significance tests (a.k.a. null hypothesis tests, significance tests) in scientific research. My goal was to provide those unfamiliar with this debate with a list of citations that pointed out the myriad of problems associated with the indiscriminate use of null hypothesis tests. For parity, I also compiled a list of references that supported, at least to a limited extent, the use of null hypothesis tests.
Ironically, null hypothesis testing as it is currently practiced is a hybridization of R. A. Fisher's significance test and J. Neyman and E. Pearson's null hypothesis test (hence the label 'null hypothesis significance test'). These two approaches were fundamentally different and were the source of heated debate between these two camps for many years (see Goodman 1993a for an excellent review of this historical debate). I sincerely doubt that the melding of these two approaches would have been acceptable to either Fisher or Neyman and Pearson.
