My level of obsessive compulsive nerd is that I continue to work on my Arrow ratings math project despite that fact that I only “meh watched” six episodes this season. I should be working on my dissertation or my revise and resubmit pile in my spare time but this remains way more fun. Plus after my Season 4 Model worked so well I had to see if I could duplicate it for Season 5.
Being right remains my heroin. #CapricornProblems
Arrow Season Five: The Beginning of the End
First, let’s review the Arrow S5 average ratings & demo + adjusted ratings & demo change since S4 season.
Everyone knows that Arrow ratings AND CW ratings have declined sharply this season overall (http://www.vulture.com/2017/05/2016-2017-tv-season-in-five-really-depressing-charts.html). This article and the CW chart in here are helpful to center the conversation but I have critiques. First, no fair comparison of ratings decline over last season can include Supergirl. Comparing ratings from CBS to CW is not Apples to Oranges it is Apples to Volkswagens. Thus I removed Supergirl from my chart. Second, for change over time comparisons the number of viewers is mathematically preferable to the demo.
Math Segway: the demo is already a fraction so comparing percentages of fractions can lead to over/under predictability (Type One and Type Two Error) especially when your base fraction is an especially low number….like the demo from most CW shows.
The 2016-2017 CW ratings decline season average (thus far since some shows are still wrapping) is -14.7 with The Vampire Diaries and -17.6 without The Vampire Diaries.
The overall network decline of live viewers is steep. The shows hit the hardest are the limited run shows and the mid-season replacement shows (The 100, Reign, iZombie, Crazy Ex-Girlfriend) and Arrow. On the average, CW shows that started in the Fall only had a -13.6% decline in viewership over last season while the mid-season replacement shows declined -22%. While CW shows that had limited run seasons declined -18.7 while full run season shows declined -14.2.
Arrow & Crazy Ex-GF are CW’s only show returning for next season that are not within one standard deviation of that -13.6% Fall average. And Crazy Ex-GF is already a limited run show. Arrow is the CW’s only full run show to have their ratings fall this hard. Arrow’s steep rating decline is far closer to the decline for limited run shows. That ain’t good. At all!
The benchmark I have used for Arrow’s season performances is the % viewership decline or growth (the rate of change) for each season compared to the average of the last two long run CW drama (Supernatural & Vampire Diaries) at the same point in their run. This chart shows you how much of the audience they lost from the prior season.
Arrow viewership was on par or better at the same point in Seasons 2 & 3. For Season 4, Supernatural & Vampire Diaries averaged a 0% decline and Arrow was about -10% off that average of the viewership decline. For Season 5, Supernatural & Vampire Diaries averaged a -15.9% decline. If Arrow had followed that trend they would have averaged about 2.09M viewers for Season 5. Instead Arrow’s 1.75M viewer average was a -29.7% decline from Season 4 and an almost -14% off that anticipated average. The Arrow decline are getting substantially more noted than prior CW shows at the same point in their run. The Season 6 average decline for Supernatural & Vampire Diaries was -23.5%. If, and it is a big if, Arrow follows that pattern they can expect to hit somewhere in the range of 1.34M viewers next season.
I got into a Twitter spat this past week with a stan who insists this decline is a-okay and the CW totally expected it because the overall trend in declining live viewing. And yes! Fewer people are watching live now. But the DVR, iTunes, Netflix, the CW ap, and illegal downloads were not ALL invented this season. The Arrow S5 ratings decline is above and beyond this season’s network average and the anticipated decline of prior CW S5 shows. The Thursday night move to 9PM BEHIND a show that is entering a 13th season with a lower average rating and a lower average demo is a panic move. Moving the air date/time is a network shakeup. It is not as serious a network shakeup as reducing the episode order or rotating out showrunners but the writing on the wall seems pretty clear here.
I’m not trying to use math to tell you why people DO or DON’T watch or gage the quality of Arrow Season 5. That’s not how math works. I’m here to tell you that viewers tuned out in mass. And it did not go unnoticed by the CW brass.
Season Four to Season Five Viewer Analysis: The Purge
Since S5 of Arrow had such a notable decline in viewership I was curious if I could parse out where the trigger point for viewer departure occurred. In my academic field (higher education) we have a concept called “summer melt” which is the number of high school graduates who report they plan to go to college next year but don’t actually enroll in the fall. For a host of reasons, these students “melt” before enrolling in college. I was curious if I could parse out S5 Arrow’s viewership declines, so I started looking for “summer melt.”
I averaged the final 6 episodes of Seasons 3 and Season 4 and compared it to an average of the first 6 episodes of Season 4 and Season 5, respectively to calculate Arrow’s summer melt. Comparatively, Season 4 lost -9.7% of the audience from S3 and only 1.1% of that viewership loss was from “summer melt.” The bulk of viewers who stopped watching Arrow from S3 to S4 did so DURING Season 4.
The hiatus before S5 Arrow lost 14.4% of viewers. That is 310K viewers who stopped watching Arrow before S5 even started. Their viewership loss will be attributed in the S5 -29.7% viewership loss but they did not even return for S5, so truly their departure should be attributed to S4. In fact, those 310K “summer melt” viewers represent an estimated 45% of ALL lost Arrow S5 viewers with 55% of viewership loss coming during S5.
From that beginning of S4 (first six episode average) and the end of S5 (last six episode average) Arrow lost about one million viewers. I made this chart to demonstrate which point those one million viewers left the show.
Again, there is no trying to parse quality or tell you what about Season 4 or Season 5 drove viewers away here. That is for viewers, critics, and the CW to speculate over. Math can’t help you there.
Arrow S5 Predictor Model: Worked Again!
I noted this in the Fall (http://rillacuthbert.tumblr.com/post/151450182766/arrow-s5-ratings-models) but to recap, I built two S5 models to test which is preferred for aggregate rate of change ratings prediction. The 5x01 rating was entered in to the models to generate the predictions for the subsequent episodes and season average for both models. The Oliver Model is based on ratings rate of change for S1-S4 and The Felicity Model is based on the ratings rate of change for S1-S4 to episode 7 and then S3-S4 from episode 8-23. The hypothesis being tested here is if the in season, post-crossover ratings bounce is sustainable and predictable.
The final result is that The Oliver Model is statistically significant predictor of Arrow S5 ratings and The Felicity Model is not. The takeaways?
The value of the post-crossover ratings bounce has limited impact on the season average rating thus The Oliver model is superior to The Felicity Model
That for two consecutive seasons a statistical model correctly predicted Arrow’s average rating
That suggest my original hypothesis about Arrow viewing remains true: Within a single season, viewership rises and falls across a predictable trend based on prior viewership in part because TV watching is cyclical and seasonal based on the counterfactual of what else is airing.
You can see this in the table below that shows the highest and lowest Arrow ratings by episode number average. I did separate averages one for the full five seasons and one for S3-S5 (again to measure crossover impact). I bolded the episodes that appear in both average.
The highest rated Arrow episodes each season tend to be the premiere and the episodes around the crossover. The lowest rated Arrow episodes each season are the episodes in the final third of the season. If you plot the episode averages for the full five seasons and for S5-S3 you can see clear viewership patterns emerge.
This again suggest that viewership is cyclical over the course of the season and rarely driven by the plot of an individual episode. Beyond the crossover, almost never can you say a plot point increased or decreased Arrows ratings. These viewership trends are predictable and observable. Viewers are far less likely to watch episodes 19 or 22 live across every season regardless of what plot or storyline is in that episode.
IF SOMEONE TELLS YOU THAT A SHIP OR A GUEST STAR OR A COMIC BOOK CHARACTER OR THE TIDES OF THE MOON ARE WHY AN EPISODE DID OR DID NOT GET A GOOD RATING THEY ARE MAKING SHIT UP.
They will continue to do so. And it will continue to annoy the piss out of me.
I have another ratings post on the back burner about applying the Arrow ratings model to other CW shows and networks, what I found out, and what it means for modeling Arrow Season 6 but that is for another day.
Arrow Season 4 Predictor Model: Gloating With Math
Fandom factions are always looking to claim an advantage. Folks want to present “evidence,” however arbitrary, that their opinion is right or good or popular. I’m as guilty of the need to be right as anyone else. Hell, I did a season long math project because being right is my heroin (#CapricornProblems). TV ratings are one place where this plays out with people using ratings as “proof” of their opinions.
I started Arrow Season 4 with what by fandom standard is a pretty controversial hypothesis. I didn’t think plot or storyline or characters or even ships drive weekly ratings. I believe trying to interpret week to week viewership ups and downs as a metric of what is and is not popular about Arrow is a fool’s errand with no basis in facts. You can speculate about why ratings rise or fall week to week, but there is no qualitative evidence to say why people do or not don’t watch.
My hypothesis was that Arrow has established a steady core audience and that ratings could be predicted based on prior Arrow and CW viewership trends. To test my hypothesis, I designed 2 predictor models based on prior Arrow season viewership trends. The full statistical methodology can be found in this link (http://rillacuthbert.tumblr.com/post/134520685781/arrow-ratings-predictor-models-s4).
These models don’t factor in the relative popularity or dislike for any plot or ship. They don’t factor in fandom frustrations with the death of Laurel Lance, the baby mama drama, the Olicity breakup, the use of magic, Malcolm Merlyn (STILL!) being alive, or even Felicity’s grommet coat. The models are ONLY based on prior aggregated viewership trends. My theory was that viewership rises and falls across the season in a steady and predictable trends that tell us very little about the popularity of any specific storylines.
Did my hypothesis hold up?
Model One (far and away the more reliable model) predicted Season 4 would average 2.51M viewers and Season 4 actually averaged 2.49M viewers.
If you plot Predictor Model One side by side with actual Season 4 Ratings one can see that outside a few divergences here and there (shakes fist at the crossover) the actual ratings generally follow the same trajectory as the Model One.
The actual viewership only deviated from the predicted viewership by about 20,000 viewers for the season average. My model accurately predicted (99.1%) of overall Season 4 Arrow viewership without any variables or controls for plot or storyline. No self-respecting data analyst would say my very small sample size (only 23 observations or in our case, episodes) is definitive proof of anything. However, it provides actual data-not air quotes evidence but actual statistical evidence-in support of my hypothesis that Arrow’s audience is stable and predictable across the season.
Relative to Season 3, Season 4 declined 9.7% in viewers and 10.9% in the demo.
The viewership decline is about on par with other CW shows from Season 3 to Season 4, if a little bit lower.
If you compare Arrow to the other fourth season network dramas you can see that Arrow is doing far better at holding audience that other 4th year shows.
I know some folks are pressed over this decline. Season 3’s increase in ratings appears to be a statistically anomaly in the history of the CW network. If you were expecting S4 rating trends to continue upon that same upward trajectory for this season or beyond you are likely going to be disappointed. CW TV shows tend to decline in ratings over time. Arrow has done an average to above average job holding live audience viewers relative to other CW shows and other network shows at the same point in time.
The more pronounced decline in viewership during the back third of the season raised an alarm for some folks. However, not one of those 8 episodes (4x16-4x23) was a statistical outlier. Each episode’s ratings fell with in the predicted 95% confidence interval range for that week. So while the weekly rating may be lower than anticipated they are all within a normal, predictable range. Additionally, this same downward trend in the final stretch of the season occurred in both Season 1 & 2. (http://rillacuthbert.tumblr.com/post/142279543656/since-youre-a-numbers-person-i-thought-id)
A CW Season is best considered as 3 mini seasons that are divided into 3 parts: A-beginning (October to November including the season premiere & November Sweeps), B-mid-season (December through February including the crossover (S3 & S4 only), midseason finale, & the January premiere), and C-back third (March to May, the post February/March hiatus episodes including May sweeps & the Season Finale). We can compare the rate of change from the viewership averages for beginning (A) and back third (C) to evaluate how well each season was able to retain viewers.
S4’s back third of episodes hold beginning of the season viewers on par with S2, better than S1, & worse than S3.
I’m not sure how happy it makes people to read an analysis with phrases like “on par” or “predictable decline” but that is how CW ratings work. There is no magic bullet math that will tell you Arrow’s ratings are higher in S4 than in S3. In fact, this will be the lowest rated season of Arrow. Until Season 5. If you are looking to explain the WHY of Season 4 ratings declines? That can not be done with any legitimate quantitative analysis of any rating data. If someone says Arrow ratings declined because Laurel died OR there was too much Olicity OR not enough Olicity they are speculating without any evidence. That is data analyst speak for “they are full of shit.” Arrow ratings decline from Season 3 to Season 4 and over the course of Season 4 was 99% predicted by a statistical model based on CW & Arrow viewership rates of change. Arrow ratings declined over time because that is what CW ratings do. Full stop.
A final note for next season: I already detailed how Season 5 has historically been a point of high viewership drop-off with CW shows averaging about a 16% decline from Season 4 to Season 5 (http://rillacuthbert.tumblr.com/post/141589212111/arrow-season-five-ratings-projections-and). If Arrow follow this trend the anticipated viewership for 5x01 would be about 2.25M (down 16% from 4x01 2.67M). Arrow could lose 16% of its audience next season and that would be both normal and expected. You gotta do you, but I won’t be spending my hiatus stressing about Arrow ratings and neither will the CW.
If you are looking for my full Arrow Season 4 Ratings Predictor Model you can find it here: http://rillacuthbert.tumblr.com/post/142335679056/s4-predictor-model. Again, I tested two models (Model One and Model Two) with Model One being the far superior model.
For tonight’s episode (4x21) Model One predicts a 2.29 with a 68% chance the rating will fall between 2.47 and 2.11 and a 95% chance the rating will fall between 2.66 and 1.92.
My gut says it will probably fall within the lower range (between a 2.29-1.92) because of the previous discussed final third of the season decline http://rillacuthbert.tumblr.com/post/142279543656/since-youre-a-numbers-person-i-thought-id
This is a too long statistical analysis of Arrow ratings that no normal person would want to read. Don’t say I didn’t warn you.
October = Playoff Baseball & Fandom Idiots
I half wrote a long rant the other day about effective data analysis for ratings and ratings predictions. However, I had to use some slightly advanced statistical analysis and I struggled how to write up complex math results in a way that everybody could understand. I didn’t want to write an unreadable social science academic journal article that only @emmykentmg would read while everyone else nodded politely. But I’m going to take a whack at it because the stans and fanboys are in rare form.
Today Guggenheim jumped onto Twitter to smack down some #NOlicity idiot with actual ratings logic (aka the MLB playoffs result in a decline in October ratings). I wanted to hug him because of how correct this is.
This supports my pet ratings theory from my prior ratings posts that Arrow viewership is cyclical (probably many CW shows too). I’m currently building an Arrow rating predictor model to demonstrate how these viewership trends reoccur year to year. Ratings rise and fall at set period in the season. They rise for sweeps and crossovers and midseason finales. They fall with month-long sporting events like the World Series and March Madness, as well as spring NBA & NHL playoffs. These trends suggest that Arrow viewership is actually pretty consistent with normal fluctuations over time.
These tweets are symbolic of a larger problem about ratings. Fans want to use ratings to prove that characters or plots or ships they like are better/more popular than those they dislike.
That is not how ratings work. Ratings only tell you if people watch, not why. I will say this until I’m blue in the face but folks on both side of the fandom ship war are gonna continue to throw around ratings data they don’t understand how to interpret like chimps at the zoo throwing their own feces.
Exhibit A
Clearly she wears a tinfoil hats so it should not be worth my time, but my know-it-all Capricorn soul can not allow this level of stupid to go unanswered.
She says that a 100K (4% decline) drop in viewer from 4x02 to 4x03 is a) major concern (it really isn’t), b) that is clearly the fault of Olicity (bullshit, she has no evidence for why people do or don’t watch), and c) that the show should change the storyline to win back 100K viewers (not gonna happen).
These folks love to reference S1 ratings as “proof” of the allegedly popularity of their ship or fav.
So let’s look at the corresponding October episodes for S1.
That’s a viewer loss of 590K, 40K, & 460K a week!! *insert sarcasm* OMG!!! The sky is falling. Arrow should have been cancelled in S1. *end sarcasm* Raw totals like the change in numbers of viewers week to week are useless for data analysis without comparative context.
October Ratings Statistics
Is playoff baseball an excuse to hide Guggenheim and Arrow’s evil Olicity agenda or do Arrow rating really decline each October with playoff baseball? I could take the unsubstantiated word of stans on social media, but instead I decided to do an actual statistical analysis. Why? Because being right is my drug of choice.
First, we will compare the rate of change for October Arrow episodes. Since our unit of interest is a decline in viewers we can’t include the season premiere (as there is nothing to compare it against). I averaged the rate of change percentage in viewership for all other Arrow episodes that have aired in the month of October (S1-S3 episodes 2-4 & S4 episodes 2 &3) to create a comparison table.
On the average, October Arrow episodes decline about -5.5% over the previous week’s ratings. Season by season you can see that the greatest loss in viewers was in Season 1 (9.4%) and with the least in Season 3 (3.4%). One can infer that the Season 1 October ratings took the biggest hit not only because of playoff baseball, but because as a new show it was establishing an audience. Or I could make myself a tinfoil hat and rant about how it means ‘real Arrow fans’ hated the Oliver/Laurel/Tommy love triangle of boredom.
All statistical analysis is driven by averages (the mean). There are mathematical equations that you do (or that you run in a statistics software package like Stata or SPSS because it is the 21st Century, damn it!) to determine if there are real “statistically significant” difference between groups of numbers. And this is where statistics gets a little complicated (aka math-y).
“Statistical significance” is the 95% probability that the difference between one number vs. another number is a real mathematical difference rather than just a fluke. In this case, the test of statistical significance will tell us if the difference between an episode’s increase or decrease in viewers over the prior week (the rate of change) is real or random when tested against Arrow’s average October episode viewership decline.
Math Note: Most people are familiar with statistical significance and 95% confidence intervals, even if they aren’t aware of it, from political polling. It is the math behind “margin of error.” A poll might say Candidate X has 49% of the vote and Candidate Y is has 46% of the vote with a 4% margin of error. That means there is a 95% probability that the election results will fall within a + or - 4% range of the poll. Candidate Y could be leading with 50% of the vote. In this example, the margin of error is large enough that the election results would be too close to call because there is no real “statistical significant” majority for either candidate. The mathematical difference between Candidate X and Candidate Y is random and not real.
Back to Arrow ratings. Using a statistical test (a one sample mean t-test, if you care) you can predict with 95% confidence that individual October Arrow episode change in weekly viewership will fall somewhere between a decline of 12.3% to an increase to 1.3%. As noted above, Arrow’s average October viewership rate of change is -5.5%.
This tells us the “margin of error” for viewership is 6.8% +/- from the average. If Arrow’s October viewership rate of change falls between -12.3% and +1.3% they are statistically random and “not significant.” If an October Arrow episode increase viewership 1.4% or better or a loss of -12.4% or worse it would be significant. However, you don’t know WHY it is significant. Could it be Arrow plot related? Sure, but Arrow doesn’t exist in a vacuum. Other events are happening in life and on TV.
Episode 1x04 (-13.1%): How much of the decline is attributed to disinterest in the show and how much is attributed to the episode airing on Halloween when parents have to take their kids trick or treating and the rest of have to go out and get drunk. The 2012 World Series ended the Sunday prior to 1x04, so there is not baseball related decline in viewership.
Episodes 2x03 (-5.5%) and 2x04 (-18%): How much of the decline is attributed to disinterest in the show and how much is attributed to the episode airing on the same nights as Game 1 and the deciding Game 6 of the World Series. And not just any World Series, but a Red Sox World Series. Which means that no television in a top 10 US media market was tuned to the CW.
Episode 3x02 (-18%): How much of the decline is attributed to disinterest in the show and how much is attributed to the episode airing on the same night as TWO League Championship Series? That means not only general nationwide viewership, but four cities this time with two top 10 media markets (San Francisco & Washington DC/Baltimore) who were kinda busy that night.
We can see that 4x02 declined -6.3% and 4x03 declined -4% in viewership over the prior weeks. Statistically, we can say with 95% confidence that those variations in viewership are random and not mathematically real. And even if they were statistically significant (they are not), that tell us jack about the why of watching. Saying less people watched Arrow this October because they hate Olicity is just as valid a theory as saying less people watched because of seasonal Pumpkin Spice induced insanity. In that they are both theories supported by zero evidence. World Series playoff baseball suppresses ratings. It is probably why “television sweeps” are in November and not October.
October and November Ratings Statistics Comparison
Do Arrow ratings go back up in November? Again, I averaged the rate of change percentage in viewership for all Arrow’s November episodes for seasons. On the average, November Arrow episodes increase about 8.6% over the previous week’s ratings. N.B. If you exclude last year’s Flarrow ratings boom November’s average ratings still increase 5% per episode.
A different statistical test (a two sample mean t-test if you care, p.s. you still don’t care) can predict with 95% probability if November’s average viewership increase is a “statistically significant “ increase over October’s average viewership decline or if the monthly difference in ratings is mathematically random. I ran two tests-one that includes Flarrow in the November average and one that does not.
Statistical significance is predicted with a p-value under 0.05 (p<.05). The November increase in ratings with Flarrow has a p-value=0.028 and the November increase in ratings without Flarrow has a p-value=0.042. The results of both test predict with 95% confidence that November’s ratings increase over October is “statistically significant” and not mathematically random. Using the political polling example, the margin of error between October’s viewer decrease and November’s viewer increase is small enough that we can “call the election” in November’s favor to use the metaphor.
Summary
Using mathematical evidence, one can see that EVERY YEAR playoff baseball cuts into Arrow’s October viewership and that EVERY YEAR Arrow’s viewership rebound again in November. When Marc Guggenheim tells you this is so, it is not because he is a closet shipper who won’t face the truth that Olicity is destroying Arrow and probably society as we know it. It is because the man works in professional television for a living and has probably had to sit in so many network ratings meetings that the very topic now makes his eyeballs bleed.
I am a crazy Olicity shipper, but I am also first, last, and always a wonky data analyst. This is math and logic. And math x logic are my real OTP.
Coming This Week: Ratings Armageddon
Game Two of the World Series is this Wednesday night (And if it goes to Game Seven that is scheduled for the following Wednesday). There is a New York team in the World Series. That means in the #1 media market in the country far fewer Nielsen households will watch Arrow this week. On the average, Arrow ratings decline about 4% on nights when a World Series game airs. But on night’s with big playoff baseball games featuring major market teams that have huge nationwide fan bases, Arrow has seen a decline in ratings as high as 18%. Brace yourself for a new round of ratings stupid this coming Thursday, Olicity family.
Ratings takeaways for the week:
Arrow is most likely gonna get walloped in the ratings this week (and maybe next) because of a Mets World Series
Since fools never realize they are fools, stans will still blame Olicity for the decline
Ratings will rebound with November sweeps
Then these same fools will try to credit the increase to their fav thus further proving they are fools.
Per usual, I collect raw ratings data from tvseriesfinale.com and calculate percentage changes and statistical tests myself. Wikipedia, CW press releases, and many fan sites are littered with bogus data and mathematical errors. If you message me to say that my analysis is wrong ‘cause you found something different on the Arrow Wiki may you be haunted by all your high school teachers who should have taught what is and what is not a reliable source.
