Are snitch catches just a coin toss?
The snitch is the perhaps most iconic aspect of quidditch, when played either by wizards or college-aged nerds. When friends and coworkers ask about the non-magical interpretation of the game, they invariably focus in on how the snitch works, and are often both amused and perplexed by the image of a player running around with a sock stuffed down their shorts. For many, this is where conversation ends, but those who remain interested (and are somewhat familiar with the Harry Potter series) often present a second question, which I find far more interesting:
“Is it still worth 150 points?”
For those of us that have played organized sports throughout our lives, the snitch presents an odd way of determining the end of a match. Most sports have games that are limited by time, often divided into periods, halves, or quarters, while a few others finish when a team or individual reaches a certain score. In quidditch, games do not end at a specific time or score, but rather, through an event—the catching of a snitch by a seeker—that both ends the game and gives one team a large number of points.
Those who ask about the point value of a snitch recognize that if an individual player can have such a tremendous impact on the outcome of a game through a single event, a team sport isn’t particularly competitive. While the snitch serves as a great plot device to grow the hero of Harry Potter, if the same rules applied to muggle quidditch, all members of a team aside from the seeker would be understandably put off by the relative unimportance of their role on the pitch. As a result, the International Quidditch Association rulebook has gradually evolved to reduce the importance of seeker play through a reduction in snitch point value (from 150 to 30) and installation of a minimum game time before the snitch can be caught.
Nonetheless, the role of the snitch remains among the most contentious features of quidditch as it’s played in the muggle world. In 5 years of participating on teams in the Midwest and West Coast, I’ve yet to experience a tournament where snitch catches go largely undisputed—arguments over the validity of catches have become so commonplace at this point that I expect captains to talk to the referees in almost every game. This is not without reason. Given the tension that builds at the end of a quidditch match, I don’t find it surprising that discussion over the validity of snitch catches break out, especially in competitive games. The snitch is a large part of what makes the endgame of quidditch exciting, as teams have to balance their attention to both catching the snitch and preventing the other team from racking up a lead big enough to render the snitch catch irrelevant to the game outcome.
However, the snitch catch can also generate intense frustration, much to the game's detriment. Many games end after a series of potential snitch grabs are ruled unsuccessful as a result of highly subjective “illegal” play, or worse, occur when a snitch is distracted and fails to notice an opposing seeker. Over time, it’s struck me that the result of snitch play between competitive teams often seems to be quite random. This isn’t to say that some seekers aren’t better athletes than others, but instead that the relationship between seeker ability and successful catches is fairly weak. Maybe it’s a result of a rulebook that isn’t very clear on the rules of seeking, or the nature of trying to grab a sock stuck to a runner’s shorts without knocking them down, but the process of determining a winner in a quidditch game seems much more variable than I would like in a sport that is increasingly focused on competition and the crowning of regional, national, and international champions.
To better understand the impact snitches have on the game of quidditch, I decided to start by asking a simple question:
How different are the actual win percentages of quidditch teams over the course of a season when compared to their win percentage if snitch catches were assigned at random?
If we find that a large number of teams are over/underperforming their expected win percentage under a scenario where the game ends with a coin toss deciding a snitch catch, that might be evidence for teams being able to substantially change the outcome of games by having an effective seeker. To answer this, I made a fairly straightforward framework for estimating how likely a team will win a game based on point differential at the game’s conclusion, assuming snitch catches are determined randomly.
How I determined expected win percentage assuming random snitch catches
(You can skip to the next section if you think statistics are boring)
Expected win percentages of individual games under a random snitch catch scenario were determined on the basis of quaffle point differential at the point at which the regulation time snitch catch occurs:
If the snitch is caught when quaffle differential is greater than 30, the team in the lead can expect to win 100% of the time, as the snitch cannot decide the winner.
If the snitch is caught when quaffle differential is less than or equal to 20, each team can expect to win 50% of the time, as a snitch catch for either team will determine the winner.
If the snitch is caught when quaffle differential is equal to thirty, the team in the lead can expect to win 75% of the time. In this case, there is a 50% chance the leading team will win with a snitch catch and a 50% chance the trailing team will catch the snitch and send the game to overtime. To simplify things, I assumed each team had an equal chance of winning in overtime, so the leading team has a 50% to win without OT, a 50% chance to play OT, and a 50% chance to win that OT (50% + (50% x 50%) = 75%).
This model isn’t perfect, as it doesn’t factor in game duration or any of the nuance that occurs at the end of a game when teams change their strategies to accommodate scoring the quaffle while trying to catching the snitch. However, it does form a simple basis that we can compare team performance against. Here’s how it looks in practice:
Let’s say we have a fictional team, named the Society of Quidditch at the University In Bermuda Southeastern (Squibs for short), who have played 6 games in the 2015-2016 season. The Squibs are a new team and have had mixed success:
The Squibs played 6 games and won 3, so their season win percentage is .5 (50%)
Their quaffle point differential (QPD) in these games varied from -100 to +50. This was input into in the model above to calculate their expected win percentage in each game if snitch catches happened at random.
To compare their performance relative to this random catch scenario, total expected win percentage (.25 + .5 + .5 + 1 + .75 = 3) was compared to recorded win percentage (3).
In this case, the Squibs had as many wins as they would have expected to receive if snitches were caught at random.
By performing this same comparison for all of a team’s games over the course of a season, we can evaluate how much better (or worse) a team is performing than they would have expected if snitches were caught at random.
In this analysis, I used game data from all official games played in the 2015-2016 quidditch season. Unofficial games, or those that were not recorded by the US Quidditch Association were not a part of this dataset.
This graph is interactive! Mouse over each point to see data on individual teams. Use the buttons on the top of the window to change view.
Displayed on the horizontal axis of this graph is the expected winning percentage of a team based on difference in quaffle score at the end of their games, assuming random snitch grabs. On the vertical axis, the actual win percentage of each team is recorded. The black 1:1 line in the middle denotes where expected and actual win percentages are equal. Teams that are above this line are winning more games than they would expect if snitch catches occurred at random, while teams below the line lose more games. The distance of each point from the line denotes the magnitude of this over/underperformance. Points represent individual teams, which are colored by region and grow or shrink in size depending on the number of games that team has played in the 2015-2016 season.
What’s initially striking about this graph is that there is a very clear relationship between a team’s actual and expected win percentage. While these percentages don’t always line up perfectly, teams tend not to deviate much from that center line—there are no teams that win an additional 30% or more games than we would expect in a random snitch grab scenario. In fact, this model predicts a team’s win percentage over a season just based on their quaffle point differential with 94% accuracy. Most of the time, teams have a season-long win percentage that is about the same as if snitch catches were just decided with a coin flip.
There are some notable exceptions, however. For example, teams from Arizona State University and the New York Quidditch Club won 12-13% more games last season than this model expected. Conversely, the University of Rochester Thestrals and Moscow Manticores lost nearly 20% more of their games. This isn’t to say that these teams have good or bad seekers necessarily, just that in games where the quaffle score was close, the snitch was caught at different rates than would be expected at random. These same seekers might have had wildly different catch rates in all other games, but perform better/worse when the snitch catch matters (think of this as being related to the Snitch When It Matters (SWIM) statistic that USQuidditch reports on its standings page).
The story is also a bit more complicated than that, so before anyone might accuse me of disparaging their team, I urge you to please read on.
Looking more closely at this figure, we also see an inflation in the cloud of points centered at roughly 50% win percentage. This is interesting, as it suggests that teams with a middling win percentage seem to be influenced more by their snitch catch rates than teams with high or low win percentages.
There are two factors that play into how teams over/underperform this random expectation. First, teams could be catching a greater or lesser proportion of snitches in close games than would be expected at random. If a team plays three games where the end game score was tied, yet wins all three, this team has won 1.5 more times than they would have expected.
Winning a few more games relative to expectation over the course of a season doesn’t necessarily translate to big shifts in win percentage, however, as teams that play a small proportion of close games don’t have many opportunities where the snitch game comes into play. A dominant team with an expected 95% win percentage, for example, may have a great seeking game but can’t deviate much from the expected model because they are almost always beating their opponents by more than 30 points before the snitch is caught. In this case, even if they always caught the snitch in close games, perhaps only 5% of their games had snitch catches that determined the winner, so their seeker's success could not produce a big change in their overall win percentage.
By considering this potential interaction between a team's win percentage and the number of close games they play, we might be able to explain why there is so much variation in the middle of our graph. Judging from my experience over the past few years, I think we can assume that the distribution of strength between teams is bell-shaped (normally distributed). There are likely to be a few really strong teams, a few really weak teams, and a large number of teams that are close to average. Because teams are playing opponents from all parts of this spectrum over the course of a season, we can expect that high win percentage teams only play against teams of their same quality a few times, as do low win percentage teams, but average teams play a lot of matches against other average teams.
In general, average teams are more likely to encounter opponents of roughly the same strength, producing close games in which the snitch determines the winner. It is for this reason that we might see so much deviation from the random snitch catch expectation among teams with a ~50% win percentage, as their snitch catch success decides a greater proportion of their wins and losses. To put it simply, teams with average quaffle play are likely to benefit the most over the course of a season from having a good seeker. High win percentage teams can benefit as well, but an above-average snitch catch percentage isn’t as likely to translate into a large number of wins over the course of a season. Instead, it may well produce tournament titles, because in later rounds of regional and national championships, good teams are often squaring off against teams with similar ability. If good teams are only playing close games in tournaments, I don’t think it should come as much of a surprise that 2016 US Champions Q.C. Boston still managed to outperform their season-long win expectation by 5% despite their dominant overall record.
So, for current quidditch players, feel free to look over the graph and check out how your team performed this last year—maybe it will spark some discussion on ways to improve for the coming season. But before interpreting this graph and making assumptions on the quality of a team’s seekers, keep in mind that it’s hard to make conclusions without data on a team’s frequency of competitive games played and their performance therein.
We’ll continue this series in our next blog post by focusing in on the two factors that make up a team’s seeker effectiveness:
How well are each team’s seekers performing?
How many close games do teams play?
