Rap Data "Analysis" from FiveThirtyEight
At this point, youâve probably already heard that the new FIveThirtyEight has some problems. Â It seems a little unsporting to pile on, but if Nate Silver is gonna be a dick, Iâd like to make a nomination for FiveThirtyEightâs worst piece yet.
The piece, a blog post by Mona Chalabi, is titled âThe Fastest Rapper In the Gameâ and attempts to identify that rapper. Â Although more light-hearted than whoâs going to win the midterm elections or the NCAA tournament, itâs an empirical question where good data analysis would provide a satisfying answer. Â Sadly, despite coming to the reasonable conclusion that Twista raps faster than other rappers, the analysis is an utter disaster, making serious errors at nearly every step of the analysis process.
Before diving into her analysis, Chalabi shares a chart from Rap Geniusâs Rap Stats (Rap Geniusâs equivalent of Google N-Grams), comparing usage of the words âgirlsâ and âmoneyâ over time, then makes the following critique:
âBut here at FiveThirtyEight, itâs our job to spot axis labels. Youâll see that while the trends above may seem big, they actually represent less than 0.1 percent of all raps. That doesnât bode well for Rap Genius research on artistsâ rap speed.â
The reason the the axis stops at 0.1 percent is there are a lot of words in the English language, and even very common nouns like âgirlsâ and âmoneyâ are going to amount to a tiny percentage of the total words used in all songs in Rap Geniusâs database. Â FiveThirtyEight is blaming Rap Genius for the fact that rappers donât have tiny vocabularies.
And what is this âRap Genius research on artistsâ rap speedâ that FiveThirtyEight is so convinced is wrong? Â Hereâs the next sentence:
âFiveThirtyEight is keen to develop accurate rap research, so we tested the 10 fastest rappers according to Rap Genius to see whether their rankings were justified.â
If you click that link, youâll get a post on a Rap Genius forum by a user with the account name DeLillo. Â Itâs a ranking of ten rappers, that user DeLillo makes clear is merely his opinion. Â To clarify, what FiveThirtyEight describes as research by Rap Genius, turns out to be some dudeâs opinion on Rap Geniusâs public forums.
Keep in mind we havenât even begun the actual analysis. Â FiveThirtyEight takes DeLilloâs list of ten fast rappers and claims to look at each rapperâs most popular song of Rap Genius. Â They then go on to count the number of words in the song, and divide it by the songâs length. Â Chalabi briefly mentions one of the major issues with this methodology: that there are parts of songs in which a rapper isnât rapping. Â This flaw is probably fatal, since most rap songs feature more than one rapper, a chorus, and lyric-free seconds of music, and the percentage of a song that features actual rapping by the lead artist probably doesnât correlate with the speed at which said rapper raps (if anything, youâd expect faster rappers to get through their parts more quickly)
But Chalabi skirts over the biggest methodological problem: that determining the fastest rapper based on single song is akin to trying to pick the best player in the NBA by only looking at yesterdayâs box scores. Â And since the metric she chose is such a poor approximation of speech rate, Chalabi isnât just restricting her scope to yesterdayâs box scores, sheâs only looking at rebounds.
After baselessly criticizing Rap Genius (a site, it should be said, that is incredibly easy to criticize on merit), and presenting a completely useless methodology, the piece even fails to execute one its stated methodology. Â Chalabi claims to have picked the most popular song per Rap Genius for each artist, âexcluding ones in which the artists were merely featuredâ. Â Chalabi doesnât mention what song she uses for each artist on her list, other than for her winner, Twista, for whom she used âMista Tung Twistaâ, but if you look at the most popular songs for Twista on Rap Genius, âMista Tung Twistaâ is nowhere to be seen. Â According to Rap Genius, the most popular Twista song that he wasn't merely featured on was âSlow Jamsâ which you probably think of as a Kanye song, but is also on Twistaâs album Kamikaze.
There are more mistakes. Chalabi cites Rap Genius as her only source, but Rap Genius doesnât appear to include song lengths on its website, so those numbers seem to have come from somewhere else.  She also incorrectly labels Joey Bada$$ as an amateur rapper. It ends with two lines of rapping that are just as successful as the rest of the post.
Thank god FiveThirtyEightâs job is merely âto spot axis labelsâ.
