Advances in Spam Detection on Tumblr
As with all open platforms for user-generated content, Tumblr has been hit with a fair bit of spam. People who create spambots or abuse our platform in the interest of non-genuine social gestures are really good at finding new ways to develop and implement their spam. It’s what they do. Over the years, we have been experimenting with various tools and techniques to combat issues like spambots and non-genuine social gestures. To understand more about our work, let’s dig into the details.
Challenge: precise identification of spam
Spammers often try to disguise themselves by attempting to use a platform in the same way a real person would. As spammers learn how to develop newer, better ways of mimicking the behavior(s) of real people, the boundary between spammers and real people becomes more and more blurred, which unfortunately means non-spammers may get flagged as spam. This is what is known as a false positive.
Tumblr’s goal has always been to find the delicate balance needed in making sure we are addressing spam as aggressively as possible without dramatically increasing the number of false positives.
Challenge: evolving spam behavior
Spam evolves. Spammers learn how to dodge new spam detection as soon as a platform starts using it. Therefore, relying on fixed logic is not sufficient. We instead approach the issue with a broad set of dynamic predictors, because the best way to combat spam is to utilize an adaptable detection methodology.
At the heart of all good spam detection efforts are machine learning algorithms. These algorithms are fed data from how real people use Tumblr and use this data to enhance our classification accuracy. Thanks to this historical data, when new spam or malicious patterns start occurring, we can react faster and identify spam with higher accuracy. Our newly launched model demonstrates 98% accuracy in determining if a user is a spammer.
The diagram below describes our spam classification pipeline:
Because every machine learning algorithm starts with data, we begin with a data management system that manages and controls data streams flowing around Tumblr. Every microsecond, the data management system records this data into log files. The system then periodically transfers these logs into our database, or Hadoop File System (HDFS). We then write numerous scalding jobs that focus on identifying what parts of this mountain of data are helpful when learning who’s a spammer. To start this process, we come up with specific hypotheses on some data sources and then collect the data to test these hypotheses in the next step.
How do we test if a data source is useful? After the scalding jobs finish, we analyze and visualize the data source to determine if the collected information can be turned into a signal. If the raw data itself is not enough, we might need to combine several signals to produce better results. The whole thing may sound a little hard to grasp for some. Maybe a pseudo example could help?
What if we found that many spammers really enjoy, say, insects and they were creating posts with a massive number of insect images? Based on this observation, we would hypothesize that the more pictures of insects someone’s blog has, the more likely it is spam. If we validated this hypothesis, we would then build a feature called InsectImageNums to track how many times a blog has posted an insect. But wait! What if we realized that the majority of our users post zero insect images? This becomes problematic because most of the data in InsectImageNums are zeroes, and those that are not zero have a very diverse range. Besides, some insects specialists or nature lovers do post images of insects, and we don’t want to classify these people as spam incorrectly. We would need to dig deeper and find a more detailed differentiator. Perhaps we see that it is rare for even the most bug-loving person to post more than five pictures of insects. We’d use that finding and create a new predictor called InsectsImageNumsGreaterThanFive. After this transformed feature is verified as accurate and useful, it is included in our predictor set.
When we have a verified and helpful set of working features, we then pass them to the machine learning models in Spark through Hive. Sometimes the aggregated size of the data is way too big for a single machine to process, so we use Spark and Spark ML interface to train our larger data-sets.
What kinds of machine learning algorithms are we using?
Supervised machine learning requires training labels, but these labels are only partially defined. With imperfect labels, we use iterative semi-supervised machine learning techniques to label instances closest to the classification decision boundary by checking our predictions with human agents. When human agents stop seeing false positives, we assume the model is crafted strong enough to be placed into HDFS. Through this semi-supervised approach, we achieve a 98% accuracy rate. We then upload the trained machine learning model to our database and periodically update it.
We save the spam probability score of new groups of users daily on Redis, an in-memory data structure store. This user spam probability score becomes a useful data validation point for our internal team that leads our spam moderation effort. In a way, the machine learning spam detection pipeline’s job is not to automatically suspend suspicious blogs, but to find blogs that have suspicious behaviors—like spreading viruses or malicious content across the internet. We want our community to enjoy a friendly environment on Tumblr, and we want to avoid as many false positives as possible. That’s why our overall pipeline involves both machine and human efforts.
Spam detection work is never done. What works above may not work six months from now. Our goal is to evolve one step ahead of the spammers. Keep your eyes peeled here on @engineering to stay up-to-date!
— Vincent Guo (@dat-coder)
stormborns
theartofmadeline
ellievsbear
annaxmalina
baratheonpilled
pixelskylines
