Sean J. Taylor

social scientist and statistician

[seanjtaylor]

Characterizing Online Public Discussions through Patterns of Participant Interactions

New Paper at CSCW 2018 with Justine Zhang, Cristian Danescu-Niculescu-Mizil, and Christy Sauper [link to paper]

Facebook has an incredible number of public (between non-friends) discussions that can grow quite large (# participants and # comments). A hard problem we face is understanding which discussions are going well for users. Imagine doing that for dozens of languages and use cases.

The topic of this paper is a natural extension of my work with George Berry where we study whether seeing higher quality comments causes people to write higher quality comments (it does!). We had a very limited definition of "quality" in that paper that allowed us to focus on the social influence component.

The key idea in this new paper is to encode a discussion with multiple participants as a hypergraph. This is a generalization of earlier reply-tree and sequence models of discussions. We actually ignore text entirely and focus on interactions like reactions/replies.

From this hypergraph, we can extract a number of features and create a high-dimensional vector of discussion characteristics. An important idea that we employ is to count subgraph motifs that stylized types of interactions people can have in discussions.

This high-dimensional representation of discussion structure can be easily embedded in low dimensions. What's wild is that we can aggregate to the page level, finding certain types of pages are clearly associated with certain discussion structures. (No text/topics are used here!)

Our discussion embeddings allow us to do some neat tricks, like characterizing whether a conversation "style" emerges from the topic itself or its initiator (the latter!). The discussion vectors are also excellent input features for classification tasks, like predicting blocks.

The takeaway here is that the patterns of interaction in comment threads provides incredibly useful signal about what kind of discussion is happening, both in a descriptive and predictive sense. Some dimensions also have clear human interpretations, which we discuss in the paper.

Our hypergraph representation of online public discussions is flexible, extensible, complementary to text representations, and (we think) is more agnostic to language/culture. We also provide code that allows you to apply this method to Reddit data.

Abstract

Public discussions on social media platforms are an intrinsic part of online information consumption. Characterizing the diverse range of discussions that can arise is crucial for these platforms, as they may seek to organize and curate them. This paper introduces a computational framework to characterize public discussions, relying on a representation that captures a broad set of social patterns which emerge from the interactions between interlocutors, comments and audience reactions.We apply our framework to study public discussions on Facebook at two complementary scales. First, we use it to predict the eventual trajectory of individual discussions, anticipating future antisocial actions (such as participants blocking each other) and forecasting a discussion’s growth. Second, we systematically analyze the variation of discussions across thousands of Facebook sub-communities, revealing subtle differences (and unexpected similarities) in how people interact when discussing online content. We further show that this variation is driven more by participant tendencies than by the content triggering these discussions.

[seanjtaylor]

Randomized Experiments on Networks

My friend Dean Eckles and I wrote a book chapter on using randomized experiments to detect and estimate social influence in networks.  It will appear in the forthcoming “Spreading Dynamics in Social Systems.”  It’s really an all-star cast of chapter authors, so it’s quite humbling to be included.

You can download our chapter on Arxiv and find links to all of the other chapters on the book website.  The book will be published later this year.

It’s written mostly from a practitioner’s standpoint and is a good starting point to find many more advanced studies that might help you design and analyze field experiments in order to measure social influence.

We conceptualize an experiment as having four parts:

A target population of units (i.e. individuals, subjects, vertices, nodes) who are connected by some interaction network.

A treatment which can plausibly affect behaviors or interactions. 

A randomization strategy mapping units to probabilities of treatments. 

An outcome behavior or attitude of interest and measurement strategy for capturing it. 

Here’s the full abstract:

Estimation of social influence in networks can be substantially biased in observational studies due to homophily and network correlation in exposure to exogenous events. Randomized experiments, in which the researcher intervenes in the social system and uses randomization to determine how to do so, provide a methodology for credibly estimating of causal effects of social behaviors. In addition to addressing questions central to the social sciences, these estimates can form the basis for effective marketing and public policy. 

In this review, we discuss the design space of experiments to measure social influence through combinations of interventions and randomizations. We define an experiment as combination of (1) a target population of individuals connected by an observed interaction network, (2) a set of treatments whereby the researcher will intervene in the social system, (3) a randomization strategy which maps individuals or edges to treatments, and (4) a measurement of an outcome of interest after treatment has been assigned. We review experiments that demonstrate potential experimental designs and we evaluate their advantages and tradeoffs for answering different types of causal questions about social influence. We show how randomization also provides a basis for statistical inference when analyzing these experiments.

[seanjtaylor]

Prophet: a practical forecasting tool

My friend Ben Letham and I finally open sourced an in-house forecasting tool we developed at Facebook called Prophet.  It was incredibly exciting to release work done inside Facebook and make it available to the public.  We’ve already gotten a lot of great press and some excellent contributions.  Here are some important links:

Prophet homepage

Prophet blog post

Prophet manuscript

Github

We’d love it if you tried Prophet and found it useful.  Prophet has a very simple interface and is implemented in both R and Python so most data scientists should find it immediately useful.  Please tell us if you encounter issues or have success using it!

[seanjtaylor]

Discussion quality diffuses in the digital public square

I’m excited to announce that my paper with George Berry, “Discussion quality diffuses in the digital public square” was accepted for publication at WWW 2017.  You can download the pre-print here. 

We ran a field experiment to determine the effect of comment ranking on the comments that people see on Facebook, and more interestingly, the effect on the comments that they write.  We found that for even very high-quality comment authors, the ranking system caused them to write higher quality comments than they would have under a reverse chronological ordering.

Here is the abstract:

Studies of online social influence have demonstrated that friends have important effects on many types of behavior in a wide variety of settings. However, we know much less about how influence works among relative strangers in digital public squares, despite important conversations happening in such spaces. We present the results of a study on large public Facebook pages where we randomly used two different methods—most recent and social feedback—to order comments on posts. We find that the social feedback condition results in higher quality viewed comments and response comments. After measuring the average quality of comments written by users before the study, we find that social feedback has a positive effect on response quality for both low and high quality commenters. We draw on a theoretical framework of social norms to explain this empirical result. In order to examine the influence mechanism further, we measure the similarity between comments viewed and written during the study, finding that similarity increases for the highest quality contributors under the social feedback condition. This suggests that, in addition to norms, some individuals may respond with increased relevance to high-quality comments.

[seanjtaylor]

Identification of Peer Effects in Networked Panel Data

Excited to share that my paper with Daniel Rock and Sinan Aral, “Identification of Peer Effects in Networked Panel Data” was published in the Proceedings of the 2016 International Conference on Information Systems.

This work has a long history going back to early in graduate school for me.  Researchers are always looking for tricks to detect peer effects in observational data, and estimating time-dependent models with longitudinal data has always seemed like a promising approach.  We prove some results about a hybrid spatial-autoregressive model, extending the classic Bramoullé et al. (2009) results to the dynamic setting and estimating the model on some large-scale data.  Our results are somewhat pessimistic, in that it takes fairly strong (basically unrealistic) assumptions to identify a causal effect using panel data models. 

Here’s the abstract:

After product adoption, consumers make decisions about continued use. These choices can be influenced by peer decisions in networks, but identifying causal peer influence effects is challenging. Correlations in peer behavior may be driven by correlated effects, exogenous consumer and peer characteristics, or endogenous peer effects of behavior (Manski 1993). Extending the work of Bramoullé et al. (2009), we apply proofs of peer effect identification in networks under a set of exogeneity assumptions for the panel data case. With engagement data for Yahoo Go, a mobile application, we use the network topology of application users in an instrumental variables setup to estimate usage peer effects, comparing a variety of regression models. We find this type of analysis may be useful for ruling out endogenous peer effects as a driver of behavior. Omitted variables and violation of exogeneity assumptions can bias regression coefficients toward finding statistically significant peer effects.

AIS members can read the draft now, but we’ll be sharing a full draft with expanded findings soon.

[seanjtaylor]

Cartoon Character Space

I had a ton of fun crowdsourcing personality measurements of cartoon characters with Shiu Pei Luu. We were able to map the Big Five scale back to specific characters:

We also made a t-SNE embedding of the characters in 2D space:

Lots more in our writeup!

[seanjtaylor]

Facebook post on NFL fan friendships

A couple weeks ago I got to release my third analysis of NFL fans on Facebook:

2013: NFL Fans on Facebook

2014: The Emotional Highs and Lows of the NFL Season

2016: NFL Fan Friendships on Facebook

This year I focused on the friendships between US NFL fans.  I found some fun patterns in homophily and also re-made the original NFL map using a new visualization.

[seanjtaylor]

A Measurement Error Model of Dichotomous Democracy Status

I’m not actually a political scientist, but I like to pretend I am sometimes.  Jay Ulfelder and I just submitted a draft of our paper to SSRN that you can check out.  Here’s the abstract:

We use a Bayesian measurement error model to derive a probabilistic measure of democracy from several existing dichotomous data sets. This approach accepts the premise that democracy may usefully be construed as a bivalent concept for certain theoretical and technical purposes, but it also makes more explicit our collective uncertainty about where some cases fall in that binary scheme. We believe the resulting data provide a firmer foundation than measures of countries' degree of democracy for studies that require a dichotomous measure of regime type.

This allows us to do things like, measure the probabilistic concept like “an expert would say a country in a region is democratic,” borrowing information across time to produce plots like this one:

How we did it

This research is totally open and reproducible.  We used R and Python as well as some Stan to to implement our HMC estimation.  The code and the source data are available on github.

How this came about

I met Jay Ulfelder on Twitter a couple years ago and we periodically discuss research over email.  He emailed me last year to ask:

Is there a statistical model or process you know that solves the following problem?

Say you have 2 to 5 more or less independent, binary measures of the same thing in a time-series cross-sectional data set. Let's call those measures "votes" and think of them as each sources' best guess at the yes/no status of each record on some feature of interest. Is there a way---probably some kind of Bayesian model that didn't even exist when I was in grad school---to use those votes to generate a unified probabilistic measure or z-score or something similar of each case's status?

I had an idea about how to do this right away: item-response models provide a nice framework for thinking about this and all we had to do was model the underlying time series process. Bayesian inference was a perfect fit for this problem.

[seanjtaylor]

Learning About At-Risk Veterans Using Online Network Surveys

On December 12th last year, Carlos Diuk, Akos Lada, Alex Peysakhovich, and I hacked on Open Data and Innovations for Suicide Prevention, specifically working on addressing the issue for United States Veterans (of whom 22 commit suicide everyday). The hackathon was sponsored by the Whitehouse and the US Dept. of Veterans Affairs.

What We Did

We decided to treat at-risk veterans (those with depression, PTSD, or suicidal thoughts) as a "hidden" (i.e. hard to access and survey) population and designed a network-survey where we ask about friends who are veterans instead of the respondent, yielding information on network neighbors of the seed nodes we survey.  We’d like to thank Dennis Feehan, who’s an expert on this methodology, for his help in designing this strategy.

We first designed a survey, and then created a Facebook Page to create posts linking to the survey. We ran Facebook ads targeting those who were employees of the US Navy, Army, and Marine Corp, as well as those living near military bases and interested in those topics.

Within four hours, we had 400 brand new survey responses to analyze. We found patterns that seem to replicate existing understanding of incidences of risk factors in veterans. We also found some evidence for self-reporting bias in PTSD and suicidal thoughts, indicating that existing survey data may understate the problem. You can learn more from viewing our presentation slides:

What’s Cool About This

Using Facebook ads to recruit survey respondents yielded excellent cost per respondent (about 40 cents).

Survey instruments where you ask about the respondents friends instead of the respondent helps you find out about people who are hard to reach, such as people at-risk of mental health problems.

Asking about friends has the added benefit of potentially yielding a less-biased signal when there may be response bias. 

Next Steps

We’re actively collaborating the San Francisco VA to get this method into production:

Iterate on survey design and recruitment procedures (optimizing ads targeting and copy).

Recruit for this survey regularly to detect longitudinal trends in the incidence of mental health risk factors for veterans.

Work on survey-weighting to adjust for biased survey population. Produce estimates of population levels instead of rates.

Potentially direct survey respondents to resources for themselves and the friends they identify as at-risk.

[seanjtaylor]

Upcoming Talks and Lectures (Fall 2015)

I’ll be speaking at the following events this fall:

WIN Workshop at NYU: “Predicting Cultural Trends Using the Crowd” (10/2 - 10/3)

Babson College guest lecture for Davit Khachatryan: “Practical data science: tricks for predictive analytics” (10/15)

CODE Conference at MIT: “The Effects of Comment Quality on Comment Production” (10/16 - 10/17) [Will be presented by my collaborator George Berry]

Wharton School at University of Pennsylvania, guest lecture for Lynn Wu (10/19)

McCombs School of Business at UT Austin (11/6)

Social and Information Network Analysis at Stanford, guest lecture for Lada Adamic (12/1)

[seanjtaylor]

My friend and colleague Lada Adamic invited me to join on panel called “How Things Catch On” for the Sustainability@Scale 2015 workshop back on June 30th.  If you skip to about 18:00 in this video, you can listen to me talk about why most things don’t become popular and it’s hard to predict in advance which ones will.

[seanjtaylor]

Upcoming Talks

Lots of talks coming up for me.  Here's where to catch me and hear about the research I've been working on!

4/10-12 Data, Learning, and Inference : "Asymmetric Social Influence from Personalized Social Cues"

5/19 WWW 2015: "Tutorial on Online Experiments for Computational Social Science"

6/1 Collective Intelligence: "Sensors: Predicting the Future with Convenience Samples"

8/12 Joint Statistical Meetings: "Forecasting with big data: Finding and using behavioral 'sensors' to predict the future"

[seanjtaylor]

Putting the Magic in Data Science

Here are slides from my talk at QCon last week:

Here's my talk (which was not recorded) in a nutshell:

If we want to be valuable as data scientists, we should aspire to create as many "how did they do that?" moments as we can.  I call these "hoverboards."  If we just count things, we are being terrible magicians. We probably don't want to end up being the accountants of the 21st century.

Furthermore, these moments of magic should have impact -- they should cause strategic or tactical decisions that people make to change.

I argue that magic in data science often comes from combining various "tricks" in novel ways. I describe four common tricks we use at Facebook, as well as a grab bag of others that I've found useful.

Tricks alone are not enough.  People have to use the technology you create.  That requires considering what I call Data Science's last mile problem. How do we make the data inform/change people's behaviors?

I describe four important (non-exhaustive) last mile considerations: reliability, latency/interactivity, simplicity, and interestingness. Many of these are achieved not through data science alone, but by combining data science with tricks from software engineering, design, and computer science.

What is Data Science?

There's one little side-argument I made that I'd like to cover here. One thing we have trouble agreeing on is what data science actually is. My latest theory is we all work in different parts of technology pipeline (see below), which starts with academics publishing papers at conferences like KDD and NIPS and ends with non-experts getting utility from products and services we create. We're all probably doing "data science" but it might look a lot different (pure math, problem discovery, design/visualization, engineering) depending on what stage you're involved in.

[seanjtaylor]

World Cup Recap

I know it's a couple months late, but I wanted to recap some of the fun stuff I worked on at Facebook for the World Cup.  All of this work was a collaboration with my friends Dustin Cable and Alan Clark.

// <![CDATA[ (function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/all.js#xfbml=1"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk')); // ]]>

Post by Sports on Facebook.

First, after the first week of games, we compiled all the of the first checkins in Brazil (actually within a certain radius of any of the 12 host cities) in month of June.  We used the first checkin as a crude measure of a person's "arrival" in Brazil to attend.  Dustin then worked on animating the paths that people took across the globe from the current towns to the cities they first checked in at in Brazil.

Next, we took all these people and induced a social network on them.  We took the population of all people who checked in on Facebook in Brazil sometime during June and found any friendship that was created between two of these people after at least one of them arrived.  We grouped this by pairs of cities to show the new international friendships that formed during the World Cup.

There are many more details here in this Facebook Data Science blog post and Time magazine also did a writeup. I'm not a huge soccer fan, but the World Cup is such an huge, international event that it was fun to study from the Facebook lens.

[seanjtaylor]

JSM Talk on Big Data

My friend Sherri Rose asked me to be a discussant for a session at the Joint Statistical Meetings called Innovations in Statistics for Big Data from the Next Generation. It was a real honor to hang out with and discuss statistics for big data with so many talented people.

You can find my slides here:

The point of my talk was that in my experience at Facebook, I have noticed four main ways in which having big data is often limited:

Measuring the wrong thing: you have a lot of data because you are substituting a cheaper, easier to measure quantity for what you really want to observe.

Bias: you have big data because you're observing a specific population which is easier to get data for (e.g. the people who use your service).

Dependency: you have a lot of data because you make many repeated measurements of the same units.

Uncommon support: although you have a lot of data, it can be difficult to make comparisons you want to make (for instance for the purposes of causal inference) because the set of observations satisfying some set of complex criteria are small.

To summarize my argument:  If you have a lot of data, you should probably ask yourself "why do I have so much data?"  It's often because the data were not constructed or collected to answer your specific question and this implies that you'll likely have to cope with some of the above limitations.

[seanjtaylor]

Acknowledgments

The following is an excerpt from my dissertation. It felt sad burying all my gratitude in a gigantic research document that not many people will read. To all the amazing people in my life that aren't mentioned here: this was written somewhat (very) hastily and I'm (probably) grateful to you as well :)

At the end, graduate school felt like Zeno's dichotomy paradox. In order to get halfway to finish, I had to get halfway to that halfway point. In order to get there, I must have gone halfway once again. So there were thousands of tiny steps that were perhaps forgettable at the time, but comprised something much greater in aggregate. I took most of those infinitesimal steps with someone's help, which makes it difficult to properly acknowledge all the people who made it possible for me to achieve the most ambitious and important thing I have ever undertaken. I will do my best here.

My family has been a constant source of motivation, and most importantly, got me to the starting line of this endeavor with all the tools I needed to succeed. I have the most incredible parents I could imagine and I owe my intellectual curiosity to their emphasis on education and my ambitiousness to their unwavering support. To my brother and sisters: I'm proud to be your older brother and I'll always be more impressed with the things you accomplish than anything I do myself.

There's a social support system built into graduate school -- dozens of people who I experienced tribulations and triumphs with over the past few years. They've become friends for life. There are too many to mention here, but my biggest thanks go to my officemates Lauren Rhue, Beibei Li, and Jing Wang, as well as my study-break partners, Erin Smith and Jason Hong. I can't imagine doing something like this without people as amazing as you all to share it with. To my friends who weren't in grad school with me but were there by my side during it: thank you for dragging me out to drinks, on trips, and to all those weddings. Thanks also for picking up the check now and again when I had experienced a larger-than-expected expense shock.

The Information Systems group at NYU, characteristic of IS community in general, was an amazing resource and a perfect fit for my interests (although I perhaps did not know that in advance). I thank them for knowing that I would fit well in their program and for providing a stimulating academic environment in which I could thrive. In particular I would like to thank Arun Sundararajan, Natalia Levina, Vasant Dhar, Panos Ipeirotis, and Sonny Tambe for their open office doors and mentorship.

Many impressive people at Facebook -- too many to list here -- contributed to the quality of the work in this dissertation and made my life a lot more fun for a year during graduate school. Two in particular, Eytan Bakshy and Dean Eckles, are like the academic brothers I was separated from at (academic) birth. They've been as supportive (and sometimes as hard on me) as brothers as well. I hope to continue collaborating with them long into my "stodgy old scientist" days.

My committee members, Anindya Ghose, Foster Provost, and Duncan Watts, went above and beyond what I could have expected. I thank them all for their support, kind criticism, and lenience with due dates. I hope that someday I am able to collaborate more directly with each one of them.

I remember meeting my advisor, Sinan Aral, for the first time and realizing he had actually read the research statement in my graduate school application. It was terrifying to be pushed on my ideas and realize the high standard of thinking for which I had signed up. But Sinan was then -- and has always been -- supportive of what I think and constructive in his feedback. Hundreds more discussions like that in his office comprise much of my learning in graduate school. The only pressure I ever felt from him was through the notion that good ideas and important questions deserve hard work. His attitude brought deep meaning to the research I was doing and made it easier to push through the setbacks. I can't thank him enough for all this and too many other ways he's gone beyond what I could have ever reasonably asked for from an advisor.

[seanjtaylor]

Learn Exploratory Data Analysis

My friends Moira, Dean, and Solomon - all members of Facebook's Data Science team - worked with Udacity to create a fantastic course on exploratory data analysis. If you're new to R/ggplot, or just want to hear about how experts think about visualizing and exploring data, this would be a great place to start. I really can't recommend these instructors highly enough.

They asked several members of our team to talk about an EDA project they worked on and these interviews are included in the course. You can check out me talking about visualizing the sentiment from posts about NFL teams here.  I talk about using splines versus more flexible models for time series data and the bias-variance tradeoff.