#listmaster

[arlenestein]

The master of organization. #listmaster #rawchs @matthewjennings

[making-change-org]

At Change.org we show you petitions.

At Change.org we save a lot of petitions. We also show a lot of petitions. We show petitions based on their popularity over the past day, week, or month. We show them based on what country and language the user is browsing in. We show them based on their tags. We show them based on which Wildfire campaign[1] they’re in. We show a lot of petitions.

We really like showing the list of petitions that are victories including the special set of petitions that we feature on our homepage.

Here’s what a Cucumber (test) scenario might look like:

Given I am on "change.org/petitions" Then I should see featured petitions And I should see recent petitions And I should see popular petitions And I should see petitions in my locale

In order to show the user the right content we need to be able to splice our petitions into lists of all kinds of combinations. An example might be a list of the active petitions written in French that are applicable to Canada about a plastic bag Wildfire campaign ordered by last week’s popularity. Executing this in SQL is a poor decision for multiple reasons. Firstly, a query like this would require numerous indices, slowing down petition creation and increasing our database size. Secondly, any reduction in the number of SQL queries per request decreases our overall response time, another important consideration.

To solve this problem we decided to build our own indexing tool that is aware of the many petition lists we need and can return the ids of petitions in any combination of those lists. When fetching the petition data from SQL we can use primary keys which makes the SQL queries really fast.

When searching for a datastore that could arbitrarily intersect lists, we settled on Redis. Redis has two different datatypes that come in handy when splicing data into arbitrary buckets: sets and sorted sets (http://redis.io/topics/data-types.) Leveraging the performance of Redis set manipulation, we’re able to quickly organize our petitions in any set imaginable. We iterate over every petition and place them in corresponding sets in a recurring background job. This ensures our lists are up to date with what’s happening on the site within a given time constraint (based on how often we run the job).

Using the example above we have 4 sets and 1 sorted set. A set of petitions written in French, a set of petitions relevant for Canada, a set of petitions about plastic bags, a set of all active petitions, and a sorted set of all petitions by the past week’s popularity. Our indexing tool might produce the following sets in Redis:

# pure sets: (no order) locale:en-US: {1, 3, ...} locale:fr-FR: {2, 4, ...} country:US: {3, 5, ...} country:Canada: {1, 2, ...} campaign:plastic-bags: {3, 4, ...} campaign:boy-scouts: {1, 2, ...} active: {1, 2, ...} # sorted sets: popular-weekly: [4, 1, ...] popular-daily: [3, 4, ...]

Finding our special list of petitions involves intersecting the appropriate sets. For our example, we would issue the following Redis command:

zinterstore output popular-weekly locale:fr-FR country:Canada campaign:plastic-bags active

Now the ids of the petitions we want can be read from the output Redis key, and passed to SQL to fetch the petition data. Redis makes it easy to specify a limit and offset when reading the ids, which gives us pagination for free.

This method is not without its caveats. Having a recurring background job that places petitions in sets means we don't always have get up-to-the-second data, but for lists of featured petitions or victories, this method is quite satisfactory. It works best in a system that has many more reads than writes.

We have “gemified” the process above into an open source library that we use internally, called ListMaster, and we welcome feedback on the tool. It may not be ideal for every application, but we have been using this tool for over a year and it has proven extremely helpful as our read traffic has grown and as we've expanded to lots of international markets. Having a consistent way of slicing and dicing petitions allows us to explore even more ways for users to discover petitions that matter to them.

Chase Stubblefield & Thomas Shafer Web Engineers

[1] A Wildfire campaign is a group petitions on the same topic started  by different people asking their local decision makers to make change 

We are hiring! If you want to come work with us and help empower people to Change the world while working on amazing technology check out our jobs page or email us directly: chase at change dot org or thomas at change dot org