GetStream Blog

Welcome to the GetStream official blog.

[getstream]

Building news feeds & activity streams with Meteor

Today we are happy to announce stream-meteor. This integration library makes it easy to build scalable newsfeeds and activity streams for your Meteor app. It's powered by getstream.io and scales to millions of users.

In this tutorial we will demonstrate how you can use the integration package. To keep this tutorial short and skip the boilerplate, we will use the awesome demo application localmarket as a starting point and use stream-meteor to build the "What's cooking" section.

Install

Lets start by installing the localmarket example app to your local machine. Run

git clone [email protected]:matthisk/localmarket.git

to install the example app. For the next step be sure to have Meteor installed on your machine. Enter the created directory and install the integration package by running:

cd localmarket ``` <br>

meteor add getstream:stream-meteor ```

Settings

Before our Meteor backend can communicate with Stream we need to set public and private keys in our settings.json. Create a file called settings.json in the root directory of your app and add the following code:

Stream's free plan includes 3 million feed updates. You can login with Github and get your API key in the dashboard.

Running

You can now run the example app by executing the command

meteor run --settings settings.json

from the app's directory. Notice that we need to supply the settings file explicitely to the meteor command. If you forget to specify the settings file API calls to Stream will fail.

You can now open http://localhost:3000 in your browser to have a look at the example app without the getstream.io integration. You will be able to authenticate using your Twitter account, bookmark recipes and add a photo of recipes you cooked. For the remainder of this post we will focus on the What's Cooking page which is the news feed page of the localmarket app.

Following feeds

Real apps have many users, each following other users or topics. In this tutorial we don't have real users to follow so in order to see some activity on the "What's cooking" feed we will cheat a little and follow ourselves. This ensures that activities performed by the user show up on his own "What's cooking" feeds.

To do this we have to call followUser on the feedManager after we have created a new user. Luckily the localmarket app already includes a hook that is executed after a user is created. Open up the file ‘users.js’ and add the following lines to the method:

remark: In a real application the "What's cooking" feed would display the content from followed users.

Be sure to create a new account if you already created an account for your localmarket app, otherwise the What's Cooking page will not display any activities.

Adding entries to the feed

Feeds contain activities. Activities are Stream's main type of data and follow the activities specification. An example activity on localmarket is "Matt bookmarked the recipe 'Classic Ragu Bolognese'" or "Eric shared the recipe 'Lasagne'". We will explain more about this later on.

There are two different types of activity in the localmarket app: Shares and Bookmarks. Whenever a bookmark is created or recipe is shared, our application will convert that into an activity and send it to Stream's APIs.

Activity collections

Let's start by implementing shares in the "What's cooking feed". The localmarket example app already contains a Collection which stores shares. The integration package allows us to hook into Collections so that any document added to it is converted to an activity and send to Stream's API. To do this we call the method registerActivity on the Stream object, add the following code to 'activities.js’ after the Activities collection has been defined:

To be able to send valid activities to the API we need to inform the hook of which fields/values to use for the required activity properties. We set the activities verb to 'cook' and retrieve the actor from the 'userId' property on the inserted document. object is in this case the serialized document we have inserted into our collection.

You can now create an account for you localmarket app by linking it to Twitter. Try adding a picture to a recipe and see if the activity automatically shows up in Stream's databrowser.

Manually adding activities

In the file bookmarks.js you will find a Meteor method bookmarkRecipe this method is called when a user clicks the bookmark icon on a recipe page. We do not have to alter the way the app stores the bookmark on the user's document we just have to add couple of lines to the end of the method to register an activity with Stream:

The integration package exposes all its methods on the Stream object, one of the properties is an instance of the FeedManager named feedManager. The FeedManager exposes methods to retrieve the different news feeds from our current user. In this case we want to add an activity to the current users' user feed.

Activities registered should have atleast the following fields to be valid: actor, object and verb. Here we set actor to a document reference of the current user's user document (i.e. the name of the collection colon the id of the document). The verb of the action is 'bookmark' and the object is the name of the recipe.

At this point any documents added to the Activities collection and any bookmark created through the createBookmark method are added to the user feed. The next step is to define a way of retrieving the activities from getstream.io and showing them to the user on the 'What's cooking' page.

Feed subscriptions

Feed activities are exposed through a publication in the 'Stream.feeds' namespace. When we subscribe to this publication we receive the feed's last 20 activities and receive reactive updates on any new activities added to the feed. Lets have a look at how to set this up for the localmarket example app.

In the file router.js there is already a variable called feedSubscription, we are going to update this subscription to point to 'Stream.feeds.flat':

This will retrieve the last 20 activities registered with the user's flat feed. To actually show the activities on the "What's cooking" page we have to retrieve them from a different collection in 'whats-cooking.js':

A subscription to a Stream feed stores its results in a collection on Stream.feeds with the name of the feed group you subscribed to.

Templates

To display the feed activities correctly we have to make a few changes to the existing templates and add one new template to display bookmarks inside the news feed.

The feed will contain two different type of activities, 'bookmark' and 'cook', we should update the 'activity' template to be able to render both activities, change the template in 'activity.html':

The equals method is not available by default in Handlebars so add it to "helpers.js":

The bookmark template does not exist yet, so we should add it to the 'activity.html' file:

The cook template is now our old 'activity' template, add the following code to 'activity.html':

And add the following template helpers for the 'bookmark' template to 'activity.js':

Since we renamed the 'activity' template to 'cook' we also need to replace the helpers associated to the 'activity' template in 'activity.js'. Replace the helpers associated to this template with the following code:

Now any activities added on your account in the app should show up on the "What's cooking" page.

Home page

As a bonus you can also show the last 3 activities of the user's flat feed on the homepage. First add the feed subscription to the page's router in router.js by replacing the HomeController with the following code:

Then replace the activities method in 'home.js' to retrieve items from the flat feed Collection:

Now the last three activities will display on the home page.

Conclusion

Meteor is an amazing framework and I hope you enjoyed this tutorial on how to integrate Meteor with Stream. Using getstream.io you can build, scale and personalize your newsfeeds and activity streams.

This example still misses a few features. For one, there is no way to follow other users. Also we didn't add a notification feed. We'll leave that to you as an exercise.

Be sure to read the full documentation on the github page. And have a look at the Pinterest example app build with stream-meteor, which does implement all of these features.

If this is your first time using getstream.io you might also want to try this quick 5m tutorial.

[getstream]

Introducing ranked feeds

Today we want to share the first BETA release of ranked feeds. The ranked feeds allow you to take control over how your feeds are sorted and move past the simple chronological view. Ranked feeds can help improve engagement with the feed.

You can use a decay_gauss, decay_exp or decay_linear to discount older items. A few example use cases:

Show featured activities higher in the feed.

Show popular or activities with many comments higher in the feed.

Show promoted activities higher in the feed.

Note that ranked feeds are currently only available on paid plans and can only be enabled for flat feeds. 

As always, we’d love to hear from you. This is a BETA release, to use this feature email support to enable it for your account. Please share your feedback and email support at getstream.io What’s next We’re currently alpha testing our new analytics platform and the personalized feeds. Register here for early access.

[getstream]

Stream Javascript/Node client v3

Today we have released a new version of our Stream JavaScript client. In this version we have added support for new Stream API features, reduced the size of the distributable JavaScript file, created library documentation pages, and improved browser compatibility of our client. Continue reading to find out about all new features introduced in this release.

Promises

All methods that perform an asynchronous request to the Stream API now return a promise. These promises help you avoid 'callback hell' and chain API calls. If you've never heard about promises read this article on HTLM5 rocks. As the following example demonstrates the promises allow us to chain two asynchronous calls after eachother while only handling errors at the end of the promise chain, instead of nesting within callbacks. Callbacks are still supported but the XHR Request object can nolonger be manipulated since it is not returned.

The error object supplied to an error handler contains two properties: the response object created by the XHRRequest and an error object with additional information on the origin of the failure.

Batch Operations

Two new API features for batch operations have been added to the client. First it is now possible to add one activity to multiple feeds through a single API call, with the method addToMany on the client object. The second batch operation allows you to create multiple follow relations through the use of followMany.

This code example illustrates how to use addToMany call to add one activity to multiple feeds, the activity is supplied as an object and feed slugs + identifiers are supplied as a string in an array.

To create multiple follow relations within a single api call, use the followMany function of client.

Copy limit

If you want to create a follow relation without copying the history of the target feed, this is now possible through the use of 'activity_copy_limit' API feature. The following example illustrates how to use the copy limit to either copy over no history or limit it to a maximum number of activities:

Expire tokens

When connecting to stream through .connect you can supply additional arguments in the fourth parameter. It is now possible to enable expiration of tokens generated by the stream client. The default behaviour is to never expire the generated tokens.

[getstream]

Stream raises $1.75m seed round

Today we're excited to announce that Stream has raised $1.75m from Brad Feld/FGAngels, Techstars Ventures, Tahoma Ventures, Social Starts, Galvanize, Dharmesh Shah, Wayne Chang, Jud Valeski, Kyle Wild, James Powell and others. 

Stream (getstream.io) is a new kind of data store designed for building, scaling and personalizing feeds. The company is founded by the authors of the popular open source Stream-Framework, which has over 250,000 downloads.

More than 500,000 apps are still building and maintaining their own feed technology. All these companies go through the same struggle and reinvent the same technology. Stream enables developers to build feeds in days instead of months. Feeds are intelligently aggregated and re-ranked based on relevancy. Feeds were pioneered by social apps such as Facebook and Twitter. Over the past years this pattern has spread to every app with a large amount of content and a desire to personalize what’s shown to the user. Feeds have become the dominant way through which information is discovered online.

Stream is using this investment to expand the team and improve the analytics and personalization features. The company uses machine learning and big data to improve the relevancy of content shown to the user. The personalization functionality enables apps to improve engagement, conversion and retention. At the moment Stream’s running closed trials with the personalization features, you can use this link to apply for early access.

Techstars has been a great learning experience for us. The network is incredibly powerful. In the first weeks we were introduced to dozens of successful founders. Some of them ended up mentoring and even investing in Stream. The give first motto is a great thing and everybody has been extremely helpful. Also the quality of the founders in our batch was incredibly high. It’s inspiring to be surrounded by a group of ambitious and talented entrepreneurs.

Stream was founded in Amsterdam by Thierry Schellenbach and Tommaso Barbugli. The company participated in Techstars NYC and relocated to Boulder, CO after the program.

Additional info

Press

TechCrunch

BoulderOpelis

Quotes

Jud Valeski (Founder GNIP): “The feed UI content consumption metaphor is here to stay and saas'ing that is a natural evolution. Adding a feedback loop to it in a generic manner is a huge opportunity; simplifies the lives of developers and product owners in a huge way!”

Dharmesh Shah (Founder/CTO Hubspot): “Fantastic technical team building a service that I could use for a couple of projects myself some day.”

Ari Newman (Techstars): “If your app or service doesn't have a data stream with personalized info already, its probably on the roadmap. Don't re-invent the wheel, just use Stream.”

Thierry Schellenbach (CEO): “Techstars’ network is incredibly powerful. In the first weeks we were introduced to dozens of successful founders. Some of them ended up mentoring and even investing in Stream. The give first motto is a great thing and everybody has been extremely helpful. Also the quality of the founders in our batch was incredibly high. It’s inspiring to be surrounded by a group of ambitious and talented entrepreneurs.” Thierry Schellenbach (CEO): “There are 500,000 apps with feeds. Companies spent months scaling their feed technology and reinvent the same technology over and over again. Stream makes it easy to build a scalable feed and can improve your app’s engagement using analytics and personalization.”

Thierry Schellenbach (CEO): “Feeds are the dominant pattern through which we discover information online. Stream allows our customers to improve engagement, conversion and retention within their apps. It’s an incredibly exciting opportunity”

Tommaso Barbugli (CTO): “I've seen many companies spent months, just to reinvent their own feed technology. Stream allows you to build scalable and personalized feeds in a matter of hours.”

Links:

getstream.io AngelList & Hiring Techstars

Assets

[getstream]

GetStream/django_twitter

Django Twitter - An example app built using getstream.io

[getstream]

[getstream]

Redis - Reducing memory usage

High level tips

Most of Stream-Framework's users start out with Redis and eventually move to Cassandra because Redis becomes too expensive to host. Stream, our hosted API for building scalable newsfeeds & activity streams is also based on Cassandra. 

There are quite a few things you can do to reduce Redis' memory usage though. Here are my favourite 6 high level tips for reducing Redis memory usage. For lower level tips, see the Redis docs on memory optimization.

Tip 1 - Serialization

It is often possible to reduce your Redis memory usage by using the right serialization. Serialize objects using built-in serialization mechanism is very tempting (eg. Python's pickle, PHP's serialize...) but it will likely waste lot of memory for nothing. You should stay away from it and make sure you only serialize what you need. Using something like JSON or Protobuf is often a good idea.

Tip 2 - Compression

If the data you're storing is large enough and contains a lot of text you can often reduce memory usage by adding compression. LZO or Snappy are good options for this use case.

Tip 3 - Fallback to disk based storage (aka your rdbms)

Many times you don't need to store all the data in Redis. If your model allows it, try to store only hot or recent data and hit the database when a request for missing data is performed.

If you go for this approach be sure to include an endmarker in your data. Assuming there are 3 items in your database, Redis would store the 3 items and the end marker. This way you know you don't have to run a database query after item number 3. The end marker approach vastly reduces how frequently you fall back to the database.

Tip 4 - Normalize your storage

Memory is expensive and Redis only uses memory. Take this into account when designing your data structure. A normalized approach is often the better choice compared to a denormalized option.

Tip 5 - Expire and trim the data

If possible, set expirations on everything. This presents data which is rarely used from sticking around in your cluster. It can be a few days, a few weeks, but definitely set some sort of expiration on your data. If you're inserting into lists or sets be sure to trim them occasionally.

Tip 6 - Use the right eviction policy

If the amount of data that you store in Redis grows over time and you can't afford to keep it all in memory, you probably want to configure Redis as a LRU cache storage. Redis allows for 6 different eviction policies, make sure you pick the right one!

Most of these tips are obvious, but before you switch away from Redis, be sure to give them a try.

[getstream]

Python 3 & Stream Framework

I'm excited to let you know that Stream Framework now runs on Python 3.4! As proof have a look at this beautiful report on Travis. Many thanks to Anislav Atanasov for making this happen.  

More and more users of open source software seem to be asking for Python 3 support. This is just my very unscientific observation, but it seems as though we're reaching a tipping point. People are building their apps to run on 3.4 and library authors are adding support.

Till recently there was simply no point in upgrading to 3.4. Nobody asked for it, and our dependencies didn’t support Python 3. We've been getting requests for Python 3 for a few months now and recently CQLEngine added support for Python 3. Many thanks to Jon Haddad and Blake Eggleston!

Awesome to see that the community is moving on, it was about time for Python 3 to become a viable option :)

[getstream]

Jobs at Stream - Techstars NYC 2015

We're currently participating in the prestigious Techstars NYC program and are searching for ambitious developers to join our team. About Stream

Stream enables you to build scalable newsfeeds and activity streams. The company is founded by the authors of the popular open source Stream-Framework. Our team has more than 15 years of experience with Python development. Our stack is built on top of Python, Cassandra, Redis, RabbitMQ, AWS and Spark. Newsfeed and activity streams are present in 15% of all apps. They are however extremely hard and expensive to build and maintain at scale. We aim to make it extremely easy to setup scalable newsfeeds.

Some of our favourite articles on these topics are listed in the readme of Stream Framework.

Experienced Python developer

We're searching for an experienced Python developer to join us for the coming months at Techstars NYC. 

Requirements

Python, at least 2 years of work experience

Dedication to test driven development

Willing to stay in NYC for 1-2 months for the Techstars experience

Bonus points for experience with:

Java, Ruby, Node, Meteor, PHP, Objective C, Swift

Cassandra, Redis, Spark

Celery & RabbitMQ

Open source contributions

Machine learning

Contact [email protected] for more information about this position.

[getstream]

5 reasons to use Cassandra for building your newsfeed

Users of the open source Stream-Framework often ask us if they should use Redis or Cassandra to power their newsfeed. This article highlights 5 scenarios in which you're better of going for Cassandra. We're not comparing Redis & Cassandra, only their application for the specific domain of building newsfeeds.

Benefits of using Cassandra

Let's start by explaining some of the high level advantages of using Cassandra:

1.) Cassandra stores to disk, which gives you an important cost advantage compared to any Redis based approach.

2.) Data is automatically sharded across the nodes in your cluster. Furthermore you can configure the cluster in such a way that data is replicated and your system is highly available.

3.) The best part about Cassandra is that you can easily add nodes. As your user base is growing it's easy to add more capacity. 4.) The opscenter monitoring tool is very powerful and gives you great insights into your cluster's usage and health.

Cassandra & Newsfeeds

If one or more of the following points are true for your newsfeed functionality you probably want to look into a Cassandra based approach.

A.) Rapid growth

If you are growing rapidly and are exceeding the amount of data you can easily store in a single Redis node you're definitely better of moving to Cassandra. It takes quite some time to learn how to maintain and operate Cassandra, but it's definitely easier than building your own resharding mechanism for Redis.

B.) Cloud hosting

Redis can quickly become expensive. The most famous example is Instagram spending tons on AWS Redis hosting. Redis stores all data in memory, which is especially expensive when you're running in the cloud. If you're on cloud hosting, you want to make sure you don't end up running a large cluster of Redis machines. 

C.) Database fallback When you're using Redis you want to do your best to reduce memory usage. One common way of achieving this is to store only active users or only the first bit of the newsfeed in Redis. This approach works because you can often fallback to the database in the rare scenario that data outside of Redis is requested. Now this approach only works if you can actually fall back to the database. If your feature set combined with your traffic make it hard to fall back to your relational database you should start thinking about Cassandra. D.) Aggregated feeds

Aggregated feeds take up more storage space than flat feeds. A common example of an aggregated activity is "Thierry, Tommaso and 5 other people like your picture". Storing these aggregated feeds takes more memory than a simple flat feed. Furthermore they also make it harder to fallback to the database. Cassandra is often the better option if you want to support aggregated feeds and have a large user base. E.) Costs

If your infrastructure bill for Redis is above 5.000 a month you should definitely start looking at Cassandra based solutions. Often your Cassandra based solution will be 4 to 10 times more affordable than the Redis based alternative.

The decision

I hope this helps with your decision to use Redis or Cassandra for your newsfeed. Note that our open source Stream-Framework supports both Redis and Cassandra.  If you're looking for a more high level solution, you should have a look at getstream.io, which gives you a beautiful REST api for creating your newsfeed. (Based on a highly optimized Cassandra 2.0 cluster).

[getstream]

5 brilliant articles about newsfeeds & activity streams

I wanted to highlight some of the best articles out there about newsfeed and activity stream development. As an author of Stream-Framework and the hosted getstream.io I've read most of them and here are my favourites:

Twitter 2010 - by Nick Kallen (@nk)

Starting from slide 24 it explains the troubles with Twitter's original approach. It also goes over the tradeoffs with memory vs disk storage.

Twitter's timelines at scale 2013 - by Raffi Krikorian (@raffi)

It's nice to see how they've progressed over the years. The 2013 article lists a pure fanout approach using Redis. At the time they had 150 million active users. It's interesting to see how they've optimized the Redis memory usage. I still wonder though how much they were spending on this approach. It seems quite expensive to operate.

Yahoo Research & Princeton 2010 - by Adam Silberstein et al (@YahooResearch)

One of the issues with a pure fanout approach is that it's expensive. Furthermore it also gets tricky when you have extremely popular users on your platform. In Twitter's 2013 approach every tweet by Katy Perry leads to 65 million updates. The Yahoo paper discusses how you can reduce the overall system cost (at the expense of performance) by combining a push and a pull approach. The decision to push or pull is based on the producer/consumer relation. In theory it's a very interesting approach, I haven't seen any app use this approach in production though.

Etsy's feed architecture 2011 - by Dan McKinley (@mcfunley)

Etsy supports a very rich feature set for their newsfeed. They mix in popular content into the experience and they also do aggregation (rollups).

The presentation describes how they try to pre select a subset of follow relations based on affinity. From the presentation it looks as though they build the feed in the background using Gearman and subsequently store it in Memcached. 

I have the feeling their approach changed over the years though and would love to see a post explaining the current architecture.

Instagram & Cassandra 2014 - by Rick Branson (@rbranson)

Instagram also uses a pure fanout approach but it's based on Cassandra instead of Redis. They use their own tombstone type of approach to prevent race conditions when adding and removing activities. I hope Rick will post a more detailed approach about their setup soon. Also see, messaging at scale and Cassandra at Instagram.

Our own infrastructure at getstream.io is similar to Instagram's approach and based on Cassandra 2.0. If you think I've missed a great article be sure to add it in the comments! In the coming weeks we'll write a post about when to use Cassandra and when to go for Redis when building your newsfeed.

[getstream]

Stream BETA & Techstars NY

Thank you for all your feedback during the beta period. We're happy about the stability, feature set and documentation and are officially going to remove the beta sticker. \0/

Furthermore we're excited to announce that Stream is participating in Techstars NY. One of the world's largest business accelerators is helping us grow and scale the product.

We've released 4 new features to support our larger users. First of all, you can now import data via the json format. Secondly we've created a flow for following many users at once. This is often useful for the onboarding experience in your app. Third, you can now broadcast an activity to multiple feeds at once. Fourth we're allowing you to listen to feed changes using webhooks and SQS queues. These hooks can be used to for instance send push notifications to your users.

As always feel free to get in touch with support if you run into any issues. Also, if you're in NY, give a shout and we'll grab a drink.

Thanks for beta testing!

Thierry & Tommaso

[getstream]

REST docs

We've received many requests for Java, Go, C# and even Coldfusion clients. As a tiny step towards supporting more languages we've released our internal REST docs. These will prove helpful if you're writing your own client.

Even though there were no docs, some users already started writing their own clients. This GO client by hyperworks is starting to look pretty good. 

As always, if you run into trouble feel free to get in touch with [email protected]

[getstream]

Scalability as a service, Guest post HighScalability

Thierry wrote a guest post on HighScalability called Scalability as a service. It explains the trend how development is moving towards a hosted component based approach.

[getstream]

Multi region support

API requests to Stream take around 50-100ms to complete. However, now that we're closing in on our first 1000 users, many of these requests are from outside of Europe. In fact both the US east coast and Asia are growing quickly. Latency to US east coast can easily add an additional 400ms to your response times. The numbers for the US west coast and Singapore are even worse.

Today we're shipping the first iteration of multi-region support. We now replicate part of our infrastructure in:

us-west-1

us-east-1

ap-northeast-1

eu-west-1

For apps running in the US and Asia this represents a 50-75% performance improvement. In the coming weeks we're making region selection configurable in the dashboard. You can already take advantage of this today by getting in touch with [email protected]. Simply let us know the region you would like to use and where your app's servers are located.

We're excited about this huge step forward in terms of performance. As our user base grows we'll eventually replicate larger parts of our stack to further improve response times.

If your preferred region is missing from our 4 launching regions please get in touch.

[getstream]

Teams, Organizations and Topics

The most basic pattern in social networking is to follow other users. However, many apps will also allow you to follow more complex concepts such as teams, organizations, topics, sites etc.

One of our customers for instance is a soccer related community in Brazil which allows you to follow teams. Another example is is a travel related start up in San Francisco which allows you to follow locations. A more complex example is a job related site where you can follow topics and countries.

When you're using getstream.io you commonly write to the user feed and use the "TO" functionality to write to secondary feeds such as teams, organizations and topics.

Today we're announcing three changes which will make the TO support more powerful:

When you remove an activity from a user's feed, the feeds which were updated using the TO parameter are also updated.

When you add an activity via TO this will now trigger a fanout. This means that users which follow the team feed will also see the updates.

Stream now returns the origin field for activities. This gives you insights into which feeds caused the content to be included in the feed you're retrieving.

These changes make it incredibly easy to support communities with teams, topics, or organizations. Have a look at the documentation for an example.

[getstream]

Build a scalable Twitter clone with Django and GetStream.io