#realtime web

[whitengray-blog]

Choosing your Realtime Web App Tech Stack - Phil @leggetter

[suryaray-blog]

Realtime & the off switch

#SuryaRay #Surya Jeremy Toeman is looking to quit “real-time” and take back control of his life. If you are reading this and thinking ‘that guy’s just an old-fuddy-duddy’ (which, to be fair, no young person today would ever actually say), and you are also patting yourself on the back because YOU are a great multitasker, go take a break from this piece, google ‘multitasking myths’ (or just read this) and then come on back.  Bummer, eh? And it’s not just about getting stuff done, as that too is just massively overrated.  It’s about a lack of peace and calmness. When do we take time anymore just to do nothing.  Even standing in line for a coffee (which is, of course a take out coffee, since there’s no time to just sit in a cafe and enjoy a hot cup of coffee in a real cup) everyone’s on their phones, doing stuff.  The human brain actually needs time, every day, just to do nothing and process all of the events that are transpiring (great article here on ‘doing nothing’). I couldn’t agree with him more — real time has its benefits and it is great, but there is a time, when everything doesn’t need to be instantaneous. I have been trying to do the same: the email, Twitter, Facebook and Instagram apps have moved off the front screen to the second screen of my iPhone. But that’s still a bandaid. As Nova Spivack once said in a conversation: “With the real-time web, the amount of information we have to handle is changing the Now,’ he said. ‘Now is becoming a lot denser. There’s a lot more information in per unit of Now. The Now is getting shorter. The horizon is getting narrower. Now has gone from days to hours to seconds.’” Some might say — turn off the notifications and exercise some self control. I have tried that, but the behavior of constantly reaching for my iPhone has become too ingrained in my mind. You take out your phone to take a photo or send a text message and next thing you are doing is checking emails, liking photos on Instagram, using Foursquare and reading the latest tweets. There are some apps which give some control over real-time: Tweetbot for instance has a sleep option, which I religiously use. I have removed the Facebook app and use their website when I need to, so it has made the process a little bit slower than before. My colleague Mathew Ingram brought up the real human challenges of the realtime in a brilliant post over two years ago. As a result, our lives are becoming more “real-time,” whether we like it or not. Just as Google and Microsoft’s Bing are upgrading their search indexes to make them more real timeby capturing things as they occur, instead of hours or even days later, we are being forced to upgrade our internal processes to do the same thing. But doing that isn’t quite as simple as tinkering with a search algorithm — we have to find ways of managing the real-time demands placed on us while still maintaining something approaching a healthy personal life, something Stacey wrote about a little while ago. How do we handle the demands of our our spouses, our children, our relatives and friends? How do we maintain our health when we are always on, always available, in real time? If apps like SnapChat and Highlight are any indication, then the realtime nature of the Internet isn’t going away and in fact it is going to become more pervasive. What would be really cool is an individual “off switch” for these real time apps. In one simple toggle of a virtual button, email is off — like my stereo is when I power it down. A flick of the switch turns off Twitter. You get the idea. Why? Because I do like to use these real-time apps during parts of the day, especially when I am working. However, I don’t want to deal with them at night or say when I am hanging out with friends or when at dinner. I don’t know about others, but I would welcome that “off switch.” Turn off tweets till the dinner is done and switch it on when ready to jump into the real-time information stream. I think taking that break is becoming more and more essential. http://dlvr.it/2ljh9n @suryaray

[sadache]

Is socket.push(bytes) all what you need to program Realtime Web apps?

**One of the goals of [Play2](www.playframework.org) [architecture](http://sadache.tumblr.com/post/26258782102/bitsbout-play2-architecture) is to provide a programming model for what is called Realtime Web Applications.** ### Realtime Web Applications Realtime Web Applications are applications making use of Websockets, Server Sent Events, Comet or other protocols offering/simulating an open socket between the browser and the server for continuous communication. Basically, these applications offer to users delivery of information as it is published without having the user periodically pinging the service. There are quite a few web frameworks that target the development of this type of applications. Mostly, however, the solution is by providing an API that allows developers to push/receive messages from/to an open channel, something like: channel.push(message) //and channel.onMessage { ... } Though this kind of API offers an opportunity to get started doing Realtime Web, it doesn't offer a _programming model_ for dealing with the challenges encountered when programming with streams of data, including creating, adapting, manipulating, filtering and merging streams of data and all of the synchronization involved. ### A Programming Model Since development of Realtime Web Apps is mostly built around manipulation of streams of data, it is crucial to have a programming model which identifies clearly what a stream of data is and defines composable components to deal with it. It is obvious that the above channel api falls short when manipulating a rich set of streams of data, but even the classic InputStream/OutputStreams interfaces are not sufficient. Forgetting about their inefficient blocking runtime properties, they don't carry enough signals/information to allow building rich stream manipulation api. [Play2](www.playframework.org) uses Iteratees together with Futures for dealing with streams of data, providing a very rich model for programming rich Realtime Web Applications. ### A sample Realtime Web App The goal of this text isn't to provide a detailed description of what Iteratees and pals are. Nevertheless I will go into a fast introduction and then move into an example illustrating few aspects of how powerful this approach is. - An `Iteratee[E,A]` is an immutable interface that represents a consumer, it consumes chunks of data each of type `E` and eventually produces a computed value of type `A`. `Iteratee[String,Int]` is an iteratee that consumes chunks of strings and eventually produces an `Int` (that could be for instance number of charecters in the passed chunks) An iteratee can choose to terminate before the `EOF` sent from the stream, or could wait for `EOF` before it terminates, returning the computed `A` value. You can compose different `Iteratee`s together and this can be an opportunity for partitioning consuming logic into different parts. - An `Enumerator[E]` represents a stream that is pushing chunks of data of type `E`. An `Enumerator[String]` is a stream of strings. `Enumerator`s can be composed one after the other, or interleaved concurrently providing means of streams management. - An `Enumeratee[From,To]` is an adapter from a stream of `From`s to a stream of `To`s. Note that an `Enumeratee` can rechunk differently, add or remove chunks or parts of them. `Enumeratee`s as we will see are instrumental for stream manipulation. - There are convenient methods for creating different kinds of `Enumerator`s `Iteratee`s and `Enumeratee`s for different scenarios. Our sample application features two streams, one is a stream of financial operations: val operations: Enumerator[Event] = Enumerator.generateM[Event] { Promise.timeout( Some(Operation( if(Random.nextBoolean) "public" else "private", Random.nextInt(1000))), Random.nextInt(500)) } Here we are generating random values at random distances (of maximum 500ms). In the real world this stream could be coming from a datastore or an open socket with another server. An `Operation` is _private_ or _public_ and curries an _amount_ of type `Int`: case class Operation(level: String, amout: Int) extends Event The other stream we have is a stream of system messages, messages that talk about the status of the system: case class SystemStatus(message: String) extends Event val noise: Enumerator[Event] = Enumerator.generateM[Event] { Promise.timeout( Some(SystemStatus("System message")), Random.nextInt(5000) ) } This stream can be coming from another server or datastore. With these two streams at hand, we can prooduce one single stream that contains messages of both by interleaving them: val events: Enumerator[Event] = operations >- noise Actually those not comfortable using symbolic operators can use `interleave` method: val events: Enumerator[Event] = operations.interleave(noise) Now the model part of our application looks like: ### Our sample Realtime Web App features: Our application will publish this stream of `Event` as [Server Sent Event](http://dev.w3.org/html5/eventsource/) or [Comet](http://en.wikipedia.org/wiki/Comet_(programming)) (both are protocols for uni-directional socket from the server to the browser) and will be providing two simple features: #### Authorization: You can only see `Event`s that are allowed for your role. _Managers_ can see _private_ operations and system status whereas normal _users_ can see only _public_ operations, for this purpose we create an `Enumeratee` which collects appropriate messages: def feed(role: String) = Action { val secure: Enumeratee[Event, Event] = Enumeratee.collect[Event] { case e@SystemStatus(_) if role == "MANAGER" => e case e@Operation("private", _) if role == "MANAGER" => e case e@Operation("public", _) => e } #### Filtering: You can filter the stream by range of interest in the amount of the operation. By providing an _upper_ and _lower_ bounds you get only corresponding operations, for this we create another `Enumeratee` collecting appropriate operations: def feed(role: String, lowerBound: Int, higherBound: Int) = Action { val secure: Enumeratee[Event, Event] = ... val inBounds: Enumeratee[Event, Event] = Enumeratee.collect[Event] { case e@Operation(_, amout) if amout > lowerBound && amout < higherBound => e case e@SystemStatus(_) => e } } #### JSON: Our App will be pushing JSON messages to the browser, that's why we need one more `Enumeratee` for transforming `Event`s to `JSON` values, ie, `Enumeratee[Event,JsValue]`: val asJson: Enumeratee[Event, JsValue] = Enumeratee.map[Event] { case Operation(visibility, amount) => toJson(Map("type" -> toJson("operation"), "amount" -> toJson(amount), "visibility" -> toJson(visibility))) case SystemStatus(msg) => toJson(Map("type" -> "status", "message" -> msg)) } For convenience, let's produce one single adapter out of the three we got, for that we can use the `compose` method or its symbolic equivalent `><>`: val finalAdpater = secure ><> inBounds ><> asJson We're almost done. Now all what we need is to respond to the browser with an `Ok` status wrapping each message into the Server Sent Event (`EventSource`) protocol. Actually there is already an `Enumeratee` which adapts a stream and wraps its chunks into the SSE protocol: Ok.feed(Stream.events &> finalAdpater ><> EventSource()) .as("text/event-stream") Here we pass our stream through the `finalAdapter` and then through the `EventSource` adapter, applying then the appropriate header to the response. Our application now looks like: All what we need now from the client side is to connect to the stream using the following javascript: feed = new EventSource('/feed?role=@role&lower=' + min + '&higher=' + max) Snapshots of our Realtime Web App: /manager  /  ### Bottom Line Realtime Web involves dealing with different streams of data from different sources. It is hard to do any non-trivial application without having a programming model that contains an appropriate representation of a stream and necessary API for creating, adapting, filtering and consuming streams of data. Play2 uses `Iteratee`s to offer a programming model and a rich API. *Note: [Full working application source](https://github.com/sadache/RealtimeWebAppSample) was created by me @sadache and @guillaumebort for one of our talks* *Note2: [Probably a more readable version of this post](https://gist.github.com/3072893)* *Note3: Examples of Realtime Web Apps using Play2 [Typesafe Console](http://console-demo.typesafe.com) [lichess.org](http://en.lichess.org/)*

[jgwmaxwell]

Building Chatter: Part 6 - Welcome to the Realtime

So, we have the barebones of our application up and running, but it wouldn't be much good if you had to constantly refresh the page to see what people are talking about. When I create a post, I want anybody who I mention to know that I've mentioned them immediately, and if I am searching for a topic, then I want any new messages about that topic to be delivered to me immediately.

We will set up the ability to do this in two stages - first we will look at mentions "@username" in a post, and then we will look at searches.

Live feeding the @users.

Detecting @user tags

Delivering the message

Improving the application

Ok, so we need a system for detecting the tags in the posts as they are submitted. To do this, we are going to open up our Post model again, and make some changes. Processing notifications isn't really something that a post needs to be aware of, that is something that happens in the background, and because of this, Rails provides us with a type of "listener", called an Observer. Observers monitor a model, and carry out actions whenever a callback would have been fired. This means that we can separate our concerns - the Post model describes the posts, and a PostObserver will monitor this and start the communication system.

If you need a copy of the application so far, remember it is on Github, once you've cloned it then "git checkout part5" to bring you up to date.

Let's begin.

The first thing we need to do is generate our Observer. In a terminal at the root of your application, run:

$ rails g observer Post

Now, go to "app/models/post_observer.rb" and we are going to implement some of the functionality that we already had in our Post model:

class PostObserver < ActiveRecord::Observer observe Post def after_commit(post) TAG_PROCESSOR.push(:tag_class => Tag, :post_id => post.id) end end

We can now remove some lines from "app/models/post.rb":

delete - after_commit :process_tags delete - def process_tags delete - TAG_PROCESSOR.push(:tag_class => Tag, :post_id => self.id) delete - end

Finally, to get this observer observing, we need to update a line in "config/application.rb":

# Activate observers that should always be running. config.active_record.observers = :post_observer

If we were to restart the server now, then we would see that our previous functionality still works, but it now located in a more sensible and logical place. The task at hand now is to work out how we want users mentioned in posts to interact with them. The most simple method, and the one we will use, is to create another relationship between users and posts, however you could look at serializing a field, or a number of other tricks. We are going to use another "has_many, :through =>" type relationship, this time called Mention:

$ rails g model mention post_id:integer user_id:integer

And then in the migration file for this, (in "db/migrate") we need to add a few things to create the indexes for the table. Add the following below the change method block:

add_index :mentions, :post_id add_index :mentions, :user_id

Now we can migrate the database ("rake db:migrate") again, and then set up the relationship in our models. In "app/models/user.rb" we need to add the following, probably near your other relationship declarations ("belongs_to", "has_many" etc).

has_many :mentions has_many :mentioned_ins, :through => :mentions, :source => :post

We also need to add something similar to our Post model in "app/models/post.rb":

has_many :mentions has_many :mentionings, :through => :mentions, :source => :user

Finally we need to add a couple of lines to the Mention model in "app/models/mention.rb":

belongs_to :user belongs_to :post

Perfect. We've got the join model set up properly, so we can go ahead and detect usernames in the posts as they come in now.

Detecting Mentions

To detect the mentions, it's actually going to be very easy, as we have done most of the hard work already! The tool that we need we made earlier, so open up "app/models/post_observer.rb" and make the following changes:

class PostObserver < ActiveRecord::Observer observe Post def after_commit(post) TAG_PROCESSOR.push(:post_id => post.id) users = User.find_all_by_username(detect_usernames(post.body)) if users users.each do |u| u.mentioned_ins << post end end end private def detect_usernames(post) post.scan(/B@(w*[A-Za-z0-9_]+w*)/).flatten end end

This now will find all the usernames in the body of the post, and then for each user it finds that corresponds, it will add a mention for them. The reason we aren't delaying this and using Girl Friday again is that we want these notifications to be absolutely realtime - if someone says my name I don't want to wait to hear it, whereas if the notifications about topics arrive a second or two later when the service is busy, that isn't the end of the world (although obviously that isn't desirable).

Let's spin up the server and create a new post that mentions another (existing) user. To do this, we use "foreman start", if you've forgotten since last time.

So, we have created a new post with (in my case) "@billgates" in there, which means that our observer should have picked this up and created a new record in the mention model for us. We can check this by going to the terminal and typing:

$ rails c

Which fires up a command line IRB session, with our Rails environment fully loaded in it. If you aren't familiar with the rails console - spend some time playing around, you can do an amazing amount of stuff, and it serves as a phenomenal test-bed for experimentation.

> Mention.first

You'll see the console perform a database query, and come up with a Mention, linking your last post with the user you tagged in it! Perfect. We can just double check that it worked in both directions, by typing (in my case):

> User.find_by_username("billgates").mentioned_ins

And you should see the post that you mentioned the user in. Type "exit" to get out of the console, and we'll get on with the next steps. Now we have a system for detecting when we have tagged a person, we need to start implementing the "realtime" part of the application.

Realtime Rails

Rails, and Ruby in general, isn't particularly good at "realtime" web applications. This is due to something called the GIL or Global Interpreter Lock, which massively hampers the amount of concurrency (things done at once) that Ruby can cope with. However, this situation is improving, and often being bound by a constraint like the GIL forces the developer to come up with a cleverer solution to the problem.

There are a lot of "push api" providers - companies that offer a product to abstract this functionality away from the application, and to be honest in most cases using them is a good idea. In a later tutorial we will revisit this next section, and show you how to implement your own push server using something called Faye, but this is a little more complex that we are going to deal with yet. We're going to use a really solid service - PubNub that offers unlimited free testing whilst in development mode. A caveat here - to do this, you have to use a shared "demo" key, which means that anyone could potentially listen in to your messages, so don't transmit anything too important!

To use PubNub, we're going to add a gem to our Gemfile:

gem "pubnub-ruby"

And then we need to run "bundle" to get that installed. We also will create a new initializer file - "config/initializers/pubnub.rb":

PUBNUB = Pubnub.new( "demo", ## PUBLISH_KEY "demo", ## SUBSCRIBE_KEY "", ## SECRET_KEY false ## SSL_ON? )

We will also need another observer, so run "rails g observer Mention" and then open the file it created ("app/models/mention_observer.rb").

class MentionObserver < ActiveRecord::Observer observe Mention def after_commit(mention) end end

Before we go any further, we need to know how PubNub (and almost all push notification clients) work, so we can make a design decision for our application. The PubNub client that runs in the browser listens on what is called a "channel" - we need to know what channel we are sending our messages out on, in order to be sure that the right message is going to the right people. Rather than simply using the username, we will do something slightly more advanced.

The Message Queue

We aren't going to use a proper message queuing system such as AMQP or RabbitMQ - those are far beyond our needs, but we will need a system for managing our "live" channels. This will become more important when we bring in push functionality for tags as well.

We are going to create another model to hold this data, and we will call it Channel. It is going to need a number of fields, to allow us to access it and to provide some control over how long it keeps broadcasting for. This Channel model is going to be polymorphic - which means we can attach it to a number of other models by declare two fields an _id and a _type, prefixed with the polymorphic identity, which in our case will be broadcastable:

broadcastable_id - the id of the related model

broadcastable_type - the class of the related model

channel_ident - a string to identify the channel

last_validated - a timestamp for when the channel was last declared to be needed

So, lets generate this model:

$ rails g model channel broadcastable_id:integer broadcastable_type:string channel_ident:string last_validated:datetime

Add this line underneath the create_table block in the migration that produced:

add_index :channels, [:broadcastable_id, :broadcastable_type]

You can now run "rake db:migrate" to add this table. Then we need to add some code to a few models:

# in 'app/models/channel.rb' belongs_to :broadcastable, :polymorphic => true # in 'app/models/user.rb' has_one :channel, :as => :broadcastable # in 'app/models/tag.rb' has_one :channel, :as => :broadcastable

Ok, so now we have a mechanism for controlling our channels, lets go back to MentionObserver ("app/models/mention_observer.rb") and finish creating it. Change the after_commit method to the following:

def after_commit(mention) NOTIFY_QUEUE.push(:post_id => mention.post_id, :user_id => mention.user_id) end

You might recognise this as another Girl Friday method, and earlier I said that we didn't want to wait for this to happen. There is a good reason though why we need to at this stage. The call to send the post to PubNub to send to the user requires a call to an external server, and we don't know how long this will take to respond. We don't want to block up our whole application if PubNub is a bit slow in getting back to us, so at this point we need to call on this web service behind the scenes. Let's add another processing action into Girl Friday - edit "config/initializers/girl_friday.rb" and change as so (note the subtle change to the TAG_PROCESSOR as well):

TAG_PROCESSOR = GirlFriday::WorkQueue.new(:post_process, :size => 2) do |msg| Tag.process_tags(msg[:post_id]) end NOTIFY_QUEUE = GirlFriday::WorkQueue.new(:post_process, :size => 2) do |msg| Notify.deliver_message_to_user(msg) end

You'll note here that we are calling a method on a class called Notify. We haven't written that yet, so lets create a new file in the models folder called "notify.rb", and fill it with the following:

class Notify def self.deliver_message_to_user(params) post = Post.find(params[:post_id]) user = User.find(params[:user_id]) user.channel ||= Channel.new( :channel_ident => Digest::SHA1.hexdigest(user.username, user.created_at.to_s)) PUBNUB.publish({ 'channel' => user.channel.channel_ident, 'message' => post.to_json(:include => :user) }) end end

Now, we are nearly ready to test this out. We just need to edit a couple of client side files, and we are good to go. First, we need to add the following to "app/views/dashboard/index.html.erb", just at the bottom of the file.

<script> // ------------------- // LISTEN FOR MESSAGES // ------------------- PUBNUB.subscribe({ channel : "<%= Digest::SHA1.hexdigest(current_user.username, current_user.created_at) %>", callback : function(message) { alert(message) } }) </script>

Then we need to add a couple of bits to "app/views/layouts/application.js", insert these directly before the "<%= yield %>" line:

<div id=pubnub pub-key=demo sub-key=demo ssl=off origin=pubsub.pubnub.com ></div> <script src=http://cdn.pubnub.com/pubnub-3.1.min.js ></script>

We also need to add the MentionObserver to "config/application.rb" - so change that line so that it reads:

config.active_record.observers = :post_observer, :mention_observer

Ok, we are going to need two terminal windows for this bit, and a web browser. Fire up the webserver in one terminal, and a console in the other ("foreman start", and "rails c" respectively). Then, go to the home page in a web browser, and - leaving the browser visible in front of you, type the following into the console, replacing "@USERNAME" with the username of the user you are signed in as.

Post.create(:body => "hello there @USERNAME")

You should be able to see a Javascript alert window pop up with the content of the post that you just created, fired to you in Realtime. Next time we'll go over how to make this a bit smarter and display the post, and we'll be using something called Handlebars.js to make the whole of the interface a lot snappier.

If anyone has any questions, please feel free to ask away, and as ever, the source code is available on Github, this time in the "part6" branch.

[sambervalley]

This article showed you five reasons why you should be using WebSockets for your web applications. You learned that HTML5 WebSockets can enable true full-duplex communication, can handle large numbers of concurrent users, can extend TCP-based protocols to the browser, is easy to use, and can extend your SOA over the Web and in the cloud.

[amzam]

Twitter doesn’t replace any other form of media or journalism, any more than YouTube replaces television, or Facebook replaces the need for normal human interaction. Twitter is just a tool, like the telephone or the video camera — it doesn’t replace the need for traditional journalists.

Amen: No, Twitter Is Not a Replacement For Journalism Tech News and Analysis

[amzam]

Live is not ‘yet another thing’ for a working journalist to understand , it is the great journalistic challenge of our time. The skill involved in providing real time valuable information for audiences around stories as they happen is crucial to being a credible journalist and a resilient news organisation.

Journalism professor Emily Bell: Real time, All the time: Why every news organisation has to be live (h/t Terry Heaton)

[amzam]

The “tweet first, verify later” approach is a great help for source diversification and leads to richer coverage. But this strategy also seems very dangerous for one of journalism’s golden rules: each news story must be verified first.

Interesting research from Nicola Bruno on how the 'Twitter effect' is impacting news gathering (for better or worse) // How Social Media Creates a Rough Draft of History: Tech News and Analysis «