Pepper Talk Messaging Backend
We keep getting asked often what is our backend architecture, are we XMPP/MQTT based, how we scale up etc. This post will attempt to address most of these questions and give a brief insight into how we have desgined and built Pepper Talk.
We started out Pepper Talk as a messaging solution primarily for mobile devices. Consumers today are used to mobile messaging from standalone apps which provide a very sophisticated user experience. We wanted to provide this same level of experience on all apps via the Pepper Talk SDK. Our primary goals
Fast message delivery
Multiple device support
Extensible / Custom data support
Message delivery status
Platform specific remote notifications
Ownership of chat data
Well we decided to cook something up on our own, a persistent WebSocket connection with messages as simple JSON/ProtoBuf structure. All messages have a uuid and are acknowledged from receiving end. The WebSocket endpoint itself is authenticated via standard HTTP mechanisms. Users can register devices for their account and messages are queued for each registered device for the user. Messages are held on the server queues for a limited time and removed on succesful delivery or expiration. User devices connection status is tracked, disconnected devices are notified of messages via the device specific notification protocols (APNS / GCM).
We provide iOS, Android and Java Script SDK's. A REST API is also available for automating tasks from server side. The SDK's implement the network protocol, data storage and UI for the chat interface. In addition to the chat interface we transport any json packet in the messages. These are available as custom data extensions on the SDK and callbacks are triggered whenever a custom data packet is received. Your app can decide on what data to transport and how to render, we act as a data pipe in this scenario.
Another key aspect to the infrastructure is the message stream. Often apps want to hold the chat history for regulatory purposes, analytics, personalization, data mining and many more reasons. We enable you to tap into the real-time stream of messages for your app as the messages flow through the system. All key events, ie; message sending, delivery, read on a message are notified. The stream is delivered over a HTTP request similar to the Twitter Firehose.
The backend is a shared infrastructure with logically separated apps. Once you sign up for Pepper Talk you can create any number of apps under your account. Each app is sandboxed and messages can be exchanged only within the users or groups of the app. Users of the app are auto provisioned by the SDK on first connect. Apps are responsible for authenticating their users. Once the app has an authenticated user, the SDK can be initialized for that particular user. The SDK in turns creates a persistent connection on behalf of the app for this user. All messages intended for the user are then delivered on this connection. Messages from this connection are sent as the connected user.
The backend is implemented in Erlang, with Cowboy, and OAuth2 for WebSocket and OAuth support. A Redis based name registry is used for global name lookups using the Erlang via name registry. MongoDB is used for temporary data storage. The backend is horizontally scaleable with each node in the cluster being stateless. A node which is part of the cluster can service a user connection from any app. A user connection lands on one of the nodes via the load balancer. The Erlang process servicing this connection is managed by the process managing the user. Messages arriving on this connection are deserialized and processed. Messages are routed to the appropriate user process and inturn to all the connections under that user. Groups follow a similar hierarchy where messages for a group are delivered to all users who are part of the group. In case the user process does not have any active connection for a particular device for the user the notification mechanism for the device notifies a queued message. When the device next comes back online queued messages are delivered to the device.
This is a very high level overview of how things operate in Pepper Talk. I do intend get into more details on how the individual pieces fit together in later posts, and feedback is welcome.