#tempodb

[e-verythingy]

Visualizing a TempoDB Data Series with Cubism.js

Cubism.js is a D3.js plugin for time series visualization which seems like a perfect match for displaying the time series data that we are storing in TempoDB. The result of the visualization looks as follows:

Cubism pulls the initial data set to render the graph and then fetches time series data incrementally as it scrolls to the left as time passes.

While TempoDB offers a ReST API to read and write data, every request - even read requests - requires authentication with the user's API key and API secret.

If authentication for read requests were not required, Cubism could pull the data directly from the TempoDB data API and take advantage of all its built-in features (e.g. rollups, dataset statistics, timezones, intervals). So TempoDB, please add optional non-authenticated public read-requests to your feature set!

As the read requests are made from D3/Cubism in the browser, we can't handle the API key/secret in the client-side JavaScript code either. So we need build our own lightweight data API/proxy that fetches the result set from TempoDB and provides it to D3/Cubism. The Ruby/Sinatra code can be found here on GitHub. A more detailed description which I used largely as the basis and only slightly adopted for this project can be found over here at iwantmyreal.name (Thanks!)

Cubism expects one data point per pixel when it renders the chart. Note that it renders the data points from left to right, not honoring any timestamps. In case of discontinuous data, the chart will end early and a gap shows up on the right. This can be observed in above graph where a total of ~2 hours worth of data is missing in the 24-hour data set. Cubism starts painting one data point per pixel, from left to right, until it runs out of values.

There are three options to prevent this:

Fill in missing values at the time the data values are read from the sensor, or written into TempoDB. Not really a good idea...

Interpolate and fill in missing values at the time the result set is retrieved from TempoDB, made available through the API/proxy, or received at the client-side. This approach is discussed here server-side and here client-side.

Make the visualization component timestamp-aware and render the data points according to their timestamps, and keep blanks whenever there is no data value available. 

The latter would be my preferred approach, especially since TempoDB stores timestamp/value pairs already. Something to look into sometimes later.

The demo is deployed to Heroku and its full client-side and server-side code can be found on GitHub.

 References:

https://github.com/square/cubism

https://tempo-db.com/

http://iwantmyreal.name/blog/2012/09/16/visualising-conair-data-with-cubism-dot-js/

http://xaranke.github.io/blog/cubism-intro/

[e-verythingy]

"Streaming" MQTT Sensor Data into TempoDB

We want to store sensor data from Arduino or any other device in a time-series database like TempoDB. TempoDB offers ReST APIs that we could use to write the data series from Arduino directly into TempoDB. But that would be too easy ;)

In fact, we don't want each and every device to directly access the database. Instead, want a more generic solution that leverages the MQTT messaging protocol and the concept of message brokers.

Our Arduino publishes its sensor values already to an MQTT broker.

So how do we get the time series from CloudMQTT (or any other MQTT broker) into TempoDB?

As TempoDB does not provide a native MQTT interface (yet?), we will build a listener that is subscribed to a MQTT broker, and upon receipt of a new message, it simply writes the message into TempoDB.

The listener is based on the Ruby MQTT pub/sub client and the Ruby TempoDB client (clients for other languages are available, too): 

https://gist.github.com/6790192

Deployed to Heroku as a web app:

Clone the listener web app from GitHub to your local machine.

Follow the Getting Started with Ruby on Heroku instructions to install the necessary gems and familiarize yourself with app deployment to Heroku.

Deploy the web.rb app from the cloned Git repository with git push heroku master

Start the app with heroku open

Publish MQTT messages and use TempoDB's console to verify that the values are indeed stored.

Note: In Heroku's free plan, the Dyno starts hibernating after 2 hours of inactivity (the listener's activity does not count as it is not receiving HTTP requests). As you want the sensor values to be continuously stored, you would need to make sure to keep the Heroku app alive. One way is to add a second web dyno (monthly charges). Another one is to be a bit creative (hint: cron, curl)

[e-verythingy]

Storing Time Series of Sensor Values in the Cloud with Heroku and TempoDB

Our temperature sensor produces a time series of values.

We want to store this time series in a database.

What if the database would natively support time series? The answer is TempoDB, a "simple, scalable, fast time series database service purpose-built to store & analyze time series data from sensors, smart meters, servers & more".  It's a cloud-based Database-as-a-Service (DaaS, anyone?), and it's available as an add-on to Heroku.

How to add it to Heroku:

Log in to Heroku and select your app (see previous post).

Add the TempoDB add-on.

Done!

Run heroku config in the console to retrieve the authentication and connection details. You can use these to write/read data through TempoDB's API and clients.

Note: If added through Heroku, there is no need to sign-up on their website. I did and it took me a half day to figure out why it rejected my API calls... I used the website's set of credentials, and not the ones provided via Heroku!

That was easy. Next up: Getting the sensor values into TempoDB.

[plainblackguy]

TempoDB is a database service purpose-built to store and analyze massive streams of time series data.

Looks pretty cool. I think I'm going to explore using it to store trends data.