The Internet of E.verything

E.verything from Arduino to MQTT.

[e-verythingy]

IoT Investment Opportunities

This page moved to a permanent location to keep IoT investment opportunities up-to-date - http://e.verything.co/IoT_Investment

[e-verythingy]

Visualizing a TempoDB Data Series with Cubism.js

Cubism.js is a D3.js plugin for time series visualization which seems like a perfect match for displaying the time series data that we are storing in TempoDB. The result of the visualization looks as follows:

Cubism pulls the initial data set to render the graph and then fetches time series data incrementally as it scrolls to the left as time passes.

While TempoDB offers a ReST API to read and write data, every request - even read requests - requires authentication with the user's API key and API secret.

If authentication for read requests were not required, Cubism could pull the data directly from the TempoDB data API and take advantage of all its built-in features (e.g. rollups, dataset statistics, timezones, intervals). So TempoDB, please add optional non-authenticated public read-requests to your feature set!

As the read requests are made from D3/Cubism in the browser, we can't handle the API key/secret in the client-side JavaScript code either. So we need build our own lightweight data API/proxy that fetches the result set from TempoDB and provides it to D3/Cubism. The Ruby/Sinatra code can be found here on GitHub. A more detailed description which I used largely as the basis and only slightly adopted for this project can be found over here at iwantmyreal.name (Thanks!)

Cubism expects one data point per pixel when it renders the chart. Note that it renders the data points from left to right, not honoring any timestamps. In case of discontinuous data, the chart will end early and a gap shows up on the right. This can be observed in above graph where a total of ~2 hours worth of data is missing in the 24-hour data set. Cubism starts painting one data point per pixel, from left to right, until it runs out of values.

There are three options to prevent this:

Fill in missing values at the time the data values are read from the sensor, or written into TempoDB. Not really a good idea...

Interpolate and fill in missing values at the time the result set is retrieved from TempoDB, made available through the API/proxy, or received at the client-side. This approach is discussed here server-side and here client-side.

Make the visualization component timestamp-aware and render the data points according to their timestamps, and keep blanks whenever there is no data value available. 

The latter would be my preferred approach, especially since TempoDB stores timestamp/value pairs already. Something to look into sometimes later.

The demo is deployed to Heroku and its full client-side and server-side code can be found on GitHub.

 References:

https://github.com/square/cubism

https://tempo-db.com/

http://iwantmyreal.name/blog/2012/09/16/visualising-conair-data-with-cubism-dot-js/

http://xaranke.github.io/blog/cubism-intro/

[e-verythingy]

"Streaming" MQTT Sensor Data into TempoDB

We want to store sensor data from Arduino or any other device in a time-series database like TempoDB. TempoDB offers ReST APIs that we could use to write the data series from Arduino directly into TempoDB. But that would be too easy ;)

In fact, we don't want each and every device to directly access the database. Instead, want a more generic solution that leverages the MQTT messaging protocol and the concept of message brokers.

Our Arduino publishes its sensor values already to an MQTT broker.

So how do we get the time series from CloudMQTT (or any other MQTT broker) into TempoDB?

As TempoDB does not provide a native MQTT interface (yet?), we will build a listener that is subscribed to a MQTT broker, and upon receipt of a new message, it simply writes the message into TempoDB.

The listener is based on the Ruby MQTT pub/sub client and the Ruby TempoDB client (clients for other languages are available, too): 

https://gist.github.com/6790192

Deployed to Heroku as a web app:

Clone the listener web app from GitHub to your local machine.

Follow the Getting Started with Ruby on Heroku instructions to install the necessary gems and familiarize yourself with app deployment to Heroku.

Deploy the web.rb app from the cloned Git repository with git push heroku master

Start the app with heroku open

Publish MQTT messages and use TempoDB's console to verify that the values are indeed stored.

Note: In Heroku's free plan, the Dyno starts hibernating after 2 hours of inactivity (the listener's activity does not count as it is not receiving HTTP requests). As you want the sensor values to be continuously stored, you would need to make sure to keep the Heroku app alive. One way is to add a second web dyno (monthly charges). Another one is to be a bit creative (hint: cron, curl)

[e-verythingy]

Storing Time Series of Sensor Values in the Cloud with Heroku and TempoDB

Our temperature sensor produces a time series of values.

We want to store this time series in a database.

What if the database would natively support time series? The answer is TempoDB, a "simple, scalable, fast time series database service purpose-built to store & analyze time series data from sensors, smart meters, servers & more".  It's a cloud-based Database-as-a-Service (DaaS, anyone?), and it's available as an add-on to Heroku.

How to add it to Heroku:

Log in to Heroku and select your app (see previous post).

Add the TempoDB add-on.

Done!

Run heroku config in the console to retrieve the authentication and connection details. You can use these to write/read data through TempoDB's API and clients.

Note: If added through Heroku, there is no need to sign-up on their website. I did and it took me a half day to figure out why it rejected my API calls... I used the website's set of credentials, and not the ones provided via Heroku!

That was easy. Next up: Getting the sensor values into TempoDB.

[e-verythingy]

Bridging two MQTT brokers

Messages containing sensor values that are published to the local MQTT broker (Mosquitto) should also be available for subscription on the remote MQTT broker (CloudMQTT).

       This can be achieved with "bridges", a concept that MQTT servers provide. The bridge between Mosquitto and CloudMQTT can be established as followsSSH into the local server and edit the Mosquitto config file: nano /etc/mosquitto/mosquitto.conf

Add the following lines:

https://gist.github.com/6790584

Restart mosquitto (sudo systemctl restart mosquitto.service)

Test the connection with mosquitto_sub -h broker.cloudmqtt.com -p <port> -u <username> -P <password> -t arduino/temperature

Note: Get port/username/password for CloudMQTT by running the following command in a console: heroku config

To debug, systemctl status mosquitto.service may be helpful.

  References:

http://mosquitto.org/man/mosquitto-conf-5.html

[e-verythingy]

Operating a Cloud-based MQTT Broker with Heroku and CloudMQTT

Today, we will connect our home network to the outside world, MQTT-style:

Sign up for free on Heroku. Heroku is a cloud-based Platform as a Service (PaaS), which basically means that you can host your own applications in the cloud. 

Create a new app (click "Create new App"). It's free as long as you only use one web dyno.

Add the CloudMQTT add-on (beta). As the name suggests, CloudMQTT is a cloud-based MQTT pub/sub broker, just like Mosquitto. Only in the cloud.

Click on the newly installed add-on within the "Add-ons" section in your Heroku Dashboard. It'll show you the username/password as well as the endpoint and port for the CloudMQTT instance.

You can use these connection details to connect to CloudMQTT and publish/subscribe topics with any MQTT client.

So far, so good. Next, we'll create a bridge between the local Mosquitto broker and the remote CloudMQTT instance.

[e-verythingy]

Connecting the Home Server to the Outside World

Changing router settings to allow and forward inbound MQTT messages to the Mosquitto broker:

Enable port forwarding in the firewall settings of your router for TCP port 1883 (the MQTT default port) and TCP port 8883 (for MQTT over SSH).

Test the connection:

Subscribe on internal IP: mosquitto_sub -h my_local_ip -p 1883 -t my_topic

Publish through external IP: mosquitto_pub -h my_external_ip -p 1883 -m 'Hello World' -t my_topic

[e-verythingy]

Publishing Arduino Sensor Data through MQTT over Ethernet

Now that we have our home server with Archlinux and Mosquitto running, let's connect an Arduino and publish some sensor data.

You need:

Arduino Uno (Edit: or the new Arduino Yún with built-in Ethernet/WiFi)

Ethernet Shield

I've attached a temperature sensor to the Arduino. Adafruit provides a good instructions on how to do this in general.

The following sketch is based on the MQTT client library for Arduino which provides pub/sub capabilities. Open the sketch in the Arduino API and upload it.

https://gist.github.com/justx1/6790670

The temperature is published through the client's publish method client.publish("arduino/temperature",tempC); Note that the variable tempC has to be converted to char first.

The Arduino is now publishing the temperature to the Mosquitto broker in regular intervals.

To verify that it's working, open a console, and subscribe to the topic: mosquitto_sub -h ip_of_mqtt_broker -t arduino/temperature

[e-verythingy]

#MQTTinside

MQTTinside stickers just arrived.

Get yours for your MQTT-enabled devices at http://mqtt.org/goodies 

[e-verythingy]

Arduino + Bluetooth LE = BLEduino

Everything you know and love about your Arduino, but with Bluetooth 4.0 (BLE). Small, sexy, and fully compatible with Arduino shields.

$34 on Kickstarter.

[e-verythingy]

Texas Instruments Bluetooth Smart SensorTag

The Texas Instruments Bluettoth Smart SensorTag Development Kit ships for 25$ and looks quite interesting to explore Bluetooth 4.0 connectivity. It includes sensors for temperature, humidity, pressure, as well as accelerometer, gyroscope and magnetometer.

The SensorTag Wiki page showcases some use cases (e.g. measuting temperature with the IR temp sensor) and provides links to code resources, iOS apps and projects.

[e-verythingy]

Arduino + 802.15.4 = Pinocc.io

Looks promising - It's an "Arduino Mega with Wings": Pinocc.io

A small wireless Arduino-compatible microcontroller. Wireless communication via WiFi and mesh radio. An API for buiding web-enabled DIY hardware projects. An open-source hardware platform for makers.

[e-verythingy]

Installing an MQTT Server on an Archlinux-powered Pogoplug

MQTT is an open message protocol for M2M communications that enables the transfer of telemetry-style data from pervasive devices to a MQTT message broker. Pervasive devices may range from sensors and actuators, to mobile phones, embedded systems on vehicles, or laptops and full scale computers. The protocol was invented by Andy Stanford-Clark of IBM, and Arlen Nipper of Cirrus Link Solutions.

Let's install an open source MQTT message broker (Mosquitto) on our Archlinux Pogoplug:

Change into /tmp directory (cd /tmp).

Download the package (wget https://aur.archlinux.org/packages/mo/mosquitto/mosquitto.tar.gz). More info.

Unzip it (tar -zxvf mosquitto.tar.gz).

Change directory (cd mosquitto).

Compile the package (sudo makepkg -s).

Install Mosquitto (sudo pacman -U mosquitto-1.1.3-3-arm.pkg.tar.xz).

Enable systemd to start the service automatically on startup (sudo systemctl enable mosquitto.service).

Start Mosquitto (sudo systemctl start mosquitto).

Done.

Mosquitto is now up and running and will automatically start after reboots. Let's play around:

Open a new console, and ssh into the Archlinux Pogoplug.

Console #1: Subscribe to a topic (mosquitto_sub -d -t hello/world).

Console #2: Publish a message (mosquitto_pub -d -t hello/world -m "Hello, MQTT. Hello, World.").

Play around publishing and subscribing to topics and watch the output on the consoles.

Congratulations to your new MQTT broker! Next up: We'll teach Arduino to talk to Mosquitto.

  References:

http://diabolicalws.blogspot.com/2012/02/mosquitto-on-pogoplug.html

[e-verythingy]

Installing Archlinux on the Pogoplug Home Server

Get a Pogoplug and use it as your low-cost, low-energy 24/7 home server to power your personal Internet of E.verything.

Get a Pogoplug. Make sure its model number is E02.

You'll also need a USB memory stick.

Install Archlinux. Follow the excellent installation instructions here.

First thing, change the root password (passwd root).

Update the system (pacman -Suy).

Install the development package (pacman -S base-devel). More info.

Update the kernel (pacman -Sy linux-kirkwood linux-headers-kirkwood; then answer all questions with yes) and reboot the machine (reboot).

Fix the LED (fw_setenv machid dd6). Run this on E02 models only! More info.

Create a regular user account (useradd -m -g users your_name). More info.

Run visudo, uncomment # wheel. Now the usergroup wheel is eligible to run the sudo command.

Add your new user from step 7 to the group wheel (gpasswd -a your_name wheel).

Change to the non-root user account (su your_name) and use it as your primary account going forward. Use the sudo command whenever admin rights are required.

Congratulations to your new 30$ Linux home server with endless possibilities! Next up: Installing the Mosquitto MQTT broker.

References:

http://lifehacker.com/5680453/build-a-killer-customized-arch-linux-installation-and-learn-all-about-linux-in-the-process

http://blog.qnology.com/2013/03/tutorial-pogoplug-e02-with-arch-linux.html

[e-verythingy]

A Home Server for 30 Bucks

There is a variety of 'plug' computers on the market, Seagate Dockstar, TonidoPlug, SheevaPlug, to name a few.

I'm going to focus on the Pogoplug - for a single reason: I like their design ("Designed in California, assembled in China").

You can find them on eBay, quite affordable for around 20-40$. But before you purchase, let's make sure you get the right model:

Only get a POGO-E02. It is usually pink-colored, however there are also grey and black E02 models available.

Specs: 1.2GHz ARM, 256MB RAM, 4x USB, Gigabit Ethernet.

I am not sure about the Pogoplug Series 4. I haven't had the chance to play around with a Series 4 model.

Why?

We are going to install a Linux system (Archlinux ARM) on the Pogoplug.

Archlinux support has been deprecated for below models ("If you have not purchased a device yet, we recommend you look into other options as this device is no longer being supported by Arch Linux ARM.")

Unsupported models are:

POGO-P01 - Pro

POGO-P21 - V3

POGO-P24 - V3

POGO-P25 - V3

POGO-B01 - Classic

POGO-B02 - Classic

POGO-B03 - Classic

POGO-B04 - Classic

Got an E02? Good choice! Next up: Installation of Archlinux.

References:

http://archlinuxarm.org/platforms/armv6/pogoplug-v3oxnasend-life