Graphdat Actions - Automation and Notification
Six months ago Graphdat released Smart Alarms. We gave you the ability to filter out the noise and minimize the false positives. By selecting the time interval and the aggregate (sum, min, max, avg), our alarms became more powerful, but they were limited to only sending an email. While many services will let you action data off of an email (nice job Kevin), it can be limiting.
So today, we announce the release of Graphdat Actions.
Graphdat Actions allow you add multiple actions to an alarm. Now when an alarm triggers you can do anything:
send an email
post to your chatroom (campfire, flowdock, hipchat)
trigger an issue with PagerDuty,
call a URL to add a new server to your infrastructure
call a URL to restart an application,
call a URL to turn on your coffee machine
We have hand picked the initial list of actions based on your feedback, if there are more you would like to see, let us know.
Being notified when something happens is fantastic. Opening and Resolving incidents in PagerDuty is great, but what if we could resolve the issue automatically? When an alarm triggers, let action it.
Instead of dreading that 2am email, how about we resolve it.
In this example, we will create an alarm on the memory usage on a server, and if it goes beyond a certain threshold, we'll restart the service.
Open the Settings dialog and select the Actions tab. We do not have any actions installed yet, so lets create one. Click on the New Action button.
We see a list of the available actions. You can click on the action to read more about it, or click Add to configure an action of that type for your account. We want to add the URL action in this case, as we want Graphdat to call us when something happens, so click on that one.
The URL action takes in a couple of parameters, so lets go through them.
Name: what should we call this? This will show up in the alarms dialog, so we want something descriptive.
Method: what HTTP Method should Graphdat use when calling our server? This example uses a POST, so Graphdat will send some helpful data in the body of the POST so we can adjust what action our server should take
Protocol: HTTP or HTTPS? If you have a certificate, use HTTPS.
URL: how do we contact your server? Make sure to include the port and path in the URL. HTTP will default to 80 and HTTPS will default to 443, if they are different, make sure you set them.
Username: does the URL require authentication? In this example our URL will check for the username and password.
Password: same as above
POST data: Are there any key values pairs that you want to add to the request? Enter these here key=value, one per line.
Enter in all of the values and click Save.
Now that we have an Action, we need to attach it to something. Lets create an Alarm that look at the Memory Usage on a server. When the memory usage goes above a certain threshold, restart the application.
Click on the Alarms tab on the left and then Add alarm.
We want to alarm the Memory Usage, so we select Memory Usage from the drop down, enter in our values and pick the new Restart My Service action from the drop down and hit Save.
With my new Alarm and Action in place, we need something on the server to listen to that POST. Below is a simple webserver in node.js that listens for the POST and runs a command. In this case, it's restarting a service called 'graphdat' using supervisord
https://gist.github.com/codemoran/7921146
With the server up and running:
$ node restart-graphdat-on-the-local-machine.js
We can see the service is up and running.
Now how do we test that this works? We can either spike the memory on the server, or we can edit the alarm to a very low value so we know that it will trigger. That sounds much better. Lets set the alarm to a 1 second period and to a threshold of 1%.
With the alarm saved, we can see that it triggers a couple of seconds later.
If we have a look in our console, we can now see that the service was restarted.
Put the Alarm back to it original settings.
The alarm is resolved
And the service ignores the resolution because we do not want to restart the service when everything is up and running.
With Graphdat Plugins + Graphdat Alarms + Graphdat Actions, you can be notified of when things are happening on your server infrastructure and in your applications, and now, you can do something about it.
If you have not tried a Graphdat plugin yet, getting started is simple. Welcome to automation!