pandemicsyn @pandemicsyn-blog - Tumblr Blog

Ruthless Rye is Ruthless.

swift-informant & statsd. Getting realtime telemetry for @openstack #swift

The problem.

Need to know how many object put’s your cluster is receiving per second? Or how many 401’s are being generated? Is your cluster getting spammed with container creates?

Having good insight into what your swift cluster is doing is hugely important to keeping it healthy and to running down issues when they crop up. As you get larger it also becomes apparent that 5 minute resolution on a graph isn’t enough anymore. If you’re doing 5/10/100k requests a second a lot can happen in 5 minutes.

You need realtime data, or at least with in a few seconds. The data needs to be flexible. You don’t want to slow down the request or decrease proxy performance to obtain this data.

So…whats an ops monkey todo ?

You deploy graphite + statsd and then use swift-informant on all your proxy servers. Easy. You’ll be back to playing Skyrim in no time.

Eh, wtf is swift-informant ?

swift-informant is a really small piece of middleware you run on your swift-proxy server that reports what status codes you’re returning and what method’s you’re servicing in real time. It breaks this all down by whether it was a swift op on an account, container, or object.

How it works:

After a proxy request has been serviced (using a eventlet posthook) it fires a statsd counter incrementing the request method and the status code for the request that was just serviced.

It also breaks these up by whether the request was an operation on an Account, Container, or an Object.

Sample of the generated statsd events:

The beautiful thing here is that these get sent to statsd via UDP. Since its UDP based its completely fire and forget. It’s minimal overhead for the proxy, there’s no having to worry about keeping a tcp connection established. It’s awesome.

To use informant:

Load informant as the first pipeline entry (even before catcherrors):

pipeline = informant catch_errors healthcheck cache ratelimit proxy-server

And add the following filter config:

[filter:informant] use = egg:informant#informant # statsd_host = 127.0.0.1 # statsd_port = 8125 ## standard statsd sample rate 0.0 <= 1 # statsd_sample_rate = 0.5

The commented out values are the defaults. You’ll need to adjust them to fit your environment. This middleware does not require any additional statsd client modules.

The sample rate control’s how many events we fire. A sample rate of 1 will fire events for every request received. If your proxy is servicing 1000 requests a second you’ll be generating 2000 udp packets per second (one packet for the status code, and one for the method)

A sample rate of 0.5 would do it for every other request. 0.25 for every 1 in 4. If your cluster’s already doing a few thousand requests a second a sample rate of 0.25 should be sufficient.

The sample rate is reported along with the event so that statsd can adjust values accordingly before flushing to graphite.

Next steps:

Got Object-server LockTimeout’s… rsync issues…memcache connection errors? Next time, I’ll show you how to generate statsd events in realtime for your swift error logs.

If you’re feeling adventurous, I’ve also been working on a python/eventlet based statsd server, but its a work in progress and is still missing some stuff (like timer support).

#python #swift-informant #swift #devops #openstack

minor swift-recon update to include sockstat info. @openstack #swift

Pushed a minor update to swift-recon today to include reporting info from /proc/net/sockstat and sockstat6:

view on github | view admin guide

Might come in handy for graphing cluster wide time_wait stats or if trying to tracking down socket issues. Should note though that rather than just returning mem from /proc/net/sockstat it actually does pagecount * pagesize for you.

fhines@ubuntu:~$ nc localhost 6010 GET /recon/sockstat HTTP/1.1 200 OK Content-Type: application/json Content-Length: 99 Date: Wed, 16 Nov 2011 03:53:10 GMT Connection: close {"time_wait": 2, "tcp_in_use": 18, "orphan": 0, "tcp6_in_use": 1, "tcp_mem_allocated_bytes": 12288}

#swift-recon #swift #openstack

90 drives to pull out and swap out of trays for the lab. 4 screws each way.

Taken at RackSpace Cloud (Castle)

w00t. No 18 year old but found some 12 year Yamazaki. #whisky

Racker dinner

Taken at Sel de la Terre

Storm rolling in.

Step 1: stock up

sweet. @prpwtrmln hooked me up with a Orange Monster for my bday.

Taken at Schloss von Florian

112 degree's seriously. #omgwtfbbq

#weather

Quick note on whats actually available in swift-recon

A quick note about what’s actually available in swift recon that you can use for monitoring swift. Once you’ve got swift-recon loaded on a storage node and have the cronjob running you’ll be able to pull these metric’s. You’ll need to make a GET request like so:

fhines@ubuntu:~$ w3m -dump http://localhost:6010/recon/quarantined {"objects": 2, "accounts": 0, "containers": 0}

Or something along the lines of:

import urllib2 In [25]: response = urllib2.urlopen('http://localhost:6010/recon/async') In [26]: result = json.loads(response.read()) In [27]: result Out[27]: {'async_pending': 2229}

The available metrics at the moment are:

/recon/load – get the systems current load average

/recon/async – get a count of async pending’s on the system

/recon/mem – dumps /proc/meminfo

/recon/replication – last logged object replication time

/recon/mounted – dumps ALL currently mounted filesystems

/recon/unmounted – dumps all unmounted (failed?) drives assuming mount_check = True (i.e. not a SAIO)

/recon/diskusage – get disk utilization for devices being used as storage devices.

/recon/ringmd5 – get object/container/account ring on disk md5sum’s

/recon/quarantined – get count of quarantined objects/accounts/containers on system.

With the exception of memory statistics and all mounted partitions all of these are available via the swift-recon command.

#swift-recon #python #swift #openstack

Stormtrooper RIP | via Flickr

via flickr.com

Awesome series!

#flickr #photography

swift-recon. Obtaining @openstack #swift cluster health andtelemetry #ops

Note: Recon is included in swift now. Go here for details

For a long time I’ve been thinking that one of the things that might hold Swift back from wider adoption is the operational learning curve. Getting a Swift cluster setup and running is one thing. Keep one up and running is a another. I have a bunch of items that I think would make keeping a Swift cluster up and running (and healthy) a lot simpler. Among those, was a utility that let’s to you interrogate the object servers in the cluster to obtain certain information. To start with, I wanted to be able to easily tell:

How many unmounted (failed) drives there are in the cluster, and on which servers those are located

How many async pendings are present

Drive usage and balance

Load Average (for easy access later on*)

Memory Usage (for easy access later on*)

Checking ring md5sum’s

Logged replication stats

Connection stats (tbd)

Quarantine Statistics (a new pending addition)

At Rackspace we already gather all this info, but its through a bunch of individual bash scripts, python, cronjobs, and monitoring systems that grew as we learned what exactly we needed to monitor. Some of those item’s (like Load) while handy become even more useful when you start thinking about a full on Swift operational dashboard… something that I’ve been thinking about a lot …but thats a blog post for another night.

So with that in mind, I started working on Swift Recon. At the moment, its just a piece of object server middleware and an accompanying cli. It’s already made it into swift trunk so its easy to start to mess with.

First things first, if you want to use swift-recon you’ll need to load it on your object servers:

[pipeline:main] pipeline = recon object-server

There is only one config flag at the moment (it defaults to /var/cache/swift if not present) which is the path prefix the swift-recon cronjob uses when dumping its info and which is read to provide certain statistics that can be painful to obtain in real-time (async pendings, last replication time):

[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift

For now theres also a cronjob that you need to run with “swift-recon-cron /path/to/your/storage.log”, there’s a few config options you can set:

SYSLOG_FACILITY="local0.error" ASYNC_PATH="/srv/node/sd[a-z]/async_pending/" RECON_CACHE_PATH="/var/cache/swift/object.recon"

Once enabled you can pull various operational metrics directly from the object servers:

fhines@ubuntu:~/swift-recon$ curl -i http://127.0.0.1:6010/recon/async; echo HTTP/1.1 200 OK Content-Type: application/json Content-Length: 53 Date: Sun, 12 Jun 2011 06:01:29 GMT {"async_pending": 28}

Or via the swift-recon command:

fhines@ubuntu:~/swift-recon$ swift-recon.py -h =============================================================================== Usage: usage: swift-recon.py [-v] [--suppress] [-a] [-r] [-u] [-d] [-l] [-c] [--objmd5] Options: -h, --help show this help message and exit -v, --verbose Print verbose info --suppress Suppress most connection related errors -a, --async Get async stats -r, --replication Get replication stats -u, --unmounted Check cluster for unmounted devices -d, --diskusage Get disk usage stats -l, --loadstats Get cluster load average stats --objmd5 Get md5sums of object.ring.gz and compare to local copy --swiftdir=SWIFTDIR Default = /etc/swift

Sample run:

fhines@ubuntu:~$ python swift-recon.py -adu --suppress --objmd5 =============================================================================== [2011-07-22 00:18:44] Checking async pendings on 4 hosts... Async stats: low: 7, high: 7, avg: 7, total: 21 =============================================================================== [2011-07-22 00:18:44] Getting unmounted drives from 4 hosts... Not mounted: sdb2 on http://127.0.0.1:6020/recon/unmounted Not mounted: sdb3 on http://127.0.0.1:6030/recon/unmounted Not mounted: sdb4 on http://127.0.0.1:6040/recon/unmounted =============================================================================== [2011-07-22 00:18:44] Checking disk usage on 4 hosts... -> http://127.0.0.1:6020/recon/diskusage: Error. No drive info available. Distribution Graph: 0% 1 ********************************************************************* 2% 1 ********************************************************************* Disk usage: lowest: 0.41%, highest: 2.0%, avg: 1.205% =============================================================================== [2011-07-22 00:18:44] Checking ring md5sum's on 4 hosts... 4/4 hosts matched, 0 error[s] while checking hosts. ===============================================================================

So that’s swift-recon “so far”, its still super green and I’m sure it will change once it gets sometime to cook in our staging environment at work. Also, I’m an ops monkey not a dev, there’s probably some bugs in the swift-recon cli and I’m still relatively new to python.

So what’s next ?

I’m working on cleaning up the cronjob so that “it just works”. It be awesome to eventually ditch it completely.

I’ve got a merge proposal in that add’s support for getting quarantine stats.

I also still need to find a simple/useful way to pull connection information.

There’s no real docs yet.

I’ve started messing around with laying out an ops web dash board. Thing’s like easily rebalancing the ring are must. So’s a layout that doesn’t give you sensory overload.

The only problem with all that, I’m working on this mostly after hours because this is my pet project and my day to day my job has me busy dealing with other “stuff”. So updates might be slow in coming. I’ve got a Github repo for swift-recon that I use while working on stuff and before actually pushing to Launchpad. Also, feedback on what people would like to see/need would be awesome.

#swift-recon #python #swift #devops #openstack

xkcd's "compiling" adapted #launchpad

Been using Launchpad a bit more lately. Everytime I push to a branch this is how I feel:

I don't have to use to it everyday for work like some of my co-workers. I feel for them. Perhaps the Launchpad guys could talk to the github guys and get some pointers.

#github #launchpad #openstack

business cards are here

See the full gallery on Posterous

Minecraft @rackspace castle hq.

See the full gallery on Posterous

Trending Blogs

Recently Viewed Blogs

pandemicsyn