#swift-recon

[pandemicsyn-blog]

minor swift-recon update to include sockstat info. @openstack #swift

Pushed a minor update to swift-recon today to include reporting info from /proc/net/sockstat and sockstat6:

view on github | view admin guide

Might come in handy for graphing cluster wide time_wait stats or if trying to tracking down socket issues. Should note though that rather than just returning mem from /proc/net/sockstat it actually does pagecount * pagesize for you.

fhines@ubuntu:~$ nc localhost 6010 GET /recon/sockstat HTTP/1.1 200 OK Content-Type: application/json Content-Length: 99 Date: Wed, 16 Nov 2011 03:53:10 GMT Connection: close {"time_wait": 2, "tcp_in_use": 18, "orphan": 0, "tcp6_in_use": 1, "tcp_mem_allocated_bytes": 12288}

[pandemicsyn-blog]

Quick note on whats actually available in swift-recon

A quick note about what’s actually available in swift recon that you can use for monitoring swift. Once you’ve got swift-recon loaded on a storage node and have the cronjob running you’ll be able to pull these metric’s. You’ll need to make a GET request like so:

fhines@ubuntu:~$ w3m -dump http://localhost:6010/recon/quarantined {"objects": 2, "accounts": 0, "containers": 0}

Or something along the lines of:

import urllib2 In [25]: response = urllib2.urlopen('http://localhost:6010/recon/async') In [26]: result = json.loads(response.read()) In [27]: result Out[27]: {'async_pending': 2229}

The available metrics at the moment are:

/recon/load – get the systems current load average

/recon/async – get a count of async pending’s on the system

/recon/mem – dumps /proc/meminfo

/recon/replication – last logged object replication time

/recon/mounted – dumps ALL currently mounted filesystems

/recon/unmounted – dumps all unmounted (failed?) drives assuming mount_check = True (i.e. not a SAIO)

/recon/diskusage – get disk utilization for devices being used as storage devices.

/recon/ringmd5 – get object/container/account ring on disk md5sum’s

/recon/quarantined – get count of quarantined objects/accounts/containers on system.

With the exception of memory statistics and all mounted partitions all of these are available via the swift-recon command.

[pandemicsyn-blog]

swift-recon. Obtaining @openstack #swift cluster health andtelemetry #ops

Note: Recon is included in swift now. Go here for details

For a long time I’ve been thinking that one of the things that might hold Swift back from wider adoption is the operational learning curve. Getting a Swift cluster setup and running is one thing. Keep one up and running is a another. I have a bunch of items that I think would make keeping a Swift cluster up and running (and healthy) a lot simpler. Among those, was a utility that let’s to you interrogate the object servers in the cluster to obtain certain information. To start with, I wanted to be able to easily tell:

How many unmounted (failed) drives there are in the cluster, and on which servers those are located

How many async pendings are present

Drive usage and balance

Load Average (for easy access later on*)

Memory Usage (for easy access later on*)

Checking ring md5sum’s

Logged replication stats

Connection stats (tbd)

Quarantine Statistics (a new pending addition)

At Rackspace we already gather all this info, but its through a bunch of individual bash scripts, python, cronjobs, and monitoring systems that grew as we learned what exactly we needed to monitor. Some of those item’s (like Load) while handy become even more useful when you start thinking about a full on Swift operational dashboard… something that I’ve been thinking about a lot …but thats a blog post for another night.

So with that in mind, I started working on Swift Recon. At the moment, its just a piece of object server middleware and an accompanying cli. It’s already made it into swift trunk so its easy to start to mess with.

First things first, if you want to use swift-recon you’ll need to load it on your object servers:

[pipeline:main] pipeline = recon object-server

There is only one config flag at the moment (it defaults to /var/cache/swift if not present) which is the path prefix the swift-recon cronjob uses when dumping its info and which is read to provide certain statistics that can be painful to obtain in real-time (async pendings, last replication time):

[filter:recon] use = egg:swift#recon recon_cache_path = /var/cache/swift

For now theres also a cronjob that you need to run with “swift-recon-cron /path/to/your/storage.log”, there’s a few config options you can set:

SYSLOG_FACILITY="local0.error" ASYNC_PATH="/srv/node/sd[a-z]/async_pending/" RECON_CACHE_PATH="/var/cache/swift/object.recon"

Once enabled you can pull various operational metrics directly from the object servers:

fhines@ubuntu:~/swift-recon$ curl -i http://127.0.0.1:6010/recon/async; echo HTTP/1.1 200 OK Content-Type: application/json Content-Length: 53 Date: Sun, 12 Jun 2011 06:01:29 GMT {"async_pending": 28}

Or via the swift-recon command:

fhines@ubuntu:~/swift-recon$ swift-recon.py -h =============================================================================== Usage: usage: swift-recon.py [-v] [--suppress] [-a] [-r] [-u] [-d] [-l] [-c] [--objmd5] Options: -h, --help show this help message and exit -v, --verbose Print verbose info --suppress Suppress most connection related errors -a, --async Get async stats -r, --replication Get replication stats -u, --unmounted Check cluster for unmounted devices -d, --diskusage Get disk usage stats -l, --loadstats Get cluster load average stats --objmd5 Get md5sums of object.ring.gz and compare to local copy --swiftdir=SWIFTDIR Default = /etc/swift

Sample run:

fhines@ubuntu:~$ python swift-recon.py -adu --suppress --objmd5 =============================================================================== [2011-07-22 00:18:44] Checking async pendings on 4 hosts... Async stats: low: 7, high: 7, avg: 7, total: 21 =============================================================================== [2011-07-22 00:18:44] Getting unmounted drives from 4 hosts... Not mounted: sdb2 on http://127.0.0.1:6020/recon/unmounted Not mounted: sdb3 on http://127.0.0.1:6030/recon/unmounted Not mounted: sdb4 on http://127.0.0.1:6040/recon/unmounted =============================================================================== [2011-07-22 00:18:44] Checking disk usage on 4 hosts... -> http://127.0.0.1:6020/recon/diskusage: Error. No drive info available. Distribution Graph: 0% 1 ********************************************************************* 2% 1 ********************************************************************* Disk usage: lowest: 0.41%, highest: 2.0%, avg: 1.205% =============================================================================== [2011-07-22 00:18:44] Checking ring md5sum's on 4 hosts... 4/4 hosts matched, 0 error[s] while checking hosts. ===============================================================================

So that’s swift-recon “so far”, its still super green and I’m sure it will change once it gets sometime to cook in our staging environment at work. Also, I’m an ops monkey not a dev, there’s probably some bugs in the swift-recon cli and I’m still relatively new to python.

So what’s next ?

I’m working on cleaning up the cronjob so that “it just works”. It be awesome to eventually ditch it completely.

I’ve got a merge proposal in that add’s support for getting quarantine stats.

I also still need to find a simple/useful way to pull connection information.

There’s no real docs yet.

I’ve started messing around with laying out an ops web dash board. Thing’s like easily rebalancing the ring are must. So’s a layout that doesn’t give you sensory overload.

The only problem with all that, I’m working on this mostly after hours because this is my pet project and my day to day my job has me busy dealing with other “stuff”. So updates might be slow in coming. I’ve got a Github repo for swift-recon that I use while working on stuff and before actually pushing to Launchpad. Also, feedback on what people would like to see/need would be awesome.