Edi Bice @edi-bice - Tumblr Blog

5 posts!

All together now: Samza, jmxtrans, OpenTSDB and Grafana

I have a Samza pipeline consisting of about 20 Samza/YARN jobs. Similarly to an assembly line each job tacks on additional calculated fields to the running message - the automobile. The overall throughput and latency of the pipeline is determined by the slowest job. One really needs a custom dashboard of all the important metric graphs in order to spot issues. VisualVM does not provide such facilities.

For each job in the pipeline, from first to last, find the job in the running jobs in the YARN Resource Manager UI, click on Application Master link, and look up the JMX access URL.

Copy the JMX URL, open a terminal to the host where you installed jmxtrans, change to the /var/lib/jmxtrans directory, and create a file with your job’s name and .json extension.

Here’s what mine (join_billaddr_feats.json) looks like

{ "servers" : [ { "url" : "service:jmx:rmi://lvsdevfrsa09.gspt.net:45412/jndi/rmi://lvsdevfrsa09.gspt.net:45411/jmxrmi", "queries" : [ { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "org.apache.samza.container.SamzaContainerMetrics:type=samza-container-0,name=process-envelopes", "resultAlias": "join_billaddr_feats.process-envelopes", "attr" : [ "Count" ] }, { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "com.ebay.precog.samza.JoinBillAddrFeatsTask:type=TaskName-Partition 0,name=duration-ms", "resultAlias": "join_billaddr_feats.duration-ms", "attr" : [ "Value" ] }, { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "org.apache.samza.system.kafka.KafkaSystemProducerMetrics:type=samza-container-0,name=kafka-producer-sends", "resultAlias": "join_billaddr_feats.kafka-producer-sends", "attr" : [ "Count" ] } ], "numQueryThreads" : 2 } ] }

Make sure you change all the metrics, aliases, hosts, JMX URLs, and ports to suit your setting.

As soon as you save and exit, if all is well, the Samza job will report values for those metrics to JMX, jmxtrans will pick up and send to OpenTSDB, and Grafana will update the graph on the dashboard.

Speaking of which, it should be very intuitive to build out a dashboard in Grafana. Add a OpenTSDB data source, add a Graph Panel, start typing your resultAlias for your desired metric and off you go.

Here’s one of my dashboards with several of the panels (corresponding to jobs in the middle of the pipeline) collapsed:

#apache samza #grafana #opentsd #jmxtrans

Monitoring Apache Samza job pipelines

Apache Samza processes Apache Kafka streams via Apache Hadoop YARN jobs. Each job runs one task with a very simple interface - a process method which receives one message at a time from one or more input streams and can send a message to one or more output streams. Any real use case strings together many of these tasks where the output of one job is the input of another much like an assembly line.

Samza publishes metrics via JMX and via a “metrics” Kafka topic. YARN jobs consist of one or more containers which run on the same or different cluster nodes. It’s really each container that publishes JMX metrics and one can find out the JMX access URL via the Application Master dashboard of each Samza/YARN job. Here’s one of mine for example:

Ordinary: service:jmx:rmi://localhost:43321/jndi/rmi://localhost:43320/jmxrmi

Tunneling: service:jmx:rmi://lvsdevfrsa04.gspt.net:43321/jndi/rmi://lvsdevfrsa04.gspt.net:43320/jmxrmi

VisualVM is a great visual tool which “enhances the capability of monitoring and performance analysis for the Java SE platform”. I use it to explore all the JMX metrics published by Samza jobs.

Install it, run it, click on File, Add JMX Connection and then copy and paste the Tunneling URL from your Samza job. Once added, double click to open, view the Monitor, Threads, Sampler tabs but definitely go to MBeans for all the custom Samza metrics and more.

jmxtrans - “the missing connector between speaking to a JVM via JMX on one end and whatever logging / monitoring / graphing package”

So I installed and configured Grafana - graphing check.

Installed and configured OpenTSDB - graphing data storage check.

Now for the graphing data.

You can certainly feed it by hand! Which is really neat for testing what we have so far.

telnet your_opentsdb_host 4242

once you’re connected type

put sys.cpu.user 1356998400 42.5 host=webserver01 cpu=0

then go to Grafana and check that the metric data came through

Now for feeding real data OpenTSDB has the tcollector http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html

While tcollector is great and supports quite some “big data” software it does not provide for your own JMX metrics. I’m sure it can be done but first it needs Python >=2.7 which RedHat Enterprise Linux 6 does not have (and it’s not worth messing with).

Because jmxtrans.

It supports OpenTSDB, in addition to Ganglia, Graphite, InfluxDB, ...

I downloaded the latest RPM from http://central.maven.org/maven2/org/jmxtrans/jmxtrans/252/jmxtrans-252.rpm

Installed as root via

yum install jmxtrans-252.rpm

Started as root via

/etc/init.d/jmxtrans start

And tailed its log via

tail -f /var/log/jmxtrans/jmxtrans.log

Now to get it to do anything place any of the example files from https://github.com/jmxtrans/jmxtrans/tree/master/jmxtrans-examples/src/main/resources in your /var/lib/jmxtrans folder.

Or use my example test.json (which polls a Samza Yarn container for its mem-heap-used-mb metric) modifying url and host to suit your configuration:

{ "servers" : [ { "url" : "service:jmx:rmi://one_of_my_yarn_nodes:54411/jndi/rmi://one_of_my_yarn_nodes:54410/jmxrmi", "queries" : [ { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "my_opentsdb_host" } }, { "@class":"com.googlecode.jmxtrans.model.output.KeyOutWriter", "settings":{ "outputFile" : "/tmp/stats.txt" } } ],

"obj" : "org.apache.samza.metrics.JvmMetrics:type=samza-container-0,name=mem-heap-used-mb",

"resultAlias": "heap",

"attr" : [ "Value" ] } ], "numQueryThreads" : 2 } ] }

First check that the metric is being output to /tmp/stats.txt

Then check Grafana (or OpenTSDB) for a metric named “heap”.

Grafana

Having decided on using OpenTSDB for its scalability (http://serverfault.com/questions/362338/what-is-the-difference-between-opentsdb-and-graphite), I needed a frontend to go with it.

Grafana was an easy choice. It supports OpenTSDB as a data source out-of-the-box. Community looks very vibrant - 217 contributors, 33 branches, 35 releases, 5,249 commits. More importantly it is easy to get up and running and it works great with OpenTSDB.

I am using RedHat Enterprise Linux 6.6 and opted for the "Install via YUM repository" choice (http://docs.grafana.org/installation/rpm/) in order to easily upgrade etc:

Add following to /etc/yum.repos.d/grafana.repo

[grafana] name=grafana baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch repo_gpgcheck=1 enabled=1 gpgcheck=1 gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana sslverify=1 sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Save and exit and then proceed to install.

yum install grafana

/sbin/chkconfig --add grafana-server service grafana-server start /sbin/chkconfig | grep grafana

Check out Grafana UI at http://your_machine:3000

OpenTSDB on Hortonworks

Make sure HBase is set up and running.

Find out the list of zookeeper servers and ports your cluster is using.

Pick a node - any node - in the cluster and log in.

sudo su -

wget https://github.com/OpenTSDB/opentsdb/releases/download/v2.1.3/opentsdb-2.1.3.noarch.rpm

edit /etc/opentsdb/opentsdb.conf uncommenting the tsd.storage.hbase.zk_quorum line and assigning it a comma separated list of zookeeper servers in your cluster

env COMPRESSION=NONE HBASE_HOME=/usr/hdp/current/hbase-client /usr/share/opentsdb/tools/create_table.sh

service opentsdb start

tail -f /var/log/opentsdb/opentsdb.log

... and you’ll most likely see a ton of these

“ERROR [HBaseClient.processResult] - The znode for the -ROOT- region doesn't exist!”

find configured value of zookeeper.znode.parent in HBase config via Hortonworks HDP Ambari - mine was /hbase-unsecure

change tsd.storage.hbase.zk_basedir in /etc/opentsdb/opentsdb.conf pointing it to the value above (/hbase-unsecure)

service opentsdb start

tail -f /var/log/opentsdb/opentsdb.log

The “ERROR [HBaseClient.processResult] - The znode for the -ROOT- region doesn't exist!” should be gone and you should see:

INFO [TSDMain.main] - Ready to serve on /0.0.0.0:4242

Trending Blogs

Recently Viewed Blogs

Edi Bice