All together now: Samza, jmxtrans, OpenTSDB and Grafana
I have a Samza pipeline consisting of about 20 Samza/YARN jobs. Similarly to an assembly line each job tacks on additional calculated fields to the running message - the automobile. The overall throughput and latency of the pipeline is determined by the slowest job. One really needs a custom dashboard of all the important metric graphs in order to spot issues. VisualVM does not provide such facilities.
For each job in the pipeline, from first to last, find the job in the running jobs in the YARN Resource Manager UI, click on Application Master link, and look up the JMX access URL.
Copy the JMX URL, open a terminal to the host where you installed jmxtrans, change to the /var/lib/jmxtrans directory, and create a file with your job’s name and .json extension.
Here’s what mine (join_billaddr_feats.json) looks like
{ "servers" : [ { "url" : "service:jmx:rmi://lvsdevfrsa09.gspt.net:45412/jndi/rmi://lvsdevfrsa09.gspt.net:45411/jmxrmi", "queries" : [ { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "org.apache.samza.container.SamzaContainerMetrics:type=samza-container-0,name=process-envelopes", "resultAlias": "join_billaddr_feats.process-envelopes", "attr" : [ "Count" ] }, { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "com.ebay.precog.samza.JoinBillAddrFeatsTask:type=TaskName-Partition 0,name=duration-ms", "resultAlias": "join_billaddr_feats.duration-ms", "attr" : [ "Value" ] }, { "outputWriters" : [ { "@class" : "com.googlecode.jmxtrans.model.output.OpenTSDBWriter", "settings" : { "port" : 4242, "host" : "lvsdevfrsa09.gspt.net" } } ], "obj" : "org.apache.samza.system.kafka.KafkaSystemProducerMetrics:type=samza-container-0,name=kafka-producer-sends", "resultAlias": "join_billaddr_feats.kafka-producer-sends", "attr" : [ "Count" ] } ], "numQueryThreads" : 2 } ] }
Make sure you change all the metrics, aliases, hosts, JMX URLs, and ports to suit your setting.
As soon as you save and exit, if all is well, the Samza job will report values for those metrics to JMX, jmxtrans will pick up and send to OpenTSDB, and Grafana will update the graph on the dashboard.
Speaking of which, it should be very intuitive to build out a dashboard in Grafana. Add a OpenTSDB data source, add a Graph Panel, start typing your resultAlias for your desired metric and off you go.
Here’s one of my dashboards with several of the panels (corresponding to jobs in the middle of the pipeline) collapsed: