Mesos animated traffic graphs with vizceral
So, when netflix released vizceral, I immediately wanted to have the same ability to monitor our system.
We’re running a ~100 node mesos cluster with a bunch of microservices, for a POC I’ve used my 10 node rasperry pi 3 cluster
Since all our microservices are started using marathon and are in docker and mostly doing web We decided to create a system which will automaticly monitor running dockers started by marathon and display the animated graph on vizceral.
Some guys already made similar system but on docker swarm so We’ve decided to take a similar route.
Which is.. launch an docker on all slaves (or agents from 1.0) which monitors which dockers are running, If the docker is started from marathon start a packetbeat docker to monitor it.
At the end all slaves were sending packets from all marathon started services which were tagged as task name from marathon into the elasticsearch.
Then comes the create a json for vizceral part. Agh.
Since packets were tagged by marathon task id I knew what came into the docker but I didn’t know where it came from (since many services live on the same host and apps were behind docker NAT)
When reading release logs from mesos 1.0 I saw It now support CNI which could come in handy - start containers on separate ip’s and then I would know where the packes came from. Docker also has this ability now: to create macvlan networks - so we went this route. In 1.12.1 docker version macvlan is not in global scope, so recompiling was needed but after that It all came together. Tasks were started all on separate ips.
Then We needed to create a “scraper” service which creates a json for vizceral in order to display the graph. We took all apps from marathon and query elasticsearch for packets which had a direction “IN” and group by service name. Then All client_ips that came into that service we could match by service name from marathon.
In the video below you can see how traffic starts flowing when I refresh the web app.