Cassandra ring problems
We're using Cassandra for a project at work. In one lower environment I've launched without problems a 6 nodes cluster at the beginning of the week. I'm saying without problems because all boxes came up ok and no clients had problems talking to the cluster.
But when I ran nodetool ring from one of the boxes it only showed 5 machines, one of which was owning 33% of the ring:
Address DC Rack Status State Load Owns Token 141784319550391026443072753098378663705 <IP1> us-east 1a Up Normal 271.02 KB 16.67% 1808575600 <IP2> us-east 1d Up Normal 417.66 KB 33.33% 56713727820156410577229101240436610842 <IP3> us-east 1a Up Normal 324.21 KB 16.67% 85070591730234615865843651859750628463 <IP4> us-east 1d Up Normal 125.87 KB 16.67% 113427455640312821154458202479064646084 <IP5> us-east 1e Up Normal 268.65 KB 16.67% 141784319550391026443072753098378663705
When ran from any of the 6 boxes, including the one that was missing from the list - let's call it IPX - nodetool ring showed everyone was up and normal & pretty much the same information. Running nodetool info on IPX it seemed to think it had the same token as IP5 (the other box in AZ 1e):
Token : 141784319550391026443072753098378663705 Gossip active : true Load : 322.3 KB Generation No : 1388719379 Uptime (seconds) : 254182 Heap Memory (MB) : 89.50 / 3753.88 Data Center : us-east Rack : 1e Exceptions : 0
To try and make things whole again I decided to move the token of IPX to a better place on the ring.
As a consequence running nodetool info from IP5 produced:
Token : 113427455640312821154458202479064646084 Gossip active : true Load : 317.96 KB Generation No : 1388719379 Uptime (seconds) : 257647 Heap Memory (MB) : 85.41 / 3753.88 Exception in thread "main" java.lang.AssertionError: Could not find myself in the endpoint list, something is very wrong! at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:557) at org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:564) at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:313) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:651)
And running nodetool ring from another box showed:
Address DC Rack Status State Load Owns Token 141784319550391026443072753098378663705 <IP1> us-east 1a Up Normal 275.44 KB 16.67% 1808575600 <IPX> us-east 1d Up Normal 273.17 KB 16.67% 28356863910078205288614550619314017621 <IP2> us-east 1d Up Normal 422.08 KB 16.67% 56713727820156410577229101240436610842 <IP3> us-east 1a Up Normal 315.42 KB 16.67% 85070591730234615865843651859750628463 <IP4> us-east 1e Up Normal 273.07 KB 33.33% 141784319550391026443072753098378663705
So it looks like IPX got in the ring and IP5 got kicked out (and it's not happy about it)...
Restarting Cassandra on IP5 seemed to make everyone happy, so I ran nodetool cleanup followed by nodetool repair to wrap things up.
All is left now is to find some metric (hopefully JMX) that can detect such intriguing situations.