Is this from the 1.1 line, perhaps? In my experience it could be very flappy for no particular reason we could discover. 1.1 is a pretty dusty version. Upgrading into the 2.1 or later would be a good idea. If you have to upgrade in place without down time, you will need to go through many upgrades to get to the latest versions. (The late versions of 1.2 are pretty stable, though.)
Also, you can look at the phi_convict_threshold Cassandra.yaml parameter to help with network flakiness. Default is 8, I believe in most versions. Increasing to 10 or 12 might help. Finally, a 2 node cluster doesn't give you many of the benefits of a distributed system. Consider adding some nodes, if you can. Sean Durity - Lead Cassandra Admin Big DATA Team For support, create a JIRA<https://portal.homedepot.com/sites/bigdata/Shared%20Documents/Jira%20Hadoop%20Support%20Workflow.pdf> From: Michael Fong [mailto:michael.f...@ruckuswireless.com] Sent: Wednesday, March 23, 2016 4:12 AM To: user@cassandra.apache.org Subject: Gossip heartbeat and packet capture Hi, all, We are trying to reason the possible scenarios when a C*(v1.x) cluster connection keeps flapping in production. (Two node cluster, each node keeps marking the other node DOWN but came back UP within seconds; multiple times a day) We have checked the load on the cluster i- very light and low GC activities also. We have also checked the network interface / devices were working just fine on the nodes during the incidence. We are changing our investigation direction to the network topology/settings, so we are thinking to capture gossip heartbeat packet to verify if the packet is received as expected on the other end. Has anyone tried to capture the packet of gossip internode communication? What would be the filter / criteria to grep heartbeat-related packet only? Thanks in advance! Michael ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
