You didn’t mention version, but I saw this kind of thing very often in the 1.1 line. Often this is connected to network flakiness. Are these VMs? In the cloud? Connected over a WAN? You mention that ping seems fine. Take a look at the phi_convict_threshold in c assandra.yaml. You may need to increase it to reduce the UP/DOWN flapping behavior.
Sean Durity From: Joel Samuelsson [mailto:samuelsson.j...@gmail.com] Sent: Tuesday, February 23, 2016 9:41 AM To: user@cassandra.apache.org Subject: Re: Nodes go down periodically Hi, Thanks for your reply. I have debug logging on and see no GC pauses that are that long. GC pauses are all well below 1s and 99 times out of 100 below 100ms. Do I need to enable GC log options to see the pauses? I see plenty of these lines: DEBUG [ScheduledTasks:1] 2016-02-22 10:43:02,891 GCInspector.java (line 118) GC for ParNew: 24 ms for 1 collections as well as a few CMS GC log lines. Best regards, Joel 2016-02-23 15:14 GMT+01:00 Hannu Kröger <hkro...@gmail.com<mailto:hkro...@gmail.com>>: Hi, Those are probably GC pauses. Memory tuning is probably needed. Check the parameters that you already have customised if they make sense. http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html Hannu On 23 Feb 2016, at 16:08, Joel Samuelsson <samuelsson.j...@gmail.com<mailto:samuelsson.j...@gmail.com>> wrote: Our nodes go down periodically, around 1-2 times each day. Downtime is from <1 second to 30 or so seconds. INFO [GossipTasks:1] 2016-02-22 10:05:14,896 Gossiper.java (line 992) InetAddress /109.74.13.67<http://109.74.13.67/> is now DOWN INFO [RequestResponseStage:8844] 2016-02-22 10:05:38,331 Gossiper.java (line 978) InetAddress /109.74.13.67<http://109.74.13.67/> is now UP I find nothing odd in the logs around the same time. I logged a ping with timestamp and checked during the same time and saw nothing weird (ping is less than 2ms at all times). Does anyone have any suggestions as to why this might happen? Best regards, Joel ________________________________ The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.
