Check if any warn sign stating “ Node is down” or “Node is coming up” in system.log and also see on that node using nodetool tpstats whether dropped mutations are there or not.
From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] Sent: Wednesday, October 25, 2017 3:37 AM To: user@cassandra.apache.org Subject: Re: [EXTERNAL] Lot of hints piling up No OOM or HEAP errors On Tue, Oct 24, 2017 at 2:51 PM, Mohapatra, Kishore <kishore.mohapa...@nuance.com> wrote: Check how many sstables are there for the table you are having issues with. You might be having the Heap pressure. Check your system.log for any OOM error or HEAP error. Thanks Kishore Mohapatra Principal Operations DBA Seattle, WA Email : <mailto:kishore.mohapa...@nuance.com> kishore.mohapa...@nuance.com From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] Sent: Monday, October 23, 2017 11:54 AM To: user@cassandra.apache.org Subject: Re: [EXTERNAL] Lot of hints piling up Do not see any errors in the logs or OS and compactions are happening in the regular interval and good too. Issue here is, this causing replication lag across the datacenters. On Mon, Oct 23, 2017 at 10:23 AM, Mohapatra, Kishore <kishore.mohapa...@nuance.com> wrote: Do you see any error in the cassandra log ? Check compactionstats ? Also check the OS level log messages to see if you are getting hardware level error messages. Thanks Kishore Mohapatra Principal Operations DBA Seattle, WA Ph : 425-691-6417 (cell) Email : <mailto:kishore.mohapa...@nuance.com> kishore.mohapa...@nuance.com From: Jai Bheemsen Rao Dhanwada [mailto:jaibheem...@gmail.com] Sent: Friday, October 20, 2017 9:44 AM To: user@cassandra.apache.org Subject: [EXTERNAL] Lot of hints piling up Hello, We have cassandra cluster in 3 regions with version 2.1.13, and all of a sudden we started seeing lot of hints accumulating on the nodes. We are pretty sure there is no issue with the network between the regions and all the nodes are up and running all the time. Is there any other reason for the hints accumulation other than the n/w? eg: wide rows or bigger objects? Any pointers here could be very helpful. b/w the hints gets processed after some point of time.
