Why we think it might be related to hints is , because if we truncate the hints then load goes normal on the nodes. FYI , We had to run repair after truncating hints. Any thoughts ?
On Mon, 27 Jan 2020 at 15:27, Deepak Vohra <dvohr...@yahoo.com.invalid> wrote: > > Hints are a stopgap measure and not a fix to the underlying issue. Run a > full repair. > On Monday, January 27, 2020, 10:17:01 p.m. UTC, Surbhi Gupta < > surbhi.gupt...@gmail.com> wrote: > > > Hi, > > We are on Open source 3.11 . > We have a issue in one of the cluster where lots of hints gets piled up > and they don't get applied within hinted handoff period ( 3 hour in our > case) . > And load and CPU of the server goes very high. > We see lot of messages in system.log and debug.log . Our read repair > chance and dc_local_repair chance is 0.1 . Any pointers are welcome . > > ERROR [ReadRepairStage:83] 2020-01-27 13:08:43,695 > CassandraDaemon.java:228 - Exception in thread > Thread[ReadRepairStage:83,5,main] > > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out > - received only 0 responses. > > DEBUG [ReadRepairStage:111] 2020-01-27 13:10:06,663 ReadCallback.java:242 > - Digest mismatch: > > org.apache.cassandra.service.DigestMismatchException: Mismatch for key > DecoratedKey(4759131696153881383, 9a21276d0af64de28eeeed5d3023b69e) > (142a55e1e28de7daa2ddc34a361 > > 474a0 vs fcba30f022ef25f456914c341022963d) >
