What does Read and _Trace dropped mutations mean? There is no tracing enabled on any node in the cluster, what are these _TRACE dropped messages?
INFO [ScheduledTasks:1] 2019-07-25 21:17:13,878 MessagingService.java:1281 - READ messages were dropped in last 5000 ms: 1 internal and 0 cross node. Mean internal dropped latency: 5960 ms and Mean cross-node dropped latency: 0 ms INFO [ScheduledTasks:1] 2019-07-25 20:38:43,788 MessagingService.java:1281 - _TRACE messages were dropped in last 5000 ms: 5035 internal and 0 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 0 ms On Thu, Jul 25, 2019 at 1:49 PM Ayub M <hia...@gmail.com> wrote: > Thanks Jeff, does internal mean local node operations - in this case > mutation response from local node and cross node means the time it took to > get response back from other nodes depending on the consistency level > choosen? > > On Thu, Jul 25, 2019 at 11:51 AM Jeff Jirsa <jji...@gmail.com> wrote: > >> This means your database is seeing commands that have already timed out >> by the time it goes to execute them, so it ignores them and gives up >> instead of working on work items that have already expired. >> >> The first log line shows 5 second latencies, the second line 6s and 8s >> latencies, which sounds like either really bad disks or really bad JVM GC >> pauses. >> >> >> On Thu, Jul 25, 2019 at 8:45 AM Ayub M <hia...@gmail.com> wrote: >> >>> Hello, how do I read dropped mutations error messages - whats internal >>> and cross node? For mutations it fails on cross-node and read_repair/read >>> it fails on internal. What does it mean? >>> >>> INFO [ScheduledTasks:1] 2019-07-21 11:44:46,150 >>> MessagingService.java:1281 - MUTATION messages were dropped in last 5000 >>> ms: 0 internal and 65 cross node. Mean internal dropped latency: 0 ms and >>> Mean cross-node dropped latency: 4966 ms >>> INFO [ScheduledTasks:1] 2019-07-19 05:01:10,620 >>> MessagingService.java:1281 - READ_REPAIR messages were dropped in last 5000 >>> ms: 9 internal and 8 cross node. Mean internal dropped latency: 6013 ms and >>> Mean cross-node dropped latency: 8164 ms >>> >>> -- >>> >>> Regards, >>> Ayub >>> >> > > -- > Regards, > Ayub > -- Regards, Ayub
