Re: dropped mutations cross node

Thanks, I've done a lot of conf changes to fix  the problem but nothing worked (last one was disabling hints) and after a few days problem gone!! The source of droppedCrossNode was changing every half an hour and it was not always the new nodes No difference between new nodes and old ones in c

Re: dropped mutations cross node

Sorry for the late reply. Do you still need assistance with this issue? If the source of the dropped mutations and high latency are the newer nodes, that indicates to me that you have an issue with the commitlog disks. Are the newer nodes identical in hardware configuration to the pre-existing nod

Re: Dropped mutations

What does Read and _Trace dropped mutations mean? There is no tracing enabled on any node in the cluster, what are these _TRACE dropped messages? INFO [ScheduledTasks:1] 2019-07-25 21:17:13,878 MessagingService.java:1281 - READ messages were dropped in last 5000 ms: 1 internal and 0 cross node. M

Re: Dropped mutations

Thanks Jeff, does internal mean local node operations - in this case mutation response from local node and cross node means the time it took to get response back from other nodes depending on the consistency level choosen? On Thu, Jul 25, 2019 at 11:51 AM Jeff Jirsa wrote: > This means your data

Re: Dropped mutations

Hello Jeff, Request you to help on how to visualise the terms 1. Internal mutations 2. Cross node mutations 3. Mean internal dropped latency 4. Cross node dropped latency Thanks, Rajsekhar On Thu, 25 Jul, 2019, 9:21 PM Jeff Jirsa, wrote: > This means your database is seeing commands that have

Re: Dropped mutations

This means your database is seeing commands that have already timed out by the time it goes to execute them, so it ignores them and gives up instead of working on work items that have already expired. The first log line shows 5 second latencies, the second line 6s and 8s latencies, which sounds li

Re: Dropped Mutations

Thanks a lot Hitesh! I'll try to re-tune the heap to a lower level Shalom Sagges DBA T: +972-74-700-4035 We Create Meaningful Connections

Re: Dropped Mutations

Hi , I'll recommend tuning you heap size further( preferably lower) as large Heap size can lead to Large Garbage collection pauses also known as also known as a stop-the-world event. A pause occurs when a region of memory is full and the JVM needs to make space to continue. During a pause all oper

Re: Dropped Mutations

Dropped mutations aren't data loss. Data loss implies the data was already there and is now gone, whereas for a dropped mutation the data was never there in the first place. A dropped mutation just results in a inconsistency, or potentially no data if all mutations are dropped, and C* will tell you

Re: dropped mutations, UnavailableException, and long GC

1. Why 24GB of heap? Do you need this high heap? Bigger heap can lead to longer GC cycles but 15min look too long. 2. Do you have ROW cache enabled? 3. How many column families do you have? 4. Enable GC logs and monitor what GC is doing to get idea of why it is taking so long. You can add following