Re: Dropped mutations

[Ayub M]

What does Read and _Trace dropped mutations mean? There is no tracing
enabled on any node in the cluster, what are these _TRACE dropped messages?

INFO  [ScheduledTasks:1] 2019-07-25 21:17:13,878
MessagingService.java:1281 - READ messages were dropped in last 5000 ms: 1
internal and 0 cross node. Mean internal dropped latency: 5960 ms and Mean
cross-node dropped latency: 0 ms
INFO  [ScheduledTasks:1] 2019-07-25 20:38:43,788
MessagingService.java:1281 - _TRACE messages were dropped in last 5000 ms:
5035 internal and 0 cross node. Mean internal dropped latency: 0 ms and
Mean cross-node dropped latency: 0 ms



On Thu, Jul 25, 2019 at 1:49 PM Ayub M  wrote:

> Thanks Jeff, does internal mean local node operations - in this case
> mutation response from local node and cross node means the time it took to
> get response back from other nodes depending on the consistency level
> choosen?
>
> On Thu, Jul 25, 2019 at 11:51 AM Jeff Jirsa  wrote:
>
>> This means your database is seeing commands that have already timed out
>> by the time it goes to execute them, so it ignores them and gives up
>> instead of working on work items that have already expired.
>>
>> The first log line shows 5 second latencies, the second line 6s and 8s
>> latencies, which sounds like either really bad disks or really bad JVM GC
>> pauses.
>>
>>
>> On Thu, Jul 25, 2019 at 8:45 AM Ayub M  wrote:
>>
>>> Hello, how do I read dropped mutations error messages - whats internal
>>> and cross node? For mutations it fails on cross-node and read_repair/read
>>> it fails on internal. What does it mean?
>>>
>>> INFO  [ScheduledTasks:1] 2019-07-21 11:44:46,150
>>> MessagingService.java:1281 - MUTATION messages were dropped in last 5000
>>> ms: 0 internal and 65 cross node. Mean internal dropped latency: 0 ms and
>>> Mean cross-node dropped latency: 4966 ms
>>> INFO  [ScheduledTasks:1] 2019-07-19 05:01:10,620
>>> MessagingService.java:1281 - READ_REPAIR messages were dropped in last 5000
>>> ms: 9 internal and 8 cross node. Mean internal dropped latency: 6013 ms and
>>> Mean cross-node dropped latency: 8164 ms
>>>
>>> --
>>>
>>> Regards,
>>> Ayub
>>>
>>
>
> --
> Regards,
> Ayub
>


-- 
Regards,
Ayub

Re: Cheat Sheet for Unix based OS, Performance troubleshooting

[Jon Haddad]

http://www.brendangregg.com/linuxperf.html

On Sat, Jul 27, 2019 at 2:45 AM Paul Chandler  wrote:

> I have always found Amy's Cassandra 2.1 tuning guide great for the Linux
> performance tuning:
> https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html
>
> Sent from my iPhone
>
> On 26 Jul 2019, at 23:49, Krish Donald  wrote:
>
> Any one has  Cheat Sheet for Unix based OS, Performance troubleshooting ?
>
>

RMI TCP Connection threads

[Vlad]

Hi,
suddenly I noticed that one of three nodes started consume CPU in RMI TCP 
Connection threads.

What it could be?
Thanks.

Re: RMI TCP Connection threads

[Jeff Jirsa]

Someone running nodetool or exporting metrics via JMX are the two mostly likely 
explanations 


> On Jul 28, 2019, at 10:57 PM, Vlad  wrote:
> 
> Hi,
> 
> suddenly I noticed that one of three nodes started consume CPU in RMI TCP 
> Connection threads.
> 
> What it could be?
> 
> Thanks.

Re: RMI TCP Connection threads

[Dinesh Joshi]

Try obtaining a thread dump. It will help debug. Anything that goes via JMX 
such as nodetool could be responsible for it.

Dinesh

> On Jul 28, 2019, at 10:57 PM, Vlad  > wrote:
> 
> Hi,
> 
> suddenly I noticed that one of three nodes started consume CPU in RMI TCP 
> Connection threads.
> 
> What it could be?
> 
> Thanks.