Re: Cassandra 4.0 randomly freezes on heavy load

2022-02-25 Thread Azamat Hackimov
OK, I think I figured out what was wrong in my setup. Appears that during the upgrade course I totally missed Garbage Collector settings in jvm11-server.options. After installation from RPM it defines the default CMS garbage collector, which is totally wrong for my hi-load environment. I returned b

Re: Cassandra 4.0 randomly freezes on heavy load

2022-02-25 Thread Paulo Motta
> I can reproduce this with a huge load using dsbulk, but still can't determine the cause of the problem. Can you get a thread dump (jstack ) when the system freezes? This might be helpful to determine the cause of the freeze. Also, can you reproduce this in a simpler environment (ccm + dsbulk)?

Re: Cassandra 4.0 randomly freezes on heavy load

2022-02-25 Thread Bowen Song
Okay, that ruled it out. Anything interesting in the GC logs? Was Cassandra stuck at a GC safepoint? You may need to enable the detailed GC logs to see these. On 25/02/2022 10:02, Azamat Hackimov wrote: Hello! No, I have a directly attached NVMe disk, and there are no IO or network issues. п

Re: Cassandra 4.0 randomly freezes on heavy load

2022-02-25 Thread Azamat Hackimov
Hello! No, I have a directly attached NVMe disk, and there are no IO or network issues. пт, 25 февр. 2022 г. в 12:50, Bowen Song : > > Do you have any network based mountpoint, such as NFS or samba? I have > seen similar behaviour on other Java based applications at GC safepoint > when the networ

Re: Cassandra 4.0 randomly freezes on heavy load

2022-02-25 Thread Bowen Song
Do you have any network based mountpoint, such as NFS or samba? I have seen similar behaviour on other Java based applications at GC safepoint when the network based filesystem loses their connection and reconnects. On 25/02/2022 06:09, Azamat Hackimov wrote: Hello! I recently migrated Cassan

Cassandra 4.0 randomly freezes on heavy load

2022-02-24 Thread Azamat Hackimov
Hello! I recently migrated Cassandra from 3.11.x to 4.0 and got strange freezes on heavy load. It looks like some nodes in DC stopped responding and got DN status. I cannot check status directly on node via nodetool status or even restart Cassandra with `systemctl restart cassandra` command. Only