Re: Unexplained stuck memtable flush

It's interesting how they organised the documentation. So it is guaranteed that the ConcurrentLinkedQueue can be modified and won't break the iterator. But I don't see anything mentioning the reverse. Can an iterator removing items from the middle of a queue (which by definition is FIFO) bre

Re: Unexplained stuck memtable flush

Hi Bowen, > I've been reading the source code and the Java documentation. > > In the code > : > > public void signalAll() > { >

Re: Unexplained stuck memtable flush

Hi Jaydeep, Great work! I believe you are spot on. You should create a Jira ticket detail the findings. Cheers, Bowen On 10/11/2024 21:49, Jaydeep Chovatia wrote: Upon further studying the thread dump, I think there is a deadlock between CompactionExecutor (Thread1) & MemtableReclaimMemory (

Re: Unexplained stuck memtable flush

I think this is the correct explanation. It's very similar to CASSANDRA-19576, where compaction is unable to finish because we can't insert into compaction history. Really good analysis, Jaydeep. Jon On Sun, Nov 10, 2024 at 1:51 PM Jaydeep Chovatia wrote: > Upon further studying the thread d

Re: Unexplained stuck memtable flush

Upon further studying the thread dump, I think there is a deadlock between CompactionExecutor (Thread1) & MemtableReclaimMemory (Thread2) in the stack trace I have mentioned in my email below. Please find the deadlock details here: 1. *Thread1:* It has invoked *readOrdering.start()* on *baseCf

Re: Unexplained stuck memtable flush

Hi Abe, The com.datastax.oss.cdc.agent thing in the thread dump is the code forked from https://github.com/datastax/cdc-apache-cassandra/ . It sends the CDC data to Kafka instead of Pulsar, and some other additional features/changes. Due to the lack of documentation and examples on how to re

Re: Unexplained stuck memtable flush

Hey Bowen, Are you able to reproduce the issue without running com.datastax.oss.cdc.agent? I don't see any glaringly obvious bugs there indicated by the thread dump, but it would be useful to rule that out, particularly because it runs as an agent and manages new instances of C*-defined classes

Re: Unexplained stuck memtable flush

Few ideas about timing and timestamps extraction from the heap dump: 1) Here https://github.com/apache/cassandra/blob/c1d89c32d27921d1f77f05d29ee248b8922a4c76/src/java/org/apache/cassandra/db/Keyspace.java#L627 you can check the time when the write operation within "read-hotness-tracker:1" started

Re: Unexplained stuck memtable flush

Hi Bowen, lastSyncedAt is updated by taking pollStarted: lastSyncedAt = pollStarted; where long pollStarted = clock.now(); the logic uses clock from: SyncRunnable sync = new SyncRunnable(preciseTime) preciseTime by default is SystemClock clock.now() -> org.apache.cassandra.utils.Clock.Default#nano

Re: Unexplained stuck memtable flush

The syncComplete is an instance of WaitQueue, and the syncComplete.queue is an instance of ConcurrentLinkedQueue. Surprisingly, the queue is empty. There is no item in the queue's linked list, only the head and tail nodes, each has item=null. The usage of the WaitQueue within the AbstractCommi

Re: Unexplained stuck memtable flush

Sorry, I mistaken CassandraTableWriteHandler for CassandraKeyspaceWriteHandler, and thought the code has changed. I was wrong, the code has not changed, they are two different files. Your read-hotness-tracker thread dump is: "read-hotness-tracker:1" - Thread t@113 java.lang.Thread.State: WAITI

Re: Unexplained stuck memtable flush

I can see some similarities and some differences between your thread dump and ours. In your thread dump: * no MemtableFlushWriter thread * the MemtablePostFlush thread is idle * the MemtableReclaimMemory thread is waiting for a barrier, possibly stuck * the read-hotness-tracker thread is

Re: Unexplained stuck memtable flush

We have also seen this issue a few times in our production (4.1). My teammates added a thread dump here . One of my theories is

Re: Unexplained stuck memtable flush

I think I'm getting really close now. This seems to have something to do with the "read-hotness-tracker:1" thread. The thread dump is: "read-hotness-tracker:1" daemon prio=5 tid=93 WAITING     at jdk.internal.misc.Unsafe.park(Native Method)     at java.util.concurrent.locks.LockSupport.park(Lock

Re: Unexplained stuck memtable flush

I will give it a try and see what I can find. I plan to go down the rabbit hole tomorrow. Will keep you updated. On 05/11/2024 17:34, Jeff Jirsa wrote: On Nov 5, 2024, at 4:12 AM, Bowen Song via user wrote: Writes on this node starts to timeout and fail. But if left untouched, it's only

Re: Unexplained stuck memtable flush

Funny enough, we used to run on ext4 and XFS on mdarray RAID1, but the crappy disks we had (and still have) randomly spitting out garbage data every once in a while. We suspected it's a firmware bug but unable to confirm or reliably reproduce it. Other than this behaviour, those disks work fine

Re: Unexplained stuck memtable flush

> On Nov 5, 2024, at 4:12 AM, Bowen Song via user > wrote: > > Writes on this node starts to timeout and fail. But if left untouched, it's > only gonna get worse, and eventually lead to JVM OOM and crash. > > By inspecting the heap dump created at OOM, we can see that both of the > Memtable

Re: Unexplained stuck memtable flush

Yeah, I looked through your stack trace and saw it wasn't the same thing, but the steps to identify the root cause should be the same. I nuked ZFS from orbit :) This was happening across all the machines at various times in the cluster, and we haven't seen a single issue since switching to XFS.

Re: Unexplained stuck memtable flush

Hi Jon, That is interesting. We happen to be running Cassandra on ZFS. However we have not had any incident for years with this setup, the only change is the recent addition of CDC. I can see that in CASSANDRA-19564, the MemtablePostFlush thread was stuck on the unlink() syscall. But in our

Re: Unexplained stuck memtable flush

I ran into this a few months ago, and in my case I tracked it down to an issue with ZFS not unlinking commitlogs properly. https://issues.apache.org/jira/browse/CASSANDRA-19564 On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov wrote: > I am speaking about a thread dump (stack traces for all th

Re: Unexplained stuck memtable flush

Sorry, I must have misread it. The full thread dump is attached. I compressed it with gzip because the text file is over 1 MB in size. On 05/11/2024 14:04, Dmitry Konstantinov wrote: I am speaking about a thread dump (stack traces for all threads), not a heap dump. The heap dump should contain

Re: Unexplained stuck memtable flush

I am speaking about a thread dump (stack traces for all threads), not a heap dump. The heap dump should contain thread stacks info. Thread dump (stack traces) is small and does not have sensitive info. Regards, Dmitry On Tue, 5 Nov 2024 at 13:53, Bowen Song via user wrote: > It's about 18GB in

Re: Unexplained stuck memtable flush

It's about 18GB in size and may contain a huge amount of sensitive data (e.g. all the pending writes), so I can't share it. However, if there's any particular piece of information you would like to have, I'm more than happy to extract the info from the dump and and share it here. On 05/11/2024

Re: Unexplained stuck memtable flush

Hi Bowen, would it be possible to share a full thread dump? Regards, Dmitry On Tue, 5 Nov 2024 at 12:12, Bowen Song via user wrote: > Hi all, > > We have a cluster running Cassandra 4.1.1. We are seeing the memtable > flush randomly getting stuck. This has happened twice in the last 10 days, >

Re: [External]Unexplained stuck memtable flush

(your message may get ignored because it is off-topic). On 05/11/2024 12:50, Jiri Steuer (EIT) wrote: Of cause, let me explain the situation. I have a common question without direct relation the problem with “Unexplained stuck memtable flush”. I would like to know, how can I identify situation

RE: [External]Unexplained stuck memtable flush

Of cause, let me explain the situation. I have a common question without direct relation the problem with “Unexplained stuck memtable flush”. I would like to know, how can I identify situation that all nodes cross all data centers will be synch. * It is little tricky to wait e.g. 1 day, 2

Re: [External]Unexplained stuck memtable flush

outside of the organization. From:* Bowen Song via user *Sent:* Tuesday, November 5, 2024 1:12 PM *To:* d...@cassandra.apache.org; user@cassandra.apache.org *Cc:* Bowen Song *Subject:* [External]Unexplained stuck memtable flush This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly

RE: [External]Unexplained stuck memtable flush

on is Internal. It was created by and is in property of the EmbedIT. Do not distribute outside of the organization. From: Bowen Song via user Sent: Tuesday, November 5, 2024 1:12 PM To: d...@cassandra.apache.org; user@cassandra.apache.org Cc: Bowen Song Subject: [External]Unexplained stuck memtable

Unexplained stuck memtable flush

Hi all, We have a cluster running Cassandra 4.1.1. We are seeing the memtable flush randomly getting stuck. This has happened twice in the last 10 days, to two different nodes in the same cluster. This started to happen after we enabled CDC, and each time it got stuck, there was at least one