Hi,
We are currently using apache cassandra 3.11.6. We have a cluster of 4
nodes all in single DC.
All of a sudden we started experiencing high connection timeout on the
cluster. This timeout spikes are coming intermittently.
As checked from grafana, read/write time, jvm threads count, client counts
got increased at the same time. But there isn't any major GC pause at this
time.
*On client side* :
Using Local Quorum read/write and getting error Operation timed out -
received only 1 responses
*On checking debug.log:*
DEBUG [Native-Transport-Requests-39] 2020-12-23 15:05:26,843
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
DEBUG [Native-Transport-Requests-16] 2020-12-23 15:05:27,016
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
DEBUG [Native-Transport-Requests-47] 2020-12-23 15:05:30,627
ReadCallback.java:132 - Timed out; received 0 of 1 responses
DEBUG [Native-Transport-Requests-35] 2020-12-23 15:05:31,163
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
DEBUG [Native-Transport-Requests-54] 2020-12-23 15:05:31,481
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
DEBUG [Native-Transport-Requests-7] 2020-12-23 15:05:33,740
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
DEBUG [Native-Transport-Requests-8] 2020-12-23 15:05:37,479
ReadCallback.java:132 - Timed out; received 2 of 3 responses (including
data)
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,287 StatusLogger.java:47 -
Pool Name Active Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 -
MutationStage 0 0 19306477 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 -
ViewMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,295 StatusLogger.java:51 -
ReadStage 1 0 338493026 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 -
RequestResponseStage 0 0 181164440 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 -
ReadRepairStage 0 0 90518075 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,296 StatusLogger.java:51 -
CounterMutationStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 -
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 -
CompactionExecutor 0 0 390126 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 -
MemtableReclaimMemory 0 0 335 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,297 StatusLogger.java:51 -
PendingRangeCalculator 0 0 6 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 -
GossipStage 0 0 479616 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 -
SecondaryIndexManagement 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,298 StatusLogger.java:51 -
HintsDispatcher 0 0 242 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 -
MigrationStage 0 0 7 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 -
MemtablePostFlush 0 0 367 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,299 StatusLogger.java:51 -
PerDiskMemtableFlushWriter_0 0 0 335 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 -
ValidationExecutor 0 0 19 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 -
Sampler 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 -
MemtableFlushWriter 0 0 335 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,300 StatusLogger.java:51 -
InternalResponseStage 0 0 464309 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,301 StatusLogger.java:51 -
AntiEntropyStage 0 0 70 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,301 StatusLogger.java:51 -
CacheCleanupExecutor 0 0 0 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,301 StatusLogger.java:51 -
Native-Transport-Requests 2 0 122590798 0
0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,302 StatusLogger.java:61 -
CompactionManager 0 0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,302 StatusLogger.java:73 -
MessagingService n/a 0/0
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,302 StatusLogger.java:83 -
Cache Type Size Capacity
KeysToSave
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,302 StatusLogger.java:85 -
KeyCache 524287984 524288000
all
INFO [ScheduledTasks:1] 2020-12-23 15:05:43,302 StatusLogger.java:91 -
RowCache 6442302399 6442450944
all
Please help in debugging.
Regards,
Ashish
