Hi All,
We have a tiny 3-node cluster
C* version 3.9 (I know 3.11 is better/stable, but can’t upgrade immediately)
HEAP_SIZE is 2G
JVM options are default
All setting in cassandra.yaml are default (file_cache_size_in_mb not set)
Data per node – just ~ 1Gbyte
We’re getting following errors messages:
DEBUG [CompactionExecutor:87412] 2019-03-06 11:00:13,545
CompactionTask.java:150 - Compacting (ed4a4d90-4028-11e9-adc0-230e0d6622df)
[/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23248-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23247-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23246-big-Data.db:level=0,
/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23245-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:87412] 2019-03-06 11:00:13,582
CompactionTask.java:230 - Compacted (ed4a4d90-4028-11e9-adc0-230e0d6622df) 4
sstables to
[/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/mc-23249-big,]
to level=0. 6.264KiB to 1.485KiB (~23% of original) in 36ms. Read Throughput
= 170.754KiB/s, Write Throughput = 40.492KiB/s, Row Throughput = ~106/s. 194
total partitions merged to 44. Partition merge counts were {1:18, 4:44, }
INFO [IndexSummaryManager:1] 2019-03-06 11:00:22,007
IndexSummaryRedistribution.java:75 - Redistributing index summaries
INFO [pool-1-thread-1] 2019-03-06 11:11:24,903 NoSpamLogger.java:91 - Maximum
memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [pool-1-thread-1] 2019-03-06 11:26:24,926 NoSpamLogger.java:91 - Maximum
memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [pool-1-thread-1] 2019-03-06 11:41:25,010 NoSpamLogger.java:91 - Maximum
memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
INFO [pool-1-thread-1] 2019-03-06 11:56:25,018 NoSpamLogger.java:91 - Maximum
memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
What’s interesting that “Maximum memory usage reached” messages appears each 15
minutes.
Reboot temporary solve the issue, but it then appears again after some time
Checked, there are no huge partitions (max partition size is ~2Mbytes )
How such small amount of data may cause this issue?
How to debug this issue further?
Regards,
Kyrill
