Maxim Muzafarov created CASSANDRA-21188:
-------------------------------------------
Summary: Race between compaction and dictionary compression
training. Status stuck at SAMPLING. ExportImportListCompressionDictionaryTest
hangs
Key: CASSANDRA-21188
URL: https://issues.apache.org/jira/browse/CASSANDRA-21188
Project: Apache Cassandra
Issue Type: Bug
Reporter: Maxim Muzafarov
There is a race beween compaction process and dictionary compression training
start:
# CompressionDictionaryManager: We collect all live sstables
# ICompressionDictionaryTrainer: start a new traning
# currentTrainingStatus moved to SAMPLING
# All SSTables get compacted within concurrent compaction thread
# SSTableSamplingTask: in the constructor sstable.tryRef return null
# We run this task on a thread pool and it exists in cancelManualTraining
# the currentTrainingStatus remains SAMPLING (should be *FAILED* or
{*}COMPLETED!{*})
ExportImportListCompressionDictionaryTest hangs for 10 minutes (configured
constant) for now reason.
The logs:
{code}
INFO [PerDiskMemtableFlushWriter_0:1] 2026-02-21T17:07:05,061
Flushing.java:157 - Writing
Memtable-table_testexportingspecificdictionary_strateg_18@1268950324(61.523KiB
serialized bytes, 1000 ops, 506.836KiB (0%) on-heap, 0B (0%) off-heap), flushed
range = [min(-9223372036854775808), max(9223372036854775807))
INFO [PerDiskMemtableFlushWriter_0:1] 2026-02-21T17:07:05,061
Flushing.java:197 - Completed flushing
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-11-big-Data.db
(28.198KiB) for commitlog position CommitLogPosition(segmentId=1771693567140,
position=654098), time spent: 0 ms, bytes flushed: 28875 / (rate: 28.198KiB/s),
partitions flushed: 1000 / (rate: 1000/s), rows: 1000 / (rate: 1000/s), cpu
time: 0 ms, heap allocated: 220.711KiB
INFO [MemtableFlushWriter:1] 2026-02-21T17:07:05,084 LogTransaction.java:266 -
Unfinished transaction log, deleting
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa_txn_flush_bf3ac330-0f47-11f1-88d2-574197b4b378.log
DEBUG [MemtableFlushWriter:1] 2026-02-21T17:07:05,087
ColumnFamilyStore.java:1416 - Flushed to
[BigTableReader:big(path='/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-11-big-Data.db')]
(1 sstables, 30.889KiB), biggest 30.889KiB, smallest 30.889KiB
INFO [main] 2026-02-21T17:07:05,091 ColumnFamilyStore.java:1088 - Enqueuing
flush of cql_test_keyspace.table_testexportingspecificdictionary_strateg_18,
Reason: UNIT_TESTS, Usage: 506.836KiB (0%) on-heap, 0B (0%) off-heap
INFO [PerDiskMemtableFlushWriter_0:2] 2026-02-21T17:07:05,092
Flushing.java:157 - Writing
Memtable-table_testexportingspecificdictionary_strateg_18@957877902(61.523KiB
serialized bytes, 1000 ops, 506.836KiB (0%) on-heap, 0B (0%) off-heap), flushed
range = [min(-9223372036854775808), max(9223372036854775807))
INFO [PerDiskMemtableFlushWriter_0:2] 2026-02-21T17:07:05,094
Flushing.java:197 - Completed flushing
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-12-big-Data.db
(28.201KiB) for commitlog position CommitLogPosition(segmentId=1771693567140,
position=726098), time spent: 0 ms, bytes flushed: 28878 / (rate: 28.201KiB/s),
partitions flushed: 1000 / (rate: 1000/s), rows: 1000 / (rate: 1000/s), cpu
time: 0 ms, heap allocated: 220.711KiB
INFO [MemtableFlushWriter:2] 2026-02-21T17:07:05,112 LogTransaction.java:266 -
Unfinished transaction log, deleting
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa_txn_flush_bf3fa530-0f47-11f1-88d2-574197b4b378.log
DEBUG [MemtableFlushWriter:2] 2026-02-21T17:07:05,116
ColumnFamilyStore.java:1416 - Flushed to
[BigTableReader:big(path='/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-12-big-Data.db')]
(1 sstables, 30.887KiB), biggest 30.887KiB, smallest 30.887KiB
DEBUG [CompactionExecutor:2] 2026-02-21T17:07:05,117 Directories.java:554 -
FileStore /System/Volumes/Data (/dev/disk3s5) has 593792975872 bytes available,
checking if we can write 103847 bytes
INFO [CompactionExecutor:2] 2026-02-21T17:07:05,117 CompactionTask.java:229 -
Compacting (bf4375c0-0f47-11f1-88d2-574197b4b378)
[/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-11-big-Data.db,
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-12-big-Data.db,
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-9-big-Data.db,
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-10-big-Data.db,
]
DEBUG [CompactionExecutor:2] 2026-02-21T17:07:05,118 CursorCompactor.java:152 -
Cursor compaction for table: table_testexportingspecificdictionary_strateg_18
keyspace: cql_test_keyspace is supported.
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:05,146
CommandInvokerService.java:185 - Executing command 'train' with execution ID:
a5960218-7f58-41a2-a06f-d627acf20efd
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:05,147
CompressionDictionaryManager.java:237 - Starting SSTable-based training for
cql_test_keyspace.table_testexportingspecificdictionary_strateg_18 with 1
SSTables
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:05,150
CompressionDictionaryScheduler.java:101 - Starting SSTable-based dictionary
training for cql_test_keyspace.table_testexportingspecificdictionary_strateg_18
from 1 SSTables
DEBUG [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:05,150
CompressionDictionaryScheduler.java:198 - Couldn't acquire reference to SSTable
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-13-big.
It may have been removed.
WARN [NonPeriodicTasks:1] 2026-02-21T17:07:05,150
CompressionDictionaryScheduler.java:213 - No SSTables available for sampling in
cql_test_keyspace.table_testexportingspecificdictionary_strateg_18
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:05,150
ToolRunner.java:927 - >>>> Polling training status...SAMPLING
INFO [CompactionExecutor:2] 2026-02-21T17:07:05,152 CursorCompactor.java:1574
- Compaction ended bf4375c0-0f47-11f1-88d2-574197b4b378: { data bytes read =
294620, data bytes written = 297868, input (keys = [1:10000,] = 10000, rows =
[1:10000,] = 10000, cells = [1:10000,] = 10000), output (keys = 10000, rows =
10000, cells = 10000)}
INFO [CompactionExecutor:2] 2026-02-21T17:07:05,153 CompactionTask.java:336 -
Compacted (bf4375c0-0f47-11f1-88d2-574197b4b378) 4 sstables to
[build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-13-big,]
to level=0. 101.413KiB to 91.637KiB (~90% of original) in 35ms. Read
Throughput = 2.826MiB/s, Write Throughput = 2.554MiB/s, Row Throughput =
~10,000/s. 10,000 total partitions merged to 10,000. Partition merge counts
were {1:10000, }. Time spent writing keys = 10ms
INFO [NonPeriodicTasks:1] 2026-02-21T17:07:05,153 BigFormat.java:324 -
Deleting sstable:
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-11-big
INFO [NonPeriodicTasks:1] 2026-02-21T17:07:05,154 BigFormat.java:324 -
Deleting sstable:
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-12-big
INFO [NonPeriodicTasks:1] 2026-02-21T17:07:05,154 BigFormat.java:324 -
Deleting sstable:
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-9-big
INFO [NonPeriodicTasks:1] 2026-02-21T17:07:05,155 BigFormat.java:324 -
Deleting sstable:
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa-10-big
INFO [NonPeriodicTasks:1] 2026-02-21T17:07:05,155 LogTransaction.java:266 -
Unfinished transaction log, deleting
/Users/maxim.muzafarov/IdeaProjects/cassandra/build/test/cassandra/data/cql_test_keyspace/table_testexportingspecificdictionary_strateg_18-1b255f4def2540a60000000000000056/pa_txn_compaction_bf4375c0-0f47-11f1-88d2-574197b4b378.log
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:06,155
ToolRunner.java:927 - >>>> Polling training status...SAMPLING
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:07,159
ToolRunner.java:927 - >>>> Polling training status...SAMPLING
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:08,163
ToolRunner.java:927 - >>>> Polling training status...SAMPLING
INFO [RMI TCP Connection(26)-127.0.0.1] 2026-02-21T17:07:09,168
ToolRunner.java:927 - >>>> Polling training status...SAMPLING
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
