Zachary Schneider created CASSANDRA-14585:
---------------------------------------------
Summary: For compactions to progress when pending is greater than
10K cassandra must be continuously restarted and compactions_in_progress purged
Key: CASSANDRA-14585
URL: https://issues.apache.org/jira/browse/CASSANDRA-14585
Project: Cassandra
Issue Type: Bug
Components: Compaction
Environment: Apache Cassandra 2.2.12
Reporter: Zachary Schneider
We are experiencing the following behavior when utilizing DTCS on nodes that
join the ring and subsequently have many pending compactions.
The symptoms are the following:
nodetool compactionstats reports compactions progressing, however pending count
is never significantly reduced.
When investigating we notice the following:
compactions_in_progress is never compacted
{code:java}
find
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/
| wc -l
560730
{code}
Eventually we will end up running into to many open files.
In order for compaction to progress we must:
* truncate compactions_in_progress
* restart Cassandra
This compactions will progress rapidly 2K or so and eventually the process will
hang again.
In the debug log we see constant flushes to compactions_in_progress.
{code:java}
DEBUG [CompactionExecutor:10] 2018-07-24 07:39:29,552
ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428
(0%) on-heap, 354 (0%) off-heap INFO [MemtableFlushWriter:9] 2018-07-24
07:39:29,552 Memtable.java:352 - Writing
Memtable-compactions_in_progress@1407915569(0.263KiB serialized bytes, 12 ops,
0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:9] 2018-07-24
07:39:29,553 Memtable.java:386 - Completed flushing
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8140-big-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
position=28982298) DEBUG [CompactionExecutor:16] 2018-07-24 07:39:29,595
ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
(0%) on-heap, 397 (0%) off-heap INFO [MemtableFlushWriter:13] 2018-07-24
07:39:29,595 Memtable.java:352 - Writing
Memtable-compactions_in_progress@535072625(0.298KiB serialized bytes, 13 ops,
0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:13] 2018-07-24
07:39:29,595 Memtable.java:386 - Completed flushing
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8141-big-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
position=29708684) DEBUG [CompactionExecutor:15] 2018-07-24 07:39:29,619
ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428
(0%) on-heap, 356 (0%) off-heap INFO [MemtableFlushWriter:12] 2018-07-24
07:39:29,619 Memtable.java:352 - Writing
Memtable-compactions_in_progress@319301851(0.265KiB serialized bytes, 12 ops,
0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:12] 2018-07-24
07:39:29,620 Memtable.java:386 - Completed flushing
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8142-big-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
position=30591989) DEBUG [CompactionExecutor:9] 2018-07-24 07:39:29,660
ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
(0%) on-heap, 396 (0%) off-heap INFO [MemtableFlushWriter:15] 2018-07-24
07:39:29,660 Memtable.java:352 - Writing
Memtable-compactions_in_progress@1000370375(0.297KiB serialized bytes, 13 ops,
0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:15] 2018-07-24
07:39:29,660 Memtable.java:386 - Completed flushing
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8143-big-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
position=31558471) DEBUG [CompactionExecutor:14] 2018-07-24 07:39:29,688
ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460
(0%) on-heap, 389 (0%) off-heap INFO [MemtableFlushWriter:10] 2018-07-24
07:39:29,689 Memtable.java:352 - Writing
Memtable-compactions_in_progress@204811984(0.290KiB serialized bytes, 13 ops,
0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:10] 2018-07-24
07:39:29,689 Memtable.java:386 - Completed flushing
/mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8144-big-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699,
position=32153206)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
