Hi all,
I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
commitlog file in every restart, but it never delete it. We can't control
the commitlog dir size even
by explicitly setting commitlog_total_space_in_mb in cassandra.yaml.
I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
testing, now my commitlog directory size reached 1.1 GB.
Also I turned on debug log for CommitLog, This is what I get when server
starts.
[2012-08-08 11:25:30,860] INFO
{org.apache.cassandra.config.DatabaseDescriptor} - Global memtable
threshold is enabled at 151MB
[2012-08-08 11:25:31,427] INFO {org.apache.cassandra.service.CacheService}
- Initializing key cache with capacity of 12 MBs.
[2012-08-08 11:25:31,443] INFO {org.apache.cassandra.service.CacheService}
- Scheduling key cache save to each 14400 seconds (going to save all keys).
[2012-08-08 11:25:31,445] INFO {org.apache.cassandra.service.CacheService}
- Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
[2012-08-08 11:25:31,448] INFO {org.apache.cassandra.service.CacheService}
- Scheduling row cache save to each 0 seconds (going to save all keys).
[2012-08-08 11:25:31,677] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
(1644 bytes)
[2012-08-08 11:25:31,677] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
(5967 bytes)
[2012-08-08 11:25:31,784] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
(247 bytes)
[2012-08-08 11:25:31,789] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
(247 bytes)
[2012-08-08 11:25:31,887] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
(242 bytes)
[2012-08-08 11:25:31,897] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
(244 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
(80 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
(346 bytes)
[2012-08-08 11:25:31,935] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
(163 bytes)
[2012-08-08 11:25:32,980] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
(312328 bytes)
[2012-08-08 11:25:32,980] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
(312992 bytes)
[2012-08-08 11:25:32,984] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
(647552 bytes)
[2012-08-08 11:25:33,183] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
(249143 bytes)
[2012-08-08 11:25:33,266] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
(152 bytes)
[2012-08-08 11:25:33,266] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
(150 bytes)
[2012-08-08 11:25:33,362] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
(319 bytes)
[2012-08-08 11:25:33,363] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
(484 bytes)
[2012-08-08 11:25:33,412] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
(163 bytes)
[2012-08-08 11:25:33,422] INFO
{org.apache.cassandra.io.sstable.SSTableReader} - Opening
./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
(161 bytes)
[2012-08-08 11:25:33,447] INFO
{org.apache.cassandra.service.AbstractCassandraDaemon} - completed
pre-loading (6 keys) key cache.
[2012-08-08 11:25:34,227] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,230] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
[2012-08-08 11:25:34,255] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
starting at 0
[2012-08-08 11:25:34,255] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,295] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
,])}
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000,])}
[2012-08-08 11:25:34,322] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180
[2012-08-08 11:25:34,323] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.746872696674: {ColumnFamily(Versions
[version:false:7@1344363219528000,])}
[2012-08-08 11:25:34,323] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274
[2012-08-08 11:25:34,324] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.4c: {ColumnFamily(LocationInfo
[47656e65726174696f6e:false:4@1344363219567000,])}
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.426f6f747374726170: {ColumnFamily(LocationInfo
[42:false:1@1344363219785000,])}
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448
[2012-08-08 11:25:34,325] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
,])}
[2012-08-08 11:25:34,326] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544
[2012-08-08 11:25:34,326] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 548
[2012-08-08 11:25:34,326] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
[2012-08-08 11:25:34,326] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
starting at 0
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,359] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,360] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
[2012-08-08 11:25:34,360] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
[2012-08-08 11:25:34,392] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
starting at 0
[2012-08-08 11:25:34,392] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,393] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,393] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
[2012-08-08 11:25:34,393] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
starting at 0
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,429] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,430] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
[2012-08-08 11:25:34,430] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
[2012-08-08 11:25:34,449] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
starting at 0
[2012-08-08 11:25:34,450] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,450] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,451] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
[2012-08-08 11:25:34,451] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
[2012-08-08 11:25:34,493] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
starting at 0
[2012-08-08 11:25:34,493] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,494] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,494] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
[2012-08-08 11:25:34,494] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Replaying
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
starting at 0
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
[2012-08-08 11:25:34,515] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
marker at 4
[2012-08-08 11:25:34,516] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
[2012-08-08 11:25:34,516] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on
mutations from recovery
[2012-08-08 11:25:34,519] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-Versions@507924927(83/103 serialized/live
bytes, 3 ops)
[2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
[2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
(247 bytes)
[2012-08-08 11:25:34,858] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=0), column
family 7
[2012-08-08 11:25:34,859] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is ; hasNext: false*
[2012-08-08 11:25:34,859] INFO
{org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3
replayed mutations
[2012-08-08 11:25:34,879] INFO
{org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0
[2012-08-08 11:25:34,879] INFO
{org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0
[2012-08-08 11:25:34,881] INFO
{org.apache.cassandra.service.StorageService} - CQL supported versions:
2.0.0,3.0.0-beta1 (default: 2.0.0)
[2012-08-08 11:25:34,904] INFO
{org.apache.cassandra.service.StorageService} - Loading persisted ring
state
[2012-08-08 11:25:34,906] INFO
{org.apache.cassandra.service.StorageService} - Starting up server gossip
[2012-08-08 11:25:34,917] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-LocationInfo@1885256814(29/36
serialized/live bytes, 1 ops)
[2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
ops)
[2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
(80 bytes)
[2012-08-08 11:25:35,183] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=363), column
family 0
[2012-08-08 11:25:35,184] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is Versions (7), ; hasNext: false*
[2012-08-08 11:25:35,192] INFO
{org.apache.cassandra.db.compaction.CompactionTask} - Compacting
[SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
[2012-08-08 11:25:35,390] INFO {org.apache.cassandra.net.MessagingService}
- Starting Messaging Service on port 7000
[2012-08-08 11:25:35,399] INFO
{org.apache.cassandra.service.StorageService} - Using saved token
83957406514855090039949046883112789193
[2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.ColumnFamilyStore}
- Enqueuing flush of Memtable-LocationInfo@1345718067(53/66
serialized/live bytes, 2 ops)
[2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} -
Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
ops)
[2012-08-08 11:25:35,959] INFO
{org.apache.cassandra.db.compaction.CompactionTask} - Compacted to
[./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time:
657ms.
[2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} -
Completed flushing
./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
(163 bytes)
[2012-08-08 11:25:36,002] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
segments for ReplayPosition(segmentId=2233031438605, position=544), column
family 0
[2012-08-08 11:25:36,002] DEBUG
{org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete commit
log
CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
dirty is Versions (7), ; hasNext: false*
[2012-08-08 11:25:36,003] INFO
{org.apache.cassandra.service.StorageService} - Node
localhost/127.0.0.1state jump to normal
[2012-08-08 11:25:36,009] INFO
{org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move
completed! Now serving reads.
[2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} -
Will not load MX4J, mx4j-tools.jar is not in the classpath
[2012-08-08 11:25:36,058] INFO
{org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled
[2012-08-08 11:25:36,173] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to
localhost/127.0.0.1:9160
[2012-08-08 11:25:36,254] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport
with a max frame size of 15728640 bytes.
[2012-08-08 11:25:36,257] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Using
synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
[2012-08-08 11:25:36,258] INFO
{org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra
Server Controller Thread was destroyed successfully
[2012-08-08 11:25:36,258] INFO
{org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift
clients...
What is the reason for keeping old commitlogs eventhough I have
set commitlog_total_space_in_mb in cassandra.yaml?
Why is this printing "Not safe to delete" message?
Appreciate your help on fixing this issue.
Thanks,
Kasun.
