Any idea on how to fix this?
Thanks,
Kasun
On Wed, Aug 8, 2012 at 11:56 AM, Kasun Weranga <kas...@wso2.com> wrote:
> Hi all,
>
> I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB
> commitlog file in every restart, but it never delete it. We can't control
> the commitlog dir size even
> by explicitly setting commitlog_total_space_in_mb in cassandra.yaml.
> I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some
> testing, now my commitlog directory size reached 1.1 GB.
>
> Also I turned on debug log for CommitLog, This is what I get when server
> starts.
>
>
> [2012-08-08 11:25:30,860] INFO
> {org.apache.cassandra.config.DatabaseDescriptor} - Global memtable
> threshold is enabled at 151MB
> [2012-08-08 11:25:31,427] INFO
> {org.apache.cassandra.service.CacheService} - Initializing key cache with
> capacity of 12 MBs.
> [2012-08-08 11:25:31,443] INFO
> {org.apache.cassandra.service.CacheService} - Scheduling key cache save to
> each 14400 seconds (going to save all keys).
> [2012-08-08 11:25:31,445] INFO
> {org.apache.cassandra.service.CacheService} - Initializing row cache with
> capacity of 0 MBs and provider
> org.apache.cassandra.cache.SerializingCacheProvider
> [2012-08-08 11:25:31,448] INFO
> {org.apache.cassandra.service.CacheService} - Scheduling row cache save to
> each 0 seconds (going to save all keys).
> [2012-08-08 11:25:31,677] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6
> (1644 bytes)
> [2012-08-08 11:25:31,677] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5
> (5967 bytes)
> [2012-08-08 11:25:31,784] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-5
> (247 bytes)
> [2012-08-08 11:25:31,789] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-6
> (247 bytes)
> [2012-08-08 11:25:31,887] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1
> (242 bytes)
> [2012-08-08 11:25:31,897] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2
> (244 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14
> (80 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13
> (346 bytes)
> [2012-08-08 11:25:31,935] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15
> (163 bytes)
> [2012-08-08 11:25:32,980] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2
> (312328 bytes)
> [2012-08-08 11:25:32,980] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3
> (312992 bytes)
> [2012-08-08 11:25:32,984] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1
> (647552 bytes)
> [2012-08-08 11:25:33,183] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1
> (249143 bytes)
> [2012-08-08 11:25:33,266] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1
> (152 bytes)
> [2012-08-08 11:25:33,266] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2
> (150 bytes)
> [2012-08-08 11:25:33,362] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2
> (319 bytes)
> [2012-08-08 11:25:33,363] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1
> (484 bytes)
> [2012-08-08 11:25:33,412] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1
> (163 bytes)
> [2012-08-08 11:25:33,422] INFO
> {org.apache.cassandra.io.sstable.SSTableReader} - Opening
> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2
> (161 bytes)
> [2012-08-08 11:25:33,447] INFO
> {org.apache.cassandra.service.AbstractCassandraDaemon} - completed
> pre-loading (6 keys) key cache.
> [2012-08-08 11:25:34,227] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log,
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,230] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> starting at 0
> [2012-08-08 11:25:34,255] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,295] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000
> ,])}
> [2012-08-08 11:25:34,322] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.746872696674: {ColumnFamily(Versions
> [version:false:7@1344363219528000,])}
> [2012-08-08 11:25:34,323] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274
> [2012-08-08 11:25:34,324] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.4c: {ColumnFamily(LocationInfo
> [47656e65726174696f6e:false:4@1344363219567000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.426f6f747374726170: {ColumnFamily(LocationInfo
> [42:false:1@1344363219785000,])}
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448
> [2012-08-08 11:25:34,325] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for
> system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000
> ,])}
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544
> [2012-08-08 11:25:34,326] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 548
> [2012-08-08 11:25:34,326] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log
> [2012-08-08 11:25:34,326] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> starting at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,359] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,360] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log
> [2012-08-08 11:25:34,360] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> starting at 0
> [2012-08-08 11:25:34,392] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,393] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,393] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log
> [2012-08-08 11:25:34,393] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> starting at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,429] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,430] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log
> [2012-08-08 11:25:34,430] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,449] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> starting at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,450] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,451] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log
> [2012-08-08 11:25:34,451] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> starting at 0
> [2012-08-08 11:25:34,493] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,494] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,494] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log
> [2012-08-08 11:25:34,494] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> starting at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0
> [2012-08-08 11:25:34,515] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment
> marker at 4
> [2012-08-08 11:25:34,516] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading
> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log
> [2012-08-08 11:25:34,516] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on
> mutations from recovery
> [2012-08-08 11:25:34,519] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops)
> [2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db
> (247 bytes)
> [2012-08-08 11:25:34,858] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=0), column
> family 7
> [2012-08-08 11:25:34,859] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is ; hasNext: false*
> [2012-08-08 11:25:34,859] INFO
> {org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3
> replayed mutations
> [2012-08-08 11:25:34,879] INFO
> {org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0
> [2012-08-08 11:25:34,879] INFO
> {org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0
> [2012-08-08 11:25:34,881] INFO
> {org.apache.cassandra.service.StorageService} - CQL supported versions:
> 2.0.0,3.0.0-beta1 (default: 2.0.0)
> [2012-08-08 11:25:34,904] INFO
> {org.apache.cassandra.service.StorageService} - Loading persisted ring
> state
> [2012-08-08 11:25:34,906] INFO
> {org.apache.cassandra.service.StorageService} - Starting up server gossip
> [2012-08-08 11:25:34,917] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 ops)
> [2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1
> ops)
> [2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db
> (80 bytes)
> [2012-08-08 11:25:35,183] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=363), column
> family 0
> [2012-08-08 11:25:35,184] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:35,192] INFO
> {org.apache.cassandra.db.compaction.CompactionTask} - Compacting
> [SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'),
> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')]
> [2012-08-08 11:25:35,390] INFO
> {org.apache.cassandra.net.MessagingService} - Starting Messaging Service
> on port 7000
> [2012-08-08 11:25:35,399] INFO
> {org.apache.cassandra.service.StorageService} - Using saved token
> 83957406514855090039949046883112789193
> [2012-08-08 11:25:35,401] INFO
> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of
> Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 ops)
> [2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} -
> Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2
> ops)
> [2012-08-08 11:25:35,959] INFO
> {org.apache.cassandra.db.compaction.CompactionTask} - Compacted to
> [./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,].
> 669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time:
> 657ms.
> [2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} -
> Completed flushing
> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db
> (163 bytes)
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log
> segments for ReplayPosition(segmentId=2233031438605, position=544), column
> family 0
> [2012-08-08 11:25:36,002] DEBUG
> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete
> commit log
> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log);
> dirty is Versions (7), ; hasNext: false*
> [2012-08-08 11:25:36,003] INFO
> {org.apache.cassandra.service.StorageService} - Node
> localhost/127.0.0.1state jump to normal
> [2012-08-08 11:25:36,009] INFO
> {org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move
> completed! Now serving reads.
> [2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} -
> Will not load MX4J, mx4j-tools.jar is not in the classpath
> [2012-08-08 11:25:36,058] INFO
> {org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled
> [2012-08-08 11:25:36,173] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to
> localhost/127.0.0.1:9160
> [2012-08-08 11:25:36,254] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport
> with a max frame size of 15728640 bytes.
> [2012-08-08 11:25:36,257] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Using
> synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160
> [2012-08-08 11:25:36,258] INFO
> {org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra
> Server Controller Thread was destroyed successfully
> [2012-08-08 11:25:36,258] INFO
> {org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift
> clients...
>
>
> What is the reason for keeping old commitlogs eventhough I have
> set commitlog_total_space_in_mb in cassandra.yaml?
> Why is this printing "Not safe to delete" message?
>
>
> Appreciate your help on fixing this issue.
>
>
> Thanks,
> Kasun.
>
