There have been some commitlog-related fixes in later versions of 1.1, so it's worth trying an upgrade. If that doesn't resolve the issue, open a JIRA ticket with these details.
On Thu, Aug 9, 2012 at 9:15 AM, Kasun Weranga <kas...@wso2.com> wrote: > Any idea on how to fix this? > > Thanks, > Kasun > > > On Wed, Aug 8, 2012 at 11:56 AM, Kasun Weranga <kas...@wso2.com> wrote: > >> Hi all, >> >> I am facing the above issue in Cassandra 1.1.0, it will add 134.2MB >> commitlog file in every restart, but it never delete it. We can't control >> the commitlog dir size even >> by explicitly setting commitlog_total_space_in_mb in cassandra.yaml. >> I set commitlog_total_space_in_mb as 512 in cassandra.yaml and did some >> testing, now my commitlog directory size reached 1.1 GB. >> >> Also I turned on debug log for CommitLog, This is what I get when server >> starts. >> >> >> [2012-08-08 11:25:30,860] INFO >> {org.apache.cassandra.config.DatabaseDescriptor} - Global memtable >> threshold is enabled at 151MB >> [2012-08-08 11:25:31,427] INFO >> {org.apache.cassandra.service.CacheService} - Initializing key cache with >> capacity of 12 MBs. >> [2012-08-08 11:25:31,443] INFO >> {org.apache.cassandra.service.CacheService} - Scheduling key cache save to >> each 14400 seconds (going to save all keys). >> [2012-08-08 11:25:31,445] INFO >> {org.apache.cassandra.service.CacheService} - Initializing row cache with >> capacity of 0 MBs and provider >> org.apache.cassandra.cache.SerializingCacheProvider >> [2012-08-08 11:25:31,448] INFO >> {org.apache.cassandra.service.CacheService} - Scheduling row cache save to >> each 0 seconds (going to save all keys). >> [2012-08-08 11:25:31,677] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-6 >> (1644 bytes) >> [2012-08-08 11:25:31,677] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hc-5 >> (5967 bytes) >> [2012-08-08 11:25:31,784] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-5 >> (247 bytes) >> [2012-08-08 11:25:31,789] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-6 >> (247 bytes) >> [2012-08-08 11:25:31,887] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-1 >> (242 bytes) >> [2012-08-08 11:25:31,897] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hc-2 >> (244 bytes) >> [2012-08-08 11:25:31,935] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14 >> (80 bytes) >> [2012-08-08 11:25:31,935] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13 >> (346 bytes) >> [2012-08-08 11:25:31,935] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15 >> (163 bytes) >> [2012-08-08 11:25:32,980] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-2 >> (312328 bytes) >> [2012-08-08 11:25:32,980] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-3 >> (312992 bytes) >> [2012-08-08 11:25:32,984] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/EVENT_KS/org_wso2_sample_httpd_logs/EVENT_KS-org_wso2_sample_httpd_logs-hc-1 >> (647552 bytes) >> [2012-08-08 11:25:33,183] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/EVENT_KS/bam_service_data_publisher/EVENT_KS-bam_service_data_publisher-hc-1 >> (249143 bytes) >> [2012-08-08 11:25:33,266] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-1 >> (152 bytes) >> [2012-08-08 11:25:33,266] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION_ID_TO_KEY/META_KS-STREAM_DEFINITION_ID_TO_KEY-hc-2 >> (150 bytes) >> [2012-08-08 11:25:33,362] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-2 >> (319 bytes) >> [2012-08-08 11:25:33,363] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_DEFINITION/META_KS-STREAM_DEFINITION-hc-1 >> (484 bytes) >> [2012-08-08 11:25:33,412] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-1 >> (163 bytes) >> [2012-08-08 11:25:33,422] INFO >> {org.apache.cassandra.io.sstable.SSTableReader} - Opening >> ./repository/database/cassandra/data/META_KS/STREAM_ID/META_KS-STREAM_ID-hc-2 >> (161 bytes) >> [2012-08-08 11:25:33,447] INFO >> {org.apache.cassandra.service.AbstractCassandraDaemon} - completed >> pre-loading (6 keys) key cache. >> [2012-08-08 11:25:34,227] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log, >> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log >> [2012-08-08 11:25:34,230] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log >> [2012-08-08 11:25:34,255] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log >> starting at 0 >> [2012-08-08 11:25:34,255] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,295] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.6275696c64: {ColumnFamily(Versions [version:false:5@1344363219522000 >> ,])} >> [2012-08-08 11:25:34,322] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 91 >> [2012-08-08 11:25:34,322] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.63716c: {ColumnFamily(Versions [version:false:5@1344363219527000 >> ,])} >> [2012-08-08 11:25:34,322] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 180 >> [2012-08-08 11:25:34,323] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.746872696674: {ColumnFamily(Versions >> [version:false:7@1344363219528000,])} >> [2012-08-08 11:25:34,323] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 274 >> [2012-08-08 11:25:34,324] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.4c: {ColumnFamily(LocationInfo >> [47656e65726174696f6e:false:4@1344363219567000,])} >> [2012-08-08 11:25:34,325] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 363 >> [2012-08-08 11:25:34,325] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.426f6f747374726170: {ColumnFamily(LocationInfo >> [42:false:1@1344363219785000,])} >> [2012-08-08 11:25:34,325] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 448 >> [2012-08-08 11:25:34,325] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - replaying mutation for >> system.4c: {ColumnFamily(LocationInfo [546f6b656e:false:16@1344363219786000 >> ,])} >> [2012-08-08 11:25:34,326] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 544 >> [2012-08-08 11:25:34,326] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 548 >> [2012-08-08 11:25:34,326] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809612463142.log >> [2012-08-08 11:25:34,326] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log >> [2012-08-08 11:25:34,359] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log >> starting at 0 >> [2012-08-08 11:25:34,359] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,359] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,360] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986125630.log >> [2012-08-08 11:25:34,360] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log >> [2012-08-08 11:25:34,392] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log >> starting at 0 >> [2012-08-08 11:25:34,392] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,393] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,393] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986875007.log >> [2012-08-08 11:25:34,393] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log >> [2012-08-08 11:25:34,429] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log >> starting at 0 >> [2012-08-08 11:25:34,429] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,429] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,430] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986654744.log >> [2012-08-08 11:25:34,430] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log >> [2012-08-08 11:25:34,449] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log >> starting at 0 >> [2012-08-08 11:25:34,450] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,450] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,451] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986776035.log >> [2012-08-08 11:25:34,451] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log >> [2012-08-08 11:25:34,493] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log >> starting at 0 >> [2012-08-08 11:25:34,493] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,494] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,494] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986520160.log >> [2012-08-08 11:25:34,494] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log >> [2012-08-08 11:25:34,515] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Replaying >> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log >> starting at 0 >> [2012-08-08 11:25:34,515] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Reading mutation at 0 >> [2012-08-08 11:25:34,515] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Encountered end of segment >> marker at 4 >> [2012-08-08 11:25:34,516] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished reading >> ./repository/database/cassandra/commitlog/CommitLog-15809986412471.log >> [2012-08-08 11:25:34,516] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - Finished waiting on >> mutations from recovery >> [2012-08-08 11:25:34,519] INFO >> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of >> Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops) >> [2012-08-08 11:25:34,520] INFO {org.apache.cassandra.db.Memtable} - >> Writing Memtable-Versions@507924927(83/103 serialized/live bytes, 3 ops) >> [2012-08-08 11:25:34,840] INFO {org.apache.cassandra.db.Memtable} - >> Completed flushing >> ./repository/database/cassandra/data/system/Versions/system-Versions-hc-7-Data.db >> (247 bytes) >> [2012-08-08 11:25:34,858] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log >> segments for ReplayPosition(segmentId=2233031438605, position=0), column >> family 7 >> [2012-08-08 11:25:34,859] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete >> commit log >> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log); >> dirty is ; hasNext: false* >> [2012-08-08 11:25:34,859] INFO >> {org.apache.cassandra.db.commitlog.CommitLog} - Log replay complete, 3 >> replayed mutations >> [2012-08-08 11:25:34,879] INFO >> {org.apache.cassandra.service.StorageService} - Cassandra version: 1.1.0 >> [2012-08-08 11:25:34,879] INFO >> {org.apache.cassandra.service.StorageService} - Thrift API version: 19.30.0 >> [2012-08-08 11:25:34,881] INFO >> {org.apache.cassandra.service.StorageService} - CQL supported versions: >> 2.0.0,3.0.0-beta1 (default: 2.0.0) >> [2012-08-08 11:25:34,904] INFO >> {org.apache.cassandra.service.StorageService} - Loading persisted ring >> state >> [2012-08-08 11:25:34,906] INFO >> {org.apache.cassandra.service.StorageService} - Starting up server gossip >> [2012-08-08 11:25:34,917] INFO >> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of >> Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 ops) >> [2012-08-08 11:25:34,918] INFO {org.apache.cassandra.db.Memtable} - >> Writing Memtable-LocationInfo@1885256814(29/36 serialized/live bytes, 1 >> ops) >> [2012-08-08 11:25:35,182] INFO {org.apache.cassandra.db.Memtable} - >> Completed flushing >> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db >> (80 bytes) >> [2012-08-08 11:25:35,183] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log >> segments for ReplayPosition(segmentId=2233031438605, position=363), column >> family 0 >> [2012-08-08 11:25:35,184] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - * Not safe to delete >> commit log >> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log); >> dirty is Versions (7), ; hasNext: false* >> [2012-08-08 11:25:35,192] INFO >> {org.apache.cassandra.db.compaction.CompactionTask} - Compacting >> [SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-13-Data.db'), >> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-14-Data.db'), >> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-16-Data.db'), >> SSTableReader(path='./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-15-Data.db')] >> [2012-08-08 11:25:35,390] INFO >> {org.apache.cassandra.net.MessagingService} - Starting Messaging Service >> on port 7000 >> [2012-08-08 11:25:35,399] INFO >> {org.apache.cassandra.service.StorageService} - Using saved token >> 83957406514855090039949046883112789193 >> [2012-08-08 11:25:35,401] INFO >> {org.apache.cassandra.db.ColumnFamilyStore} - Enqueuing flush of >> Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 ops) >> [2012-08-08 11:25:35,401] INFO {org.apache.cassandra.db.Memtable} - >> Writing Memtable-LocationInfo@1345718067(53/66 serialized/live bytes, 2 >> ops) >> [2012-08-08 11:25:35,959] INFO >> {org.apache.cassandra.db.compaction.CompactionTask} - Compacted to >> [./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-17-Data.db,]. >> 669 to 346 (~51% of original) bytes for 3 keys at 0.000502MB/s. Time: >> 657ms. >> [2012-08-08 11:25:36,001] INFO {org.apache.cassandra.db.Memtable} - >> Completed flushing >> ./repository/database/cassandra/data/system/LocationInfo/system-LocationInfo-hc-18-Data.db >> (163 bytes) >> [2012-08-08 11:25:36,002] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - discard completed log >> segments for ReplayPosition(segmentId=2233031438605, position=544), column >> family 0 >> [2012-08-08 11:25:36,002] DEBUG >> {org.apache.cassandra.db.commitlog.CommitLog} - *Not safe to delete >> commit log >> CommitLogSegment(./repository/database/cassandra/commitlog/CommitLog-2233031438605.log); >> dirty is Versions (7), ; hasNext: false* >> [2012-08-08 11:25:36,003] INFO >> {org.apache.cassandra.service.StorageService} - Node >> localhost/127.0.0.1state jump to normal >> [2012-08-08 11:25:36,009] INFO >> {org.apache.cassandra.service.StorageService} - Bootstrap/Replace/Move >> completed! Now serving reads. >> [2012-08-08 11:25:36,010] INFO {org.apache.cassandra.utils.Mx4jTool} - >> Will not load MX4J, mx4j-tools.jar is not in the classpath >> [2012-08-08 11:25:36,058] INFO >> {org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled >> [2012-08-08 11:25:36,173] INFO >> {org.apache.cassandra.thrift.CassandraDaemon} - Binding thrift service to >> localhost/127.0.0.1:9160 >> [2012-08-08 11:25:36,254] INFO >> {org.apache.cassandra.thrift.CassandraDaemon} - Using TFastFramedTransport >> with a max frame size of 15728640 bytes. >> [2012-08-08 11:25:36,257] INFO >> {org.apache.cassandra.thrift.CassandraDaemon} - Using >> synchronous/threadpool thrift server on localhost/127.0.0.1 : 9160 >> [2012-08-08 11:25:36,258] INFO >> {org.wso2.carbon.cassandra.server.CassandraServerController} - Cassandra >> Server Controller Thread was destroyed successfully >> [2012-08-08 11:25:36,258] INFO >> {org.apache.cassandra.thrift.CassandraDaemon} - Listening for thrift >> clients... >> >> >> What is the reason for keeping old commitlogs eventhough I have >> set commitlog_total_space_in_mb in cassandra.yaml? >> Why is this printing "Not safe to delete" message? >> >> >> Appreciate your help on fixing this issue. >> >> >> Thanks, >> Kasun. >> > > > > -- Tyler Hobbs DataStax <http://datastax.com/>