[ https://issues.apache.org/jira/browse/CASSANDRA-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christoph Schnepf updated CASSANDRA-20664: ------------------------------------------ Description: Hi, We're using Cassandra 4.1.8 and specify the option {*}-Dcassandra.commitlog.ignorereplayerrors=true{*}, however we see an endless loop on starting Cassandra when there are corrupt commit log files found. The stacktrace which is printed over and over again is: {code:java} INFO [main] 2025-05-19 19:25:22,658 UTC CommitLogReader.java:257 - Finished reading /data/cassandra/commitlog/CommitLog-7-1745459535901.log INFO [main] 2025-05-19 19:25:23,614 UTC CommitLogReader.java:257 - Finished reading /data/cassandra/commitlog/CommitLog-7-1745459535902.log ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Mutation checksum failure at 60807439 in Next section at 60745241 in CommitLog-7-1745459535903.log at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Mutation size checksum failure at 60838538 in Next section at 60745241 in CommitLog-7-1745459535903.log at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:356) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 60865611 of commit log /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. The end of segment marker should be zero. at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 60865611 of commit log /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. The end of segment marker should be zero. at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error {code} This prevents the Cassandra startup on this node and it writes 50 MB to the system.log in about 2 seconds. was: Hi, We're using Cassandra 4.1.8 and specify the option `-Dcassandra.commitlog.ignorereplayerrors=true`, however we see an endless loop on starting Cassandra when there are corrupt commit log files found. The stacktrace which is printed over and over again is: {code:java} INFO [main] 2025-05-19 19:25:22,658 UTC CommitLogReader.java:257 - Finished reading /data/cassandra/commitlog/CommitLog-7-1745459535901.log INFO [main] 2025-05-19 19:25:23,614 UTC CommitLogReader.java:257 - Finished reading /data/cassandra/commitlog/CommitLog-7-1745459535902.log ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Mutation checksum failure at 60807439 in Next section at 60745241 in CommitLog-7-1745459535903.log at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Mutation size checksum failure at 60838538 in Next section at 60745241 in CommitLog-7-1745459535903.log at org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:356) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 60865611 of commit log /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. The end of segment marker should be zero. at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 60865611 of commit log /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. The end of segment marker should be zero. at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - Ignoring commit log replay error {code} This prevents the Cassandra startup on this node and it writes 50 MB to the system.log in about 2 seconds. > Endless loop on reading commitlogs when it should ignore replay errors > ---------------------------------------------------------------------- > > Key: CASSANDRA-20664 > URL: https://issues.apache.org/jira/browse/CASSANDRA-20664 > Project: Apache Cassandra > Issue Type: Bug > Components: Local/Commit Log > Reporter: Christoph Schnepf > Priority: Normal > > Hi, > We're using Cassandra 4.1.8 and specify the option > {*}-Dcassandra.commitlog.ignorereplayerrors=true{*}, however we see an > endless loop on starting Cassandra when there are corrupt commit log files > found. > The stacktrace which is printed over and over again is: > {code:java} > INFO [main] 2025-05-19 19:25:22,658 UTC CommitLogReader.java:257 - Finished > reading /data/cassandra/commitlog/CommitLog-7-1745459535901.log > INFO [main] 2025-05-19 19:25:23,614 UTC CommitLogReader.java:257 - Finished > reading /data/cassandra/commitlog/CommitLog-7-1745459535902.log > ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - > Ignoring commit log replay error > org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: > Mutation checksum failure at 60807439 in Next section at 60745241 in > CommitLog-7-1745459535903.log > at > org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:387) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) > ERROR [main] 2025-05-19 19:25:24,572 UTC CommitLogReplayer.java:501 - > Ignoring commit log replay error > org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: > Mutation size checksum failure at 60838538 in Next section at 60745241 in > CommitLog-7-1745459535903.log > at > org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:356) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) > ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - > Ignoring commit log replay error > org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: > Encountered bad header at position 60865611 of commit log > /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. > The end of segment marker should be zero. > at > org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) > ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - > Ignoring commit log replay error > org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: > Encountered bad header at position 60865611 of commit log > /data/cassandra/commitlog/CommitLog-7-1745459535903.log, with invalid CRC. > The end of segment marker should be zero. > at > org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:98) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:233) > at > org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:200) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223) > at > org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:353) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:744) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:878) > ERROR [main] 2025-05-19 19:25:24,573 UTC CommitLogReplayer.java:501 - > Ignoring commit log replay error {code} > This prevents the Cassandra startup on this node and it writes 50 MB to the > system.log in about 2 seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org