Tyler Schade created CASSANDRA-20665:
----------------------------------------
Summary: Cassandra 5.0.4 crashing repeatedly on restart when last
Key: CASSANDRA-20665
URL: https://issues.apache.org/jira/browse/CASSANDRA-20665
Project: Apache Cassandra
Issue Type: Bug
Reporter: Tyler Schade
Noticing some strange behavior today with a Cassandra 5.0.4 Docker Container
that recently likely encountered a forced shutdown. This is a single-node
"cluster".
{noformat}
INFO [main] 2025-05-20 14:49:37,065 CommitLogReader.java:103 - Skipping
playback of empty log: CommitLog-7-1747752126176.logINFO [main] 2025-05-20
14:49:37,065 CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752190016.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752253591.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752317293.logINFO [main] 2025-05-20 14:49:37,065
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752380740.logINFO [main] 2025-05-20 14:49:37,066
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752445680.logINFO [main] 2025-05-20 14:49:37,066
CommitLogReader.java:103 - Skipping playback of empty log:
CommitLog-7-1747752511283.logERROR [main] 2025-05-20 14:49:37,067
JVMStabilityInspector.java:201 - Exiting due to error while processing commit
log during
initialization.org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
Could not read commit log descriptor in file
/opt/cassandra/data/commitlog/CommitLog-7-1747377011833.log at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:196)
at
org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
at
org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:205)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:229)
at
org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:208)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865){noformat}
I investigated a little with some help from [~jjirsa] and found that the
segment referenced was empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# hexdump
CommitLog-7-1747377011833.log
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
2000000{noformat}
The subsequent segment was _also_ empty:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# md5sum $(ls | head)
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011833.log
58f06dd588d8ffb3beb46ada6309436b CommitLog-7-1747377011834.log
1f9bd5ab1f623fd921282713bf3e34cc CommitLog-7-1747377250204.log
08ed83631a94bdb902ddeb600e8f562a CommitLog-7-1747377250205.log
1093718121a69c2843dd6ed204dc6563 CommitLog-7-1747377266512.log
2a48581389c3bb68a58be3d0d14eb6f5 CommitLog-7-1747377266513.log
24ad307b55b8d5d1d33ddcc4e21395e8 CommitLog-7-1747377272735.log
1e4a7628f5f141c587d2cc94eda3c22b CommitLog-7-1747377272736.log
06f498bc5d35e2691761e39e0a6d5476 CommitLog-7-1747377276330.log
10f9e64659b333ae3b8b3e9424a4468b CommitLog-7-1747377276331.log{noformat}
However, there are 8 subsequent segments with data:
{noformat}
root@09f46103e3e2:/var/lib/cassandra/commitlog# ls -l | head
total 143136
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011833.log
-rw-r--r-- 1 999 999 33554432 May 16 06:30 CommitLog-7-1747377011834.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250204.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377250205.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266512.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377266513.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272735.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34 CommitLog-7-1747377272736.log
-rw-r--r-- 1 999 999 33554432 May 16 06:34
CommitLog-7-1747377276330.log{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
