K B Parthasarathy created KAFKA-6881:
----------------------------------------
Summary: Kafka 1.1 version crashes when deleting log
Key: KAFKA-6881
URL: https://issues.apache.org/jira/browse/KAFKA-6881
Project: Kafka
Issue Type: Bug
Environment: Linux
Reporter: K B Parthasarathy
Hello
We are running Kafka 1.1 version in Linux from past 3 weeks. Today Kafka
crashed. When we checked server.log file the following log was found
[2018-05-07 16:53:06,721] ERROR Failed to clean up log for
__consumer_offsets-24 in dir /tmp/kafka-logs due to IOException
(kafka.server.LogDirFailureChannel)
java.nio.file.NoSuchFileException:
/tmp/kafka-logs/__consumer_offsets-24/00000000000000000000.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
at kafka.log.Log.asyncDeleteSegment(Log.scala:1601)
at kafka.log.Log.$anonfun$replaceSegments$1(Log.scala:1653)
at kafka.log.Log.$anonfun$replaceSegments$1$adapted(Log.scala:1648)
at scala.collection.immutable.List.foreach(List.scala:389)
at kafka.log.Log.replaceSegments(Log.scala:1648)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:535)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:462)
at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:461)
at scala.collection.immutable.List.foreach(List.scala:389)
at kafka.log.Cleaner.doClean(LogCleaner.scala:461)
at kafka.log.Cleaner.clean(LogCleaner.scala:438)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Suppressed: java.nio.file.NoSuchFileException:
/tmp/kafka-logs/__consumer_offsets-24/00000000000000000000.log ->
/tmp/kafka-logs/__consumer_offsets-24/00000000000000000000.log.deleted
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
... 16 more
[2018-05-07 16:53:06,725] INFO [ReplicaManager broker=0] Stopping serving
replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
[2018-05-07 16:53:06,729] INFO [ReplicaFetcherManager on broker 0] Removed
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,herculesdata-0,__consumer_offsets-8,connect-status-Crimson-0,__consumer_offsets-4,herculesdata-6,herculesdataerror-0,herculesdataaccepted-0,__consumer_offsets-46,connect-offsets-Crimson-0,connect-offsets-Hercules-5,connect-status-Hercules-3,__consumer_offsets-16,__consumer_offsets-28,connect-offsets-Hercules-3,__consumer_offsets-36,connect-configs-Hercules-0,__consumer_offsets-42,connect-offsets-Hercules-0,__consumer_offsets-18,connect-status-Hercules-4,__consumer_offsets-24,herculesdata-3,connect-status-Hercules-6,__consumer_offsets-38,__consumer_offsets-48,connect-offsets-Hercules-2,connect-status-Hercules-5,connect-configs-Crimson-0,connect-offsets-Hercules-6,connect-status-Hercules-2,connect-offsets-Hercules-1,__consumer_offsets-2,__consumer_offsets-6,__consumer_offsets-14,herculesdata-5,herculesdata-2,connect-status-Hercules-1,connect-status-Hercules-0,__consumer_offsets-20,herculesdata-1,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-12,connect-offsets-Hercules-4,__consumer_offsets-26,crimsondata-0,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,herculesdata-4,__consumer_offsets-40
(kafka.server.ReplicaFetcherManager)
[2018-05-07 16:53:06,729] INFO [ReplicaAlterLogDirsManager on broker 0] Removed
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,herculesdata-0,__consumer_offsets-8,connect-status-Crimson-0,__consumer_offsets-4,herculesdata-6,herculesdataerror-0,herculesdataaccepted-0,__consumer_offsets-46,connect-offsets-Crimson-0,connect-offsets-Hercules-5,connect-status-Hercules-3,__consumer_offsets-16,__consumer_offsets-28,connect-offsets-Hercules-3,__consumer_offsets-36,connect-configs-Hercules-0,__consumer_offsets-42,connect-offsets-Hercules-0,__consumer_offsets-18,connect-status-Hercules-4,__consumer_offsets-24,herculesdata-3,connect-status-Hercules-6,__consumer_offsets-38,__consumer_offsets-48,connect-offsets-Hercules-2,connect-status-Hercules-5,connect-configs-Crimson-0,connect-offsets-Hercules-6,connect-status-Hercules-2,connect-offsets-Hercules-1,__consumer_offsets-2,__consumer_offsets-6,__consumer_offsets-14,herculesdata-5,herculesdata-2,connect-status-Hercules-1,connect-status-Hercules-0,__consumer_offsets-20,herculesdata-1,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-12,connect-offsets-Hercules-4,__consumer_offsets-26,crimsondata-0,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,herculesdata-4,__consumer_offsets-40
(kafka.server.ReplicaAlterLogDirsManager)
[2018-05-07 16:53:06,762] INFO [ReplicaManager broker=0] Broker 0 stopped
fetcher for partitions
__consumer_offsets-22,__consumer_offsets-30,herculesdata-0,__consumer_offsets-8,connect-status-Crimson-0,__consumer_offsets-4,herculesdata-6,herculesdataerror-0,herculesdataaccepted-0,__consumer_offsets-46,connect-offsets-Crimson-0,connect-offsets-Hercules-5,connect-status-Hercules-3,__consumer_offsets-16,__consumer_offsets-28,connect-offsets-Hercules-3,__consumer_offsets-36,connect-configs-Hercules-0,__consumer_offsets-42,connect-offsets-Hercules-0,__consumer_offsets-18,connect-status-Hercules-4,__consumer_offsets-24,herculesdata-3,connect-status-Hercules-6,__consumer_offsets-38,__consumer_offsets-48,connect-offsets-Hercules-2,connect-status-Hercules-5,connect-configs-Crimson-0,connect-offsets-Hercules-6,connect-status-Hercules-2,connect-offsets-Hercules-1,__consumer_offsets-2,__consumer_offsets-6,__consumer_offsets-14,herculesdata-5,herculesdata-2,connect-status-Hercules-1,connect-status-Hercules-0,__consumer_offsets-20,herculesdata-1,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-12,connect-offsets-Hercules-4,__consumer_offsets-26,crimsondata-0,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-32,herculesdata-4,__consumer_offsets-40
and stopped moving logs for partitions because they are in the failed log
directory /tmp/kafka-logs. (kafka.server.ReplicaManager)
[2018-05-07 16:53:06,762] INFO Stopping serving logs in dir /tmp/kafka-logs
(kafka.log.LogManager)
[2018-05-07 16:53:07,032] ERROR Shutdown broker because all log dirs in
/tmp/kafka-logs have failed (kafka.log.LogManager)
Please let me know what may be the issue
Partha
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
