I am seeing the following exception in one of the broker log files.
Set up contains 3 brokers.
Environment - Windows
I am ok to remove the files c:\tmp directory. However, I'm a little curious
to know why this broker got into this state and if there is a way to
rectify the issue without deleting the directory in question
log4j:ERROR Failed to rename [C:\confluent-5.5.0/logs/log-cleaner.log] to
[C:\confluent-5.5.0/logs/log-cleaner.log.2020-06-18-09].
[2020-06-18 14:10:41,361] ERROR Failed to clean up log for
__consumer_offsets-10 in dir C:\tmp\kafka-logs-3 due to IOException (kafka.s
erver.LogDirFailureChannel)
java.nio.file.FileSystemException:
C:\tmp\kafka-logs-3\__consumer_offsets-10\00000000000000000000.timeindex.cleaned
-> C:\tmp\kafka-log
s-3\__consumer_offsets-10\00000000000000000000.timeindex.swap: The process
cannot access the file because it is being used by another p
rocess.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:834)
at kafka.log.AbstractIndex.renameTo(AbstractIndex.scala:207)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:497)
at kafka.log.Log.$anonfun$replaceSegments$4(Log.scala:2288)
at kafka.log.Log.$anonfun$replaceSegments$4$adapted(Log.scala:2288)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.log.Log.replaceSegments(Log.scala:2288)
at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:605)
at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:530)
at kafka.log.Cleaner.doClean(LogCleaner.scala:529)
at kafka.log.Cleaner.clean(LogCleaner.scala:503)
at
kafka.log.LogCleaner$CleanerThread.cleanLog(LogCleaner.scala:372)
at
kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:345)
at
kafka.log.LogCleaner$CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:325)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:314)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Suppressed: java.nio.file.FileSystemException:
C:\tmp\kafka-logs-3\__consumer_offsets-10\00000000000000000000.timeindex.cleaned
->
C:\tmp\kafka-logs-3\__consumer_offsets-10\00000000000000000000.timeindex.swap:
The process cannot access the file because it is bei
ng used by another process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:831)
... 15 more
[2020-06-18 14:10:41,441] WARN [ReplicaManager broker=3] Stopping serving
replicas in dir C:\tmp\kafka-logs-3 (kafka.server.ReplicaMana
ger)
[2020-06-18 14:10:41,445] INFO [ReplicaFetcherManager on broker 3] Removed
fetcher for partitions Set(__consumer_offsets-22, __consumer
_offsets-4, stock-prices-2, __consumer_offsets-7, __consumer_offsets-46,
stock-prices-1, __consumer_offsets-25, __consumer_offsets-49,
__consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31,
__consumer_offsets-37, stock-prices-0, __consumer_offsets-19, stoc
k_topic-0, __consumer_offsets-13, __consumer_offsets-43,
__consumer_offsets-1, __consumer_offsets-34, __consumer_offsets-10,
__consumer
_offsets-40) (kafka.server.ReplicaFetcherManager)
[2020-06-18 14:10:41,448] INFO [ReplicaAlterLogDirsManager on broker 3]
Removed fetcher for partitions Set(__consumer_offsets-22, __con
sumer_offsets-4, stock-prices-2, __consumer_offsets-7,
__consumer_offsets-46, stock-prices-1, __consumer_offsets-25,
__consumer_offsets
-49, __consumer_offsets-16, __consumer_offsets-28, __consumer_offsets-31,
__consumer_offsets-37, stock-prices-0, __consumer_offsets-19,
stock_topic-0, __consumer_offsets-13, __consumer_offsets-43,
__consumer_offsets-1, __consumer_offsets-34, __consumer_offsets-10, __con
sumer_offsets-40) (kafka.server.ReplicaAlterLogDirsManager)
[2020-06-18 14:10:41,492] WARN [ReplicaManager broker=3] Broker 3 stopped
fetcher for partitions __consumer_offsets-22,__consumer_offse
ts-4,stock-prices-2,__consumer_offsets-7,__consumer_offsets-46,stock-prices-1,__consumer_offsets-25,__consumer_offsets-49,__consumer_of
fsets-16,__consumer_offsets-28,__consumer_offsets-31,__consumer_offsets-37,stock-prices-0,__consumer_offsets-19,stock_topic-0,__consume
r_offsets-13,__consumer_offsets-43,__consumer_offsets-1,__consumer_offsets-34,__consumer_offsets-10,__consumer_offsets-40
and stopped m
oving logs for partitions because they are in the failed log directory
C:\tmp\kafka-logs-3. (kafka.server.ReplicaManager)
[2020-06-18 14:10:41,494] WARN Stopping serving logs in dir
C:\tmp\kafka-logs-3 (kafka.log.LogManager)
[2020-06-18 14:10:41,576] ERROR Shutdown broker because all log dirs in
C:\tmp\kafka-logs-3 have failed (kafka.log.LogManager)
