Hi Peter, It's a good question why `log.dir` defaults to `/tmp`. I assume it's to make it easier for people to get started with Kafka, but unsafe defaults should be avoided as much as possible in my opinion.
Ismael On Sat, Jul 2, 2016 at 5:15 AM, Peter Davis <davi...@gmail.com> wrote: > > Dear Community: why does log.dir default under /tmp? It is unsafe as a > default. > > -Peter > > > > On Jun 30, 2016, at 11:19 PM, 黄杰斌 <jben.hu...@gmail.com> wrote: > > > > Hi All, > > > > Do you encounter below issue when using kafka_2.11-0.10.0.0? > > All brokers are crash due to __consumer_offsets folder are deleted. > > sample log: > > [2016-06-30 12:46:32,579] FATAL [Replica Manager on Broker 2]: Halting > due > > to unrecoverable I/O error while handling produce request: > > (kafka.server.ReplicaManager) > > kafka.common.KafkaStorageException: I/O exception in append to log > > '__consumer_offsets-32' > > at kafka.log.Log.append(Log.scala:329) > > at kafka.cluster.Partition$$anonfun$11.apply(Partition.scala:443) > > at kafka.cluster.Partition$$anonfun$11.apply(Partition.scala:429) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231) > > at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:237) > > at > > kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:429) > > at > > > kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:406) > > at > > > kafka.server.ReplicaManager$$anonfun$appendToLocalLog$2.apply(ReplicaManager.scala:392) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > > at > > > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > > at scala.collection.immutable.Map$Map1.foreach(Map.scala:116) > > at > > scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > > at > > kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:392) > > at > > kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:328) > > at > > > kafka.coordinator.GroupMetadataManager.store(GroupMetadataManager.scala:232) > > at > > > kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:424) > > at > > > kafka.coordinator.GroupCoordinator$$anonfun$handleCommitOffsets$9.apply(GroupCoordinator.scala:424) > > at scala.Option.foreach(Option.scala:257) > > at > > > kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:424) > > at > > kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:310) > > at kafka.server.KafkaApis.handle(KafkaApis.scala:84) > > at > > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.io.FileNotFoundException: > > /tmp/kafka2-logs/__consumer_offsets-32/00000000000000000000.index (No > such > > file or directory) > > at java.io.RandomAccessFile.open0(Native Method) > > at java.io.RandomAccessFile.open(RandomAccessFile.java:316) > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) > > at > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286) > > at > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:285) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231) > > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:285) > > at > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(OffsetIndex.scala:274) > > at > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:274) > > at > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:274) > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231) > > at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:273) > > at kafka.log.Log.roll(Log.scala:655) > > at kafka.log.Log.maybeRoll(Log.scala:630) > > at kafka.log.Log.append(Log.scala:383) > > ... 23 more > > > > No one remove those folders, and topic __consumer_offsets is handled by > > broker, no one can remove this topic. > > Do you know why this happened? And how to avoid it? > > > > Best Regards, > > Ben >
