For #1, one potential cause is that someone was consuming a wrong topic
name (e.g., datapoints-wr/ite-34). In 0.7, this will accidentally create a
directory datapoints-wr in the kafka log root dir. The problem has been
fixed in 0.8 by restricting the topic names to alpha-numeric plus "-" and
"_".

Thanks,

Jun


On Fri, Jul 19, 2013 at 7:45 AM, Blake Smith <blake.sm...@tempo-db.com>wrote:

> Just to close the loop here for posterity:
>
> 1. For the directory topic name corruption, it looks like there's still an
> outstanding issue in JIRA: https://issues.apache.org/jira/browse/KAFKA-411
> 2. Ensuring log recovery is run seems to be fixed in
> commit 75fc5eab35aa33cffd9c09a2070dfe287db0ef4e (
> https://issues.apache.org/jira/browse/KAFKA-188)
>
> Thanks for your help Jay, it looks like we'll have to start the 0.7.1 ->
> 0.8 upgrade process sooner than I thought.
>
>
> On Wed, Jul 17, 2013 at 12:32 PM, Blake Smith <blake.sm...@tempo-db.com
> >wrote:
>
> > 1. Cool. For now I'll write this off as a fluke side-effect of a unsafe
> > hardware shutdown. If I come up with a reproducible test case, I'll send
> it
> > (or a patch) your way.
> >
> > 2. After the process came up after the crash, recovery tried to run on
> > each partition, but because of the malformed directory issue it looks
> like
> > recovery never ran on partition 11 (the log with the corrupt message).
> >
> > 40472 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-33/00000000093417473922.kafka in mutable
> mode,
> > recovery true
> > 44032 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:199897864
> > 44032 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 3
> > seconds. 0 bytes truncated.
> > 44033 [main] INFO  kafka.log.LogManager  - Loading log
> > 'datapoints-write-35'
> > 44044 [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-35/00000000092880891040.kafka in mutable
> mode,
> > recovery true
> > 48836 [main] INFO  kafka.message.FileMessageSet  - recover high water
> > mark:268123698
> > 48836 [main] INFO  kafka.message.FileMessageSet  - Recovery succeeded in
> 4
> > seconds. 0 bytes truncated.
> > 48859 [main] INFO  kafka.log.LogManager  - Loading log 'datapoints-wr'
> > 48911 [main] FATAL kafka.server.KafkaServerStartable  - Fatal error
> during
> > KafkaServerStable startup. Prepare to shutdown
> > java.lang.NumberFormatException: For input string: "wr"
> >         at
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >         at java.lang.Integer.parseInt(Integer.java:481)
> >         at java.lang.Integer.parseInt(Integer.java:514)
> >         at
> > scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> >         at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> >         at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
> >         at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
> >         at
> >
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> >         at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> >         at kafka.log.LogManager.<init>(LogManager.scala:65)
> >         at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
> >         at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> >         at kafka.Kafka$.main(Kafka.scala:50)
> >         at kafka.Kafka.main(Kafka.scala)
> > 48912 [main] INFO  kafka.server.KafkaServer  - Shutting down Kafka server
> > 48913 [main] INFO  kafka.utils.KafkaScheduler  - shutdown scheduler
> > kafka-logcleaner-
> > 48913 [main] INFO  kafka.server.KafkaServer  - Kafka server shut down
> > completed
> > 0    [main] INFO  kafka.utils.Utils$  - The number of partitions for
> topic
> >  datapoints-write : 50
> > 2    [main] INFO  kafka.utils.Utils$  - The number of partitions for
> topic
> >  datapoints-increment : 50
> > 2    [main] INFO  kafka.utils.Utils$  - The number of partitions for
> topic
> >  series-index : 5
> > 5    [main] INFO  kafka.server.KafkaServer  - Starting Kafka server...
> > 17   [main] INFO  kafka.log.LogManager  - Loading log
> 'datapoints-write-18'
> > 32   [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-18/00000000095029068903.kafka in mutable
> mode,
> > recovery false
> > 46   [main] INFO  kafka.log.LogManager  - Loading log
> > 'datapoints-increment-15'
> > 46   [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-increment-15/00000000000000000000.kafka in mutable
> > mode, recovery false
> > 47   [main] INFO  kafka.log.LogManager  - Loading log
> > 'datapoints-increment-25'
> > 47   [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-increment-25/00000000000000000000.kafka in mutable
> > mode, recovery false
> > 47   [main] INFO  kafka.log.LogManager  - Loading log
> 'datapoints-write-27'
> > 48   [main] INFO  kafka.log.Log  - Loading the last segment
> > /var/kafka/datapoints-write-27/00000000093417873484.kafka in mutable
> mode,
> > recovery false
> >
> > How is recovery state tracked? From the behavior I'm seeing, it looks
> like
> > it is only triggered on the first restart after an unclean shutdown.
> >
> > Blake
> >
> >
> > On Tue, Jul 16, 2013 at 6:52 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
> >
> >> 1. I don't think this is something kafka can do. We never rename
> >> directories once we create them. I'm a little at a loss as to how/why
> the
> >> os would do it either, though.
> >> 2. Can you check if it ran recovery or not (you would see a bunch of
> >> messages about recovering each topic). This process goes through each
> >> message sequentially and checks the CRC. If that did happen it is
> unlikely
> >> that there was a problem in the log itself.
> >>
> >> -Jay
> >>
> >>
> >> On Tue, Jul 16, 2013 at 3:04 PM, Blake Smith <blake.sm...@tempo-db.com
> >> >wrote:
> >>
> >> > Thanks for the response Jay,
> >> >
> >> > 1. Do you think the corrupt partition directory names is happening at
> >> the
> >> > kafka level, or the file system level? If it's at the file system
> level,
> >> > perhaps we can investigate either changing FS options or selecting a
> >> > different filesystem for the log files. Is there a recommended
> >> filesystem /
> >> > filesystem options?
> >> >
> >> > 2. I'm a little confused about terminology, but from what I observed
> the
> >> > disk corruption was in the last segment for the partition. At the
> time I
> >> > investigated last segment file for the bad partition was ~300mb large
> >> and
> >> > the corrupt message was around around 288mb byte offset (Our segment
> >> files
> >> > seem to get rotated at 512mb, which I believe is the default). When
> the
> >> > server came back up, the corrupt last segment file was still being
> >> written
> >> > to beyond the corrupt message. Is this the behavior you're describing?
> >> >
> >> > Blake
> >> >
> >> >
> >> > On Tue, Jul 16, 2013 at 3:29 PM, Jay Kreps <jay.kr...@gmail.com>
> wrote:
> >> >
> >> > > 1. The corrupt directory names is not something we can really
> handle.
> >> The
> >> > > names have to work with zookeeper so if weird characters get
> inserted
> >> I
> >> > > think the best we can do is give an error. I believe we have fixed
> >> the 0
> >> > > byte file problem in 0.8.
> >> > > 2. The assumption for log recovery is that data that has been
> flushed
> >> to
> >> > > disk is stable, but data which has not been flushed can be in any
> >> state
> >> > > (garbage bytes, missing, partial writes, etc). Recovery runs only on
> >> the
> >> > > last segment of unflushed data (e.g. the file with the highest
> number)
> >> > and
> >> > > only in the case of an unclean shutdown. This is basically an
> >> > > optimization--if we recover all the log on startup startup can take
> >> hours
> >> > > for large logs (e.g. 5TB at say 10MB/sec). From what you are saying
> it
> >> > > sounds like you had disk corruption prior to the last segment--in
> >> other
> >> > > words flushed data became corrupt. This is not handled in 0.7. In
> 0.8
> >> you
> >> > > would have the option of just deleting the problematic data and
> >> restoring
> >> > > from replicas.
> >> > >
> >> > > -Jay
> >> > >
> >> > >
> >> > > On Tue, Jul 16, 2013 at 1:10 PM, Blake Smith <
> >> blake.sm...@tempo-db.com
> >> > > >wrote:
> >> > >
> >> > > > Hi everyone,
> >> > > >
> >> > > > Last week, one of our production Kafka 0.7.1 servers had a
> hardware
> >> > > failure
> >> > > > that resulted in an unclean restart. When the server came back up
> 5
> >> > > minutes
> >> > > > later, there were two topic corruption problems that we had to
> >> handle
> >> > to
> >> > > > get the pipeline working again.
> >> > > >
> >> > > > 1. The kafka log directory had malformed partition directory names
> >> [1].
> >> > > The
> >> > > > partition directories were shortened names of our topic naming
> >> scheme.
> >> > > > Inside some of these directories was a 0 byte .kafka file. This
> >> > prevented
> >> > > > the kafka server process from starting. To resolve this, we
> manually
> >> > > > removed the malformed topic directories and the empty log files.
> >> > > >
> >> > > > Is this a known issue? If so, has it been addressed in 0.8? This
> >> seems
> >> > > like
> >> > > > a bug.
> >> > > >
> >> > > > 2. One of the partitions had a corrupt log message (crc32 check
> >> > failure)
> >> > > > that prevented consumers from advancing on that partition [2]. To
> >> > resolve
> >> > > > this issue we had to hexdump the log file seeking to the byte
> offset
> >> > > > provided by the consumer exception and remove the corrupt message
> at
> >> > the
> >> > > > frame boundary. After removing the corrupt message, we created a
> >> task
> >> > to
> >> > > > parse the log file and put it back into our producer / consumer
> >> > pipeline.
> >> > > > The InvalidMessageException never bubbled to our consumer thread,
> >> > > > preventing us from handling this error without manual manipulation
> >> of
> >> > the
> >> > > > log files (our consumer code was catching InvalidMessageException,
> >> but
> >> > it
> >> > > > never reached that point).
> >> > > >
> >> > > > Is a corrupt log file a state that Kafka is supposed to handle, or
> >> is
> >> > our
> >> > > > consumer code supposed to handle it? I'm confused about how we can
> >> > avoid
> >> > > > manually cleaning up log files with a hex editor if we get into
> this
> >> > > state
> >> > > > again.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Blake
> >> > > >
> >> > > > [1] Corrupt partition directories in the broker data directories:
> >> > > >
> >> > > > 0    [main] INFO  kafka.utils.Utils$  - The number of partitions
> for
> >> > > > topic  datapoints-write : 50
> >> > > > 1    [main] INFO  kafka.utils.Utils$  - The number of partitions
> for
> >> > > > topic  datapoints-increment : 50
> >> > > > 2    [main] INFO  kafka.utils.Utils$  - The number of partitions
> for
> >> > > > topic  series-index : 5
> >> > > > 5    [main] INFO  kafka.server.KafkaServer  - Starting Kafka
> >> server...
> >> > > > 17   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-18'
> >> > > > 32   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-18/00000000095029068903.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 46   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-15'
> >> > > > 46   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-15/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 47   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-25'
> >> > > > 47   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-25/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 47   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-27'
> >> > > > 48   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-27/00000000093417873484.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 48   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-9'
> >> > > > 48   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-9/00000000092880448733.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 49   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-8'
> >> > > > 49   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-8/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 49   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-29'
> >> > > > 49   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-29/00000000008053064625.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 50   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-34'
> >> > > > 50   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-34/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 50   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-19'
> >> > > > 51   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-19/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 51   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-16'
> >> > > > 51   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-16/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 51   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-27'
> >> > > > 52   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-27/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 52   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-34'
> >> > > > 52   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-34/00000000092880904403.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 53   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-21'
> >> > > > 53   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-21/00000000095028969469.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 53   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-48'
> >> > > > 54   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-48/00000000094491048876.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 54   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-40'
> >> > > > 55   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-40/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 55   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > > 'datapoints-increment-6'
> >> > > > 55   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-increment-6/00000000000000000000.kafka in
> >> > > > mutable mode, recovery false
> >> > > > 56   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-25'
> >> > > > 56   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-25/00000000093417394574.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 56   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-33'
> >> > > > 57   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-33/00000000093417473922.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 57   [main] INFO  kafka.log.LogManager  - Loading log
> >> > > 'datapoints-write-35'
> >> > > > 58   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-write-35/00000000092880891040.kafka in
> mutable
> >> > > > mode, recovery false
> >> > > > 58   [main] INFO  kafka.log.LogManager  - Loading log
> >> 'datapoints-wr'
> >> > > > 58   [main] INFO  kafka.log.Log  - Loading the last segment
> >> > > > /var/kafka/datapoints-wr/00000000000000000000.kafka in mutable
> mode,
> >> > > > recovery false
> >> > > > 60   [main] FATAL kafka.server.KafkaServerStartable  - Fatal error
> >> > > > during KafkaServerStable startup. Prepare to shutdown
> >> > > > java.lang.NumberFormatException: For input string: "wr"
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> >> > > >         at java.lang.Integer.parseInt(Integer.java:481)
> >> > > >         at java.lang.Integer.parseInt(Integer.java:514)
> >> > > >         at
> >> > > >
> >> scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
> >> > > >         at
> >> > scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
> >> > > >         at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
> >> > > >         at
> >> kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
> >> > > >         at
> >> kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> >> > > >         at
> >> scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
> >> > > >         at kafka.log.LogManager.<init>(LogManager.scala:65)
> >> > > >         at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
> >> > > >         at
> >> > > >
> >> >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> >> > > >         at kafka.Kafka$.main(Kafka.scala:50)
> >> > > >         at kafka.Kafka.main(Kafka.scala)
> >> > > > 61   [main] INFO  kafka.server.KafkaServer  - Shutting down Kafka
> >> > server
> >> > > > 62   [main] INFO  kafka.utils.KafkaScheduler  - shutdown scheduler
> >> > > > kafka-logcleaner-
> >> > > > 62   [main] INFO  kafka.server.KafkaServer  - Kafka server shut
> down
> >> > > > completed
> >> > > >
> >> > > > [blake@kafka03 kafka]$ ls
> >> > > > dat                      datapoints-increment-40
> >>  datapoints-write-26
> >> > > > datapoints-increment-0   datapoints-increment-41
> >>  datapoints-write-27
> >> > > > datapoints-increment-1   datapoints-increment-42
> >>  datapoints-write-28
> >> > > > datapoints-increment-10  datapoints-increment-43
> >>  datapoints-write-29
> >> > > > datapoints-increment-11  datapoints-increment-44
>  datapoints-write-3
> >> > > > datapoints-increment-12  datapoints-increment-45
> >>  datapoints-write-30
> >> > > > datapoints-increment-13  datapoints-increment-46
> >>  datapoints-write-31
> >> > > > datapoints-increment-14  datapoints-increment-47
> >>  datapoints-write-32
> >> > > > datapoints-increment-15  datapoints-increment-48
> >>  datapoints-write-33
> >> > > > datapoints-increment-16  datapoints-increment-49
> >>  datapoints-write-34
> >> > > > datapoints-increment-17  datapoints-increment-5
> >> datapoints-write-35
> >> > > > datapoints-increment-18  datapoints-increment-6
> >> datapoints-write-36
> >> > > > datapoints-increment-19  datapoints-increment-7
> >> datapoints-write-37
> >> > > > datapoints-increment-2   datapoints-increment-8
> >> datapoints-write-38
> >> > > > datapoints-increment-20  datapoints-increment-9
> >> datapoints-write-39
> >> > > > datapoints-increment-21  datapoints-wr
>  datapoints-write-4
> >> > > > datapoints-increment-22  datapoints-writ
> >>  datapoints-write-40
> >> > > > datapoints-increment-23  datapoints-write-0
> >> datapoints-write-41
> >> > > > datapoints-increment-24  datapoints-write-1
> >> datapoints-write-42
> >> > > > datapoints-increment-25  datapoints-write-10
> >>  datapoints-write-43
> >> > > > datapoints-increment-26  datapoints-write-11
> >>  datapoints-write-44
> >> > > > datapoints-increment-27  datapoints-write-12
> >>  datapoints-write-45
> >> > > > datapoints-increment-28  datapoints-write-13
> >>  datapoints-write-46
> >> > > > datapoints-increment-29  datapoints-write-14
> >>  datapoints-write-47
> >> > > > datapoints-increment-3   datapoints-write-15
> >>  datapoints-write-48
> >> > > > datapoints-increment-30  datapoints-write-16
> >>  datapoints-write-49
> >> > > > datapoints-increment-31  datapoints-write-17
>  datapoints-write-5
> >> > > > datapoints-increment-32  datapoints-write-18
>  datapoints-write-6
> >> > > > datapoints-increment-33  datapoints-write-19
>  datapoints-write-7
> >> > > > datapoints-increment-34  datapoints-write-2
> datapoints-write-8
> >> > > > datapoints-increment-35  datapoints-write-20
>  datapoints-write-9
> >> > > > datapoints-increment-36  datapoints-write-21      series-index-0
> >> > > > datapoints-increment-37  datapoints-write-22      series-index-1
> >> > > > datapoints-increment-38  datapoints-write-23      series-index-2
> >> > > > datapoints-increment-39  datapoints-write-24      series-index-3
> >> > > > datapoints-increment-4   datapoints-write-25      series-index-4
> >> > > > [blake@kafka03 kafka]$ cd datapoints-wr
> >> > > > [blake@kafka03 datapoints-wr]$ ls -lah
> >> > > > total 8.0K
> >> > > > drwxr-xr-x   2 kafka kafka 4.0K Jul 11 18:51 .
> >> > > > drwxr-xr-x 110 kafka kafka 4.0K Jul 11 19:05 ..
> >> > > > -rw-r--r--   1 kafka kafka    0 Jul 11 18:51
> >> 00000000000000000000.kafka
> >> > > >
> >> > > >
> >> > > >
> >> > > > [2] In our consumer logs:
> >> > > >
> >> > > > 013-07-11 21:33:01,136 61867 [FetchRunnable-0] ERROR
> >> > > > kafka.consumer.FetcherRunnable  - error in FetcherRunnable
> >> > > > kafka.message.InvalidMessageException: message is invalid,
> >> compression
> >> > > > codec: NoCompressionCodec size: 79 curr offset: 93706191248 init
> >> > > > offset: 93706191165
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:130)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:160)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100)
> >> > > >         at
> >> > > >
> >> >
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> >> > > >         at
> >> > > kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:64)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:59)
> >> > > >         at
> >> > > >
> >> kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:57)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:79)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:65)
> >> > > >         at
> >> > > >
> >> > >
> >> >
> >>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
> >> > > >         at scala.collection.immutable.List.foreach(List.scala:76)
> >> > > >         at
> >> kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:65)
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Reply via email to