Actually I figured out what the problem was. My producer was using a
partitioner which was causing a null pointer exception. This actually
raises another question for me. I want some state in my partitioner and the
only constructor that Kafka seems to use is this one:
public MyPartitioner(VerifiableProperties props) {}
How do I inject an object here that I can use to decide what partition my
messages should go to? As an ugly hack I am using a static variable from
somewhere else.
On Thu, Oct 30, 2014 at 10:25 AM, Rajiv Kurian <ra...@signalfuse.com> wrote:
> Yes I see a ton of WARN messages on the broker logs of this form:
>
> 2014-10-30T17:21:54.281Z WARN [kafka-request-handler-6 ]
> [state.change.logger ]: Broker 0 received invalid
> LeaderAndIsr request with correlation id 158 from controller 0 epoch 29083
> with an older leader epoch 5 for partition [myTopic, 56], current leader
> epoch is 5
>
> 2014-10-30T17:21:54.282Z WARN [kafka-request-handler-6 ]
> [state.change.logger ]: Broker 0 received invalid
> LeaderAndIsr request with correlation id 158 from controller 0 epoch 29083
> with an older leader epoch 5 for partition myTopic,385], current leader
> epoch is 5
>
> 2014-10-30T17:21:54.283Z WARN [kafka-request-handler-6 ]
> [state.change.logger ]: Broker 0 received invalid
> LeaderAndIsr request with correlation id 158 from controller 0 epoch 29083
> with an older leader epoch 5 for partition [myTopic,684], current leader
> epoch is 5
>
> 2014-10-30T17:21:54.283Z WARN [kafka-request-handler-6 ]
> [state.change.logger ]: Broker 0 received invalid
> LeaderAndIsr request with correlation id 158 from controller 0 epoch 29083
> with an older leader epoch 5 for partition [myTopic,1002], current leader
> epoch is
>
> On Thu, Oct 30, 2014 at 10:03 AM, Joel Koshy <jjkosh...@gmail.com> wrote:
>
>> Do you see any errors on the broker logs? Can you check the broker's
>> public access logs and see if there are topic metadata requests coming
>> in from the producer?
>>
>> On Wed, Oct 29, 2014 at 07:15:15PM -0700, Rajiv Kurian wrote:
>> > I don't see anything else that is relevant. I traced the first of these
>> > error messages to figure out the ordering. It actually goes something
>> like
>> > this:
>> >
>> > 2014-10-30T01:51:32.400Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
>> > topic, partition due to: null
>> >
>> > 2014-10-30T01:51:34.082Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
>> > topic, partition due to: null
>> >
>> > 2014-10-30T01:51:34.422Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
>> > topic, partition due to: null
>> >
>> > 2014-10-30T01:51:34.664Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages by
>> > topic, partition due to: null
>> >
>> > 2014-10-30T01:51:34.902Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.DefaultEventHandler] {}: Failed to send requests for
>> > topics myTopic with correlation ids in [0,8]
>> >
>> > 2014-10-30T01:51:35.007Z ERROR [ProducerSendThread- ]
>> > [k.producer.async.ProducerSendThread ] {}: Error in handling batch of 1
>> > events
>> >
>> > kafka.common.FailedToSendMessageException: Failed to send messages
>> after 3
>> > tries.
>> >
>> > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
>> > ~[kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown Source)
>> > [kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > at
>> >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > at
>> >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > at scala.collection.immutable.Stream.foreach(Stream.scala:547)
>> > [scala-library-2.10.1.jar:na]
>> >
>> > at kafka.producer.async.ProducerSendThread.processEvents(Unknown Source)
>> > [kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > at kafka.producer.async.ProducerSendThread.run(Unknown Source)
>> > [kafka_2.10-0.8.0.jar:0.8.0]
>> >
>> > Thanks!
>> >
>> > On Wed, Oct 29, 2014 at 7:02 PM, Rajiv Kurian <ra...@signalfuse.com>
>> wrote:
>> >
>> > > This pattern seems to repeat:
>> > >
>> > > 2014-10-30T01:54:46.004Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.DefaultEventHandler] {}: Failed to send requests for
>> > > topics myTopic with correlation ids in [1729,1736]
>> > >
>> > > 2014-10-30T01:54:46.008Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.ProducerSendThread ] {}: Error in handling batch of
>> 4
>> > > events
>> > >
>> > > kafka.common.FailedToSendMessageException: Failed to send messages
>> after 3
>> > > tries.
>> > >
>> > > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
>> > > ~[kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown Source)
>> > > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > at
>> > >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > at
>> > >
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > at scala.collection.immutable.Stream.foreach(Stream.scala:547)
>> > > [scala-library-2.10.1.jar:na]
>> > >
>> > > at kafka.producer.async.ProducerSendThread.processEvents(Unknown
>> Source)
>> > > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > at kafka.producer.async.ProducerSendThread.run(Unknown Source)
>> > > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >
>> > > 2014-10-30T01:54:46.025Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages
>> by
>> > > topic, partition due to: null
>> > >
>> > > 2014-10-30T01:54:46.174Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages
>> by
>> > > topic, partition due to: null
>> > >
>> > > 2014-10-30T01:54:46.356Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages
>> by
>> > > topic, partition due to: null
>> > >
>> > > 2014-10-30T01:54:46.644Z ERROR [ProducerSendThread- ]
>> > > [k.producer.async.DefaultEventHandler] {}: Failed to collate messages
>> by
>> > > topic, partition due to: null
>> > >
>> > > On Wed, Oct 29, 2014 at 6:57 PM, Jun Rao <jun...@gmail.com> wrote:
>> > >
>> > >> The log before that will show you the cause of the error. Could you
>> dig
>> > >> that out?
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Jun
>> > >>
>> > >> On Wed, Oct 29, 2014 at 6:43 PM, Rajiv Kurian <ra...@signalfuse.com>
>> > >> wrote:
>> > >>
>> > >> > I keep seeing these errors in my code that is just trying to send
>> some
>> > >> data
>> > >> > using an AsyncProducer:
>> > >> >
>> > >> > kafka.common.FailedToSendMessageException: Failed to send messages
>> > >> after 3
>> > >> > tries.
>> > >> >
>> > >> > at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
>> > >> > ~[kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > at kafka.producer.async.ProducerSendThread.tryToHandle(Unknown
>> Source)
>> > >> > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > at
>> > >> >
>> > >> >
>> > >>
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > >> > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > at
>> > >> >
>> > >> >
>> > >>
>> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(Unknown
>> > >> > Source) [kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > at scala.collection.immutable.Stream.foreach(Stream.scala:547)
>> > >> > [scala-library-2.10.1.jar:na]
>> > >> >
>> > >> > at kafka.producer.async.ProducerSendThread.processEvents(Unknown
>> Source)
>> > >> > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > at kafka.producer.async.ProducerSendThread.run(Unknown Source)
>> > >> > [kafka_2.10-0.8.0.jar:0.8.0]
>> > >> >
>> > >> > 2014-10-30T01:40:45.176Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.DefaultEventHandler] {}: Failed to collate
>> messages by
>> > >> > topic, partition due to: null
>> > >> >
>> > >> > 2014-10-30T01:40:45.506Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.DefaultEventHandler] {}: Failed to collate
>> messages by
>> > >> > topic, partition due to: null
>> > >> >
>> > >> > 2014-10-30T01:40:45.647Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.DefaultEventHandler] {}: Failed to collate
>> messages by
>> > >> > topic, partition due to: null
>> > >> >
>> > >> > 2014-10-30T01:40:45.772Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.DefaultEventHandler] {}: Failed to collate
>> messages by
>> > >> > topic, partition due to: null
>> > >> >
>> > >> > 2014-10-30T01:40:45.890Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.DefaultEventHandler] {}: Failed to send requests
>> for
>> > >> > topics myTopic with correlation ids in [169,176]
>> > >> >
>> > >> > 2014-10-30T01:40:45.892Z ERROR [ProducerSendThread-
>> ]
>> > >> > [k.producer.async.ProducerSendThread ] {}: Error in handling batch
>> of 29
>> > >> > events
>> > >> >
>> > >> >
>> > >> > I created the topic before starting using bin/kafka-topics.sh. I
>> checked
>> > >> > zookeeper and seems like the topic was indeed created. Any ideas?
>> > >> >
>> > >>
>> > >
>> > >
>>
>>
>
