The issue was that there was a message that exceeded the default max.message.bytes for a kafka consumer. But rather than give any sort of error message, the thread responsible for consuming that partition would simply hang on that message. The other partition threads would continue along happily.
I have to say that the lack of any sort of timeout or error message was really a time sink for us. It would have been very helpful to have samza report that it could not read from that partition because of the size of the message. Once we configured systems.kafka.consumer.max.message.bytes to a large enough value to consume each of the messages on the partition and restarted the job, it picked up where it had left off and everything started working as expected. On Mon, Apr 25, 2016 at 6:41 PM Jason Erickson <ja...@stormpath.com> wrote: > My earlier answer to this was replying to the wrong message. I am > launching this with ThreadJobFactory. > > > On Mon, Apr 25, 2016 at 4:26 PM Jagadish Venkatraman < > jagadish1...@gmail.com> wrote: > >> Are you running in Yarn or as a local process? >> >> If you're running in Yarn, you can use the AppMaster UI to see what the >> jobmodel is. >> >> On Mon, Apr 25, 2016 at 2:33 PM, Jason Erickson <ja...@stormpath.com> >> wrote: >> >> > I have a puzzling issue with one of my samza tasks. It works correctly >> > except for messages on one partition. I have 9 partitions on the topic. >> If >> > I send 1000 messages, I only receive about 890 of them. I have checked >> with >> > kafka-console-consumer with partition keys that I know don’t get >> processed >> > by my samza job and the console consumer DOES see the message, so I know >> > it’s getting written to the topic and that at least a vanilla consumer >> can >> > see it just fine. >> > >> > >> > I’m happy to share whatever interesting configuration information would >> > help narrow this down, but right now, I’m a bit mystified about what I >> > would even share. >> > >> > >> > I’m running >> > >> > * samza-kafka_2.10 version 0.9.1 >> > >> > * kafka_2.10 version 0.8.2.1 on the client >> > >> > * kafka broker 0.9.0.0 >> > >> > >> >> >> >> -- >> Jagadish V, >> Graduate Student, >> Department of Computer Science, >> Stanford University >> >
