Hi, Jason, Sorry to hear that it caused you a long time to figure this out. Could you open a JIRA on this issue s.t. we can do some further investigation to see whether there is explicit indication from Kafka consumer on this problem and we can report correctly?
-Yi On Fri, Apr 29, 2016 at 2:45 PM, Jason Erickson <ja...@stormpath.com> wrote: > The issue was that there was a message that exceeded the default > max.message.bytes for a kafka consumer. But rather than give any sort of > error message, the thread responsible for consuming that partition would > simply hang on that message. The other partition threads would continue > along happily. > > I have to say that the lack of any sort of timeout or error message was > really a time sink for us. It would have been very helpful to have samza > report that it could not read from that partition because of the size of > the message. > > Once we configured systems.kafka.consumer.max.message.bytes to a large > enough value to consume each of the messages on the partition and restarted > the job, it picked up where it had left off and everything started working > as expected. > > > On Mon, Apr 25, 2016 at 6:41 PM Jason Erickson <ja...@stormpath.com> > wrote: > > > My earlier answer to this was replying to the wrong message. I am > > launching this with ThreadJobFactory. > > > > > > On Mon, Apr 25, 2016 at 4:26 PM Jagadish Venkatraman < > > jagadish1...@gmail.com> wrote: > > > >> Are you running in Yarn or as a local process? > >> > >> If you're running in Yarn, you can use the AppMaster UI to see what the > >> jobmodel is. > >> > >> On Mon, Apr 25, 2016 at 2:33 PM, Jason Erickson <ja...@stormpath.com> > >> wrote: > >> > >> > I have a puzzling issue with one of my samza tasks. It works correctly > >> > except for messages on one partition. I have 9 partitions on the > topic. > >> If > >> > I send 1000 messages, I only receive about 890 of them. I have checked > >> with > >> > kafka-console-consumer with partition keys that I know don’t get > >> processed > >> > by my samza job and the console consumer DOES see the message, so I > know > >> > it’s getting written to the topic and that at least a vanilla consumer > >> can > >> > see it just fine. > >> > > >> > > >> > I’m happy to share whatever interesting configuration information > would > >> > help narrow this down, but right now, I’m a bit mystified about what I > >> > would even share. > >> > > >> > > >> > I’m running > >> > > >> > * samza-kafka_2.10 version 0.9.1 > >> > > >> > * kafka_2.10 version 0.8.2.1 on the client > >> > > >> > * kafka broker 0.9.0.0 > >> > > >> > > >> > >> > >> > >> -- > >> Jagadish V, > >> Graduate Student, > >> Department of Computer Science, > >> Stanford University > >> > > >