What I am doing is, consuming two topics, samza_input and samza_command. samza_command will have some control command something like "shutdown,all" because kill-yarn-job.sh does not gracefully shutdown SamzaContainer. Am I correct? If so, what's the best way to shutdown the container without using command topic?
10ms explains why 50 null envelops were consumed per second. Does it hurt overall processing performance? I don't think so, but I want to confirm. Thank you Best, Jae On Fri, Feb 6, 2015 at 12:16 PM, Chris Riccomini <criccom...@apache.org> wrote: > Hey Jae, > > SamzaContainer polls for new messages by calling > consumerMultiplexer.choose. In a case where there are no messages > available, choose will return null. The next time choose is called, it will > be invoked with a timeout (the default is 10ms). This time, the poll call > will block until 1) the timeout is hit 2) there is a new message available > to process. This is to prevent a tight loop. > > > its frequency is too high, in my testing environment, it's more than 50 > per second. > > Why do you think this is too high? It either has to do this, or sleep for > longer. The longer the container sleeps, the more latency that's introduced > when there *is* a message available. 10ms is what we use by default. > > Cheers, > Chris > > On Fri, Feb 6, 2015 at 11:11 AM, Bae, Jae Hyeon <metac...@gmail.com> > wrote: > > > Could you explain why consumerMultiplexer.choose returns null? > > > > Can it happen when there's no message in the kafka topic? > > > > If my theory is correct, its frequency is too high, in my testing > > environment, it's more than 50 per second. > > > > Thank you > > Best, Jae > > >
