Re: Random connection errors

2015-12-15 Thread Kishore N C
The first log was from a time when there was quite a bit of load on the Kafka servers. But in the second link I shared you, will notice that the task did not even start - this was when the traffic was very light on the Kafka cluster. I enable debug mode to see if that uncovers more. On Dec 16, 201

Re: Random connection errors

2015-12-15 Thread Yi Pan
Hi, Kishore, I saw a lot of NOT_LEADER_FOR_PARTITION errors in the first log. It seems like your job is running for a while and started to lose the leader broker for the topic in Kafka and the problem was not resolved by re-try. Are you pumping through a lot of traffic to the broker? Can you try t

Re: Random connection errors

2015-12-15 Thread Kishore N C
I did manage to locate a task which was blocked without starting as I had described earlier. The log for that task is here: https://gist.githubusercontent.com/kishorenc/ef806e85478378ce2203/raw/8537d3d3644eea7cdd9efc9fa8749c0840092f3c/gistfile1.txt Thanks, Kishore. On Tue, Dec 15, 2015 at 4:30

Re: Random connection errors

2015-12-15 Thread Kishore N C
Hi Yi Pan, I'm using Samza 0.9.1 and Kafka 0.8.2.1. Here's an example of a full task log: https://gist.githubusercontent.com/kishorenc/5d65f114a50b9ef6a6b3/raw/5b9ecffdd1af831f713e8b41e5b77e5b881e8173/gistfile1.txt You will find "java.nio.channels.ClosedChannelException" towards the end. One ad

Re: Random connection errors

2015-12-14 Thread Yi Pan
Hi, Kishore, First, I would like to ask which version of Samza you are running? And if you can attach the log and config of your container (i.e. I assume the log you attached here is a container log?), it would be greatly helpful. Thanks a lot! -Yi On Mon, Dec 14, 2015 at 5:07 AM, Kishore N C