Re: New client commitAsync SendFailedException

2016-03-19 Thread Jason Gustafson
Hey Alexey, looks like we didn't have a JIRA before, but I created one just now: https://issues.apache.org/jira/browse/KAFKA-3412. As soon as I get a little time, I'll polish up my old patch and submit it. -Jason On Wed, Mar 16, 2016 at 4:10 AM, Alexey Romanchuk < alexey.romanc...@gmail.com> wrot

Re: New client commitAsync SendFailedException

2016-03-16 Thread Alexey Romanchuk
Jason, Jay, could you please share bug number to track this issue? On Tue, Mar 15, 2016 at 10:57 PM, Jason Gustafson wrote: > Yeah, I agree the bug is probably more serious than I had thought before > (I've gotten too used to examples with only a single commitAsync() call). I > had worked on a p

Re: New client commitAsync SendFailedException

2016-03-15 Thread Jason Gustafson
Yeah, I agree the bug is probably more serious than I had thought before (I've gotten too used to examples with only a single commitAsync() call). I had worked on a patch to fix this at one point. Let me see if I can dig it up. -Jason On Tue, Mar 15, 2016 at 1:06 AM, Alexey Romanchuk < alexey.rom

Re: New client commitAsync SendFailedException

2016-03-15 Thread Alexey Romanchuk
Jay, I am completely agreed with you. From API level it is nothing I can do about it. Anything else I have to do with this issue? Reproducible example here - https://gist.github.com/13h3r/42633bcd64b80ddffe6b Thanks! On Tue, Mar 15, 2016 at 11:24 AM, Jay Kreps wrote: > This seems like a bug, no

Re: New client commitAsync SendFailedException

2016-03-14 Thread Jay Kreps
This seems like a bug, no? It should just initiate the request not wait for it to be written, there is no way for the user to reason about the state of the send buffer. -jay On Monday, March 14, 2016, Jason Gustafson wrote: > Hey Alexey, > > Asynchronous commit handling could probably be improv

Re: New client commitAsync SendFailedException

2016-03-14 Thread Alexey Romanchuk
Thanks for reply Jason! Is it any way to control size of this buffer? Will it fails if I try to commit offsets for 100 topics/partitions? Sure I can work around it by batching all commits into one call, but the problem I can see is that the API does not enforce me to do this. From client point of

Re: New client commitAsync SendFailedException

2016-03-14 Thread Jason Gustafson
Hey Alexey, Asynchronous commit handling could probably be improved quite a bit. Basically what's happening is that the client's send buffer is getting filled up, which causes the subsequent commits to fail with SendFailedException. We don't currently implement any retrying for asynchronous commit

New client commitAsync SendFailedException

2016-03-14 Thread Alexey Romanchuk
Hi all! I am using new client 0.9.0.1. I found that when I call commitAsync multiple times before calling poll most of commits failed with SendFailedException. Here it is an example of code - https://gist.github.com/13h3r/42633bcd64b80ddffe6b Could you please explain commitAsync in more details