> same time I would prefer additional churn to the API.
I meant to say prefer _less_ churn to the API.
On Tue, Jun 09, 2015 at 08:35:35AM -0700, Joel Koshy wrote:
> Ewen,
>
> Sorry for the late comment, but while we are discussing this API I
> think we should also consider the addition of metada
Ewen,
Sorry for the late comment, but while we are discussing this API I
think we should also consider the addition of metadata to the offset
commit. This is supported on the broker-side but there is no means in
the current API to include metadata or retention time in the offset
commit. Ideally,
Thanks Ewen,
1. I agree with you as for the pros of Future; my argument for pure
Callback, as I mentioned, is that it sounds to me unlikely users would
usually want to explicitly set timeout. That said, if people think this
scenario is actually common like in graceful shutdown, then I am OK with
F
Thanks, great feedback everyone.
Jiangjie -- I was worried about interleaving as well. For commits using the
consumer's own current set of offsets, I agree we could easily limit to 1
outstanding request so the older one gets cancelled. For commits that
specify offsets manually, we might not need t
Hi Ewen,
Only time I can think of where Application needs to know result of offset
was committed or not during graceful shutdown and/or
Runtime.addShutdownHook() so consumer application does not get duplicated
records upon restart or does not have to deal with eliminating already
process offset.
I second Guozhang's proposal. I do think we need the callback. The current
state is that for async commits you actually don't know if it succeeded.
However there is a totally valid case where you do need to know if it
succeeded but don't need to block, and without the callback you are stuck.
I thin
Hi Ewen,
I share the same concern you have about 2), that with the new API sync
commit implementation is a bit awkward since we have a single-threaded
design in new consumer. The reason that we need to mute other nodes for
doing coordinator sync operations like join-group / offset commits / offset
Hey Ewen,
This makes sense. People usually do not want to stop consuming when
committing offsets.
One corner case about async commit with retries I am thinking is that it
is possible that two offset commits interleave with each other and that
might create problem. Like you said maybe we can canc
I'd like to get some feedback on changing the offset commit API in the new
consumer. Since this is user-facing API I wanted to make sure this gets
better visibility than the JIRA (
https://issues.apache.org/jira/browse/KAFKA-2123) might.
The motivation is to make it possible to do async commits bu
