Martin,
If you called, for example, commitAsync() before you call client.poll()
then the commit request will be sent during that period.
Guozhang
On Wed, Feb 22, 2017 at 2:04 AM, Martin Sucha wrote:
> Hi,
>
> thank you both for responses.
>
> I forgot to mention which python client I'm using
Hi,
thank you both for responses.
I forgot to mention which python client I'm using, so let me fix that first:
Currently I'm using https://github.com/dpkp/kafka-python
As Jeff mentioned, if there is a message that takes too long to process,
the group will rebalance and it will affect the whole c
Yes, in the linked issues in my email it shows both projects plan to add
support for it.
The problem with a high session.timeout.ms is if you've got a message that
is locking your consumer in a perpetual processing cycle, then the consumer
will timeout of the group w/o rejoining. So then another c
Hello Jeff,
You are right that currently kafka-python does not expose two configs, i.e.
session.timeout.ms and max.poll.timeout.ms as the Java client does, but I
think the current setting may be sufficient to tackle Martin's issue alone
as long as session.timeout.ms can be tuned to be large enough
As far as I understood it, the primary thrust of KIP-62 was making it so
heartbeats could be issued outside of the poll() loop, meaning that the
session.timeout.ms could be reduced below the length of time it takes a
consumer to process a particular batch of messages.
Unfortunately, while both lib
Hi Martin,
Since 0.10.1 KIP-62 has been added to consumer client, so that the user
does not need to manually call pause / resume.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
As for python client, as far as I know this ba
Hello,
I'm processing messages using kafka-python from a single topic, but
ocassionally I have a message that might take a while to process, so I'd
like to send heartbeats while processing the message.
My program works something like this:
consumer = KafkaConsumer(...)
for message in consumer:
