Hi Jens,
Gut feeling is that it's not a trivial patch, but you're more than welcome
to take a shot. We should probably do a KIP also since it's a change to a
public API (even if we only just released it). That's also a good way to
get feedback and make sure we're not missing a better approach. Her
Hi again Jason,
Once again thanks for your answer.
Yes, the more I think/read about this it sounds like the "max records"
approach is more viable. Without knowing the code, I guess it would make
more sense to create a "max.partition.fetch.messages" property. That way a
consumer could optimize for
Hi again Jason,
Sorry for a bit of a late response - I'm travelling and check my e-mail
spuriously.
I have a specific question regarding they pause solution quoted below:
On Tuesday, December 15, 2015, Jason Gustafson wrote:
>
> while (running) {
> ConsumerRecords records = consumer.poll(1000
Jason, maybe useful to KAFKA-2986 with this information if we ever decide
to do this?
Ismael
On 16 Dec 2015 04:42, "Jason Gustafson" wrote:
> I was talking with Jay this afternoon about this use case. The tricky thing
> about adding a ping() or heartbeat() API is that you have to deal with the
>
I was talking with Jay this afternoon about this use case. The tricky thing
about adding a ping() or heartbeat() API is that you have to deal with the
potential for rebalancing. This means either allowing it to block while a
rebalance completes or having it raise an exception indicating that a
reba
Hey Jens,
The purpose of pause() is to stop fetches for a set of partitions. This
lets you continue calling poll() to send heartbeats. Also note that poll()
generally only blocks for rebalances. In code, something like this is what
I was thinking:
while (running) {
ConsumerRecords records = con
Hi Jason,
Thanks for your response. See replies inline:
On Tuesday, December 15, 2015, Jason Gustafson wrote:
> Hey Jens,
>
> I'm not sure I understand why increasing the session timeout is not an
> option. Is the issue that there's too much uncertainly about processing
> time to set an upper b
Hey Jens,
I'm not sure I understand why increasing the session timeout is not an
option. Is the issue that there's too much uncertainly about processing
time to set an upper bound for each round of the poll loop?
One general workaround would be to move the processing into another thread.
For exam
Hi again,
For the record I filed an issue about this here:
https://issues.apache.org/jira/browse/KAFKA-2986
Cheers,
Jens
–
Skickat från Mailbox
On Fri, Dec 11, 2015 at 7:56 PM, Jens Rantil wrote:
> Hi,
> We've been experimenting a little with running Kafka internally for better
> ha
Hi,
We've been experimenting a little with running Kafka internally for better
handling temporary throughput peaks of asynchronous tasks. However, we've
had a big issues making Kafka work for us and I am starting to question
whether its a good fit.
Our usecase:
- High latency. At peaks, each
10 matches
Mail list logo
