Re: kafka-streams TopologyTestDriver problem with EXACTLY_ONCE

2018-04-05 Thread Frederic Arno
That's what I'm doing now, I override the config from within my tests. Reported here: https://issues.apache.org/jira/browse/KAFKA-6749 Thanks, Fred On 04/04/2018 10:56 PM, Matthias J. Sax wrote: Just a side remark. As a workaround it should be fine to remove the config. TopologyTestDriver will

poll returns records while paused (Kafka 0.10.2)

2018-04-05 Thread Scott Thibault
My read of the documentation is that no records should be returned when the partition is paused. I have this consumer loop which is meant to keep the heartbeat going while the processing is busy: while (!closed.get) { val records = client.poll(timeout) if (records.count() > 0 && !work.offer(r

Re: kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION

2018-04-05 Thread Matthias J. Sax
Thanks Ted! Done. On 4/4/18 7:30 PM, Ted Yu wrote: > I created KAFKA-6747. > Frederic's Id was on KAFKA-6323. > > Can some committer change the reporter of KAFKA-6747 to Frederic ? > > Thanks > > On Wed, Apr 4, 2018 at 4:35 PM, Ted Yu wrote: > >> https://github.com/apache/kafka/pull/4826 >> >

Re: kafka-streams TopologyTestDriver problem with EXACTLY_ONCE

2018-04-05 Thread Matthias J. Sax
Thanks Fred! On 4/5/18 3:41 AM, Frederic Arno wrote: > That's what I'm doing now, I override the config from within my tests. > > Reported here: https://issues.apache.org/jira/browse/KAFKA-6749 > > Thanks, Fred > > On 04/04/2018 10:56 PM, Matthias J. Sax wrote: >> Just a side remark. As a worka

Re: delivery semantics?

2018-04-05 Thread Jacob Sheck
Your question seems to be confusing the producer and consumer responsibilities in message delivery durability. Neha from Confluent wrote a blog post about this which should help your understanding of the how the Kafka clients handle this. https://www.confluent.io/blog/exactly-once-semantics-are-p

What causes partitions to be revoked?

2018-04-05 Thread Scott Thibault
I'm using the Kafka 1.0.1 Java client with 1 consumer and 1 partition and using the ConsumerRebalanceListener I can see that the partition keeps getting revoked and then reassigned. My consumer is in it's own thread to ensure poll is invoked regularly. Is there some other reason this might be hap

Re: What causes partitions to be revoked?

2018-04-05 Thread Gabriel Giussi
There is some other consumer (in the same process or another) using the same group.id? 2018-04-05 14:36 GMT-03:00 Scott Thibault : > I'm using the Kafka 1.0.1 Java client with 1 consumer and 1 partition and > using the ConsumerRebalanceListener I can see that the partition keeps > getting revoked

Move to Kafka 1.1 from 0.10.2.x ?

2018-04-05 Thread Raghav
Hi Are there anything that needs to be taken care for if we want to move from 0.10.2.x to latest 1.1 release ? Is this stable release and is it recommended for production use ? Thanks Raghav

Re: Move to Kafka 1.1 from 0.10.2.x ?

2018-04-05 Thread Matthias J. Sax
Check out the upgrade notes: https://kafka.apache.org/documentation/#upgrade You should consider all notes for 0.11, 1.0 and 1.1 releases. And yes, 1.1 is absolutely ready for production. -Matthias On 4/5/18 11:57 AM, Raghav wrote: > Hi > > Are there anything that needs to be taken care for

Re: What causes partitions to be revoked?

2018-04-05 Thread Scott Thibault
No, there is only one consumer in the group. On Thu, Apr 5, 2018 at 2:39 PM, Gabriel Giussi wrote: > There is some other consumer (in the same process or another) using the > same group.id? > > 2018-04-05 14:36 GMT-03:00 Scott Thibault >: > > > I'm using the Kafka 1.0.1 Java client with 1 cons

Re: accessing Kafka broker 0.9.0.1 with Kafka-client 1.0.0

2018-04-05 Thread Ismael Juma
1.0.0 supports brokers with version 0.10.0 and higher. Ismael On Tue, Apr 3, 2018 at 6:36 AM, Nomar Morado wrote: > Hi, > > I am trying to access my Kafka brokers using springbok 2.0 which is using > Kafka-client 1.0.0 and it seems like running into timeout exceptions > waiting for metadata upd

Re: message queueing questions?

2018-04-05 Thread Matthias J. Sax
Hi, multiple answers to this question: 1) it depends of you send messages sync or async to the brokers. Producers do buffer messages in-memory for more efficient writes to the brokers. If messages are successfully sent to the brokers, you can get an acknowledgment back the you can check on the pr