Re: How distributed countByKey works in KStream ?

It works after calling through() before countByKey, so many 0.10.0.1 examples on the web missing the `through()` call and it will fail to get the right output when running with input topic > 1 partitions. Thanks very much all ! Finally got the correct results. On Thu, Sep 1, 2016 at 4:52 AM, Matt

Re: How distributed countByKey works in KStream ?

FYI: We updated the 0.10.0.x demos for Kafka Streams at https://github.com/confluentinc/examples to use #partitions >1 and include `through()`. See for example [1]. Hope this helps! Michael [1] https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/main/java/io

Monitoring Kafka client

Hi, My application recently experience a network connectivity issue which lead into getting the client (0.8.2.2) disconnected. After the network was restored the client failed to reconnect because while trying to do this, resolving the Zookeeper Server hostname to an IP failed as well (DNS failure

Producer: metadata refresh when leader down

Hi, Until recently, I thought that the process for producers and metadata went like this: 1. Producer published to a leader broker that was down, which would fail. 2. Producer tried a few times (or zero, depending on config). 3. Eventually Producer would “fail” publishing that message. 4. That wou

Re: Retention on compacted topics

It is not applied. An upcoming release will have the ability to combine the two, but right now they are mutually exclusive. On Thu, Sep 1, 2016 at 6:18 PM, David Yu wrote: > Hi, > > Does Kafka "log.retention.bytes" or "log.retention.ms" apply to compaction > enabled topic? I'll be surprised it d

Re: Retention on compacted topics

Ok, thanks for confirming. On Thu, Sep 1, 2016 at 12:21 PM Tom Crayford wrote: > It is not applied. An upcoming release will have the ability to combine the > two, but right now they are mutually exclusive. > > On Thu, Sep 1, 2016 at 6:18 PM, David Yu wrote: > > > Hi, > > > > Does Kafka "log.re

Retention on compacted topics

Hi, Does Kafka "log.retention.bytes" or "log.retention.ms" apply to compaction enabled topic? I'll be surprised it does not, since this means that a compacted topic will potentially grow unbounded if deletion is not happening quickly enough (if we do deletion at all). Thanks, David

Error setting consumer offset using SimpleConsumer#commitOffsets.

Hello, I am having problems setting the partition offset for a given consumer group using SimpleConsumer#commitOffsets. I encounter error code "25". Am I going about this the right way? I am using Kafka 0.10.0. The consumers use the new API. Here is a runnable example

Kafka 9 consumer generates EOF exceptions when parsing responses

I've seen these before but we have recently moved to the 0.9 consumer code for one of our big Kafka use cases and now we see about 1 million EOF exceptions in a 5 minute period. This can't be very good for performance. My guess is that these exceptions are expected since it uses DataInputStream to

kafka streams: join 2 Windowed tables

Hi, Is joining 2 streams based on Windowed keys supposed to work? I have 2 KTables: - KTable, T> events: I process events and aggregate events that have a common criteria using aggregateByKey and UnlimitedWindows as window (for now) - KTable, S> hourlyStats: I calculate some stats using aggreg

Producer request latency increase after client 0.10 upgrade

We tried to upgrade the Kafka clients dependency from 0.8.2.1 to 0.10.0.0. As I understand, the producer client doesn't have major changes in 0.10, so we kept the same producer config in the upgrade: retries 3 retry.backoff.ms 5000 timeout.ms 1 block.on.buffer.full false linger.ms 5000 metadat

Re: Producer request latency increase after client 0.10 upgrade

With a linger of 5 seconds, 2-3 seconds would make sense when the load is smaller, are are sure the measurements with 0.8.2.1 where with the same load and/or linger worked correctly there? On Fri, Sep 2, 2016 at 1:12 AM Yifan Ying wrote: > We tried to upgrade the Kafka clients dependency from 0.