Re: Single Key Aggregation

You can look into KStreamReduceProcessor#process() and KStreamAggregateProcessor#process() This should help to see the input data for each instance. count()/aggregate() will use AggregateProcessor while reduce() will use ReduceProcessor Hope this helps. -Matthias On 6/20/17 10:54 PM, Sameer K

Re: Aggregation operations and Joins not working as I would expect.

Daniel, maybe I did not extract your test logic correctly form you code. It's a long piece... However, I do see two order-events: >>> playOrderstream(clock, user, 2, "Japan"); >>> playOrderstream(clock, user, 6, "Peru"); Maybe your test logic is somehow not correct (just a wild guess).

Fw:help!Kafka failover do not work as expected in Kafka quick start tutorial

I post this question on the Stackoverflow.com in details. https://stackoverflow.com/questions/44718439/kafka-high-available-feature-not-work?answertab=oldest#tab-top Up to now, no answer fully explains it. Forwarding messages From: "夏昀" Date: 2017-06-22 20:15:59 To: user

Re: Kafka 10.2.1 KStreams rebalancing issue

Hi Eno, Yes, I was concerned that machine2 ispite of being free is not utilized at all. Partitions would be 60 and streams would have 63 threads. So, I understand now why threads were not started on machine2. For others, 1 partition would be consumed by 1 thread. So, thus the mismatch. Thanks,

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

ok, thanks ... i'll check that and update in case i see issues. regds, Karan Alang On Fri, Jun 23, 2017 at 12:01 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > I think what you are seeing with `--time -1` and '--time -2` confirms that > the messages are deleted from th

Re: [VOTE] 0.11.0.0 RC2

Hi Ismael, Not sure if my response on RC1 was lost or this issue is not a show-stopper: I checked again and with RC2, tests still fail in my Windown 64 bit environment. :clients:checkstyleMain [ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache

max.poll.records and pause

How the pause influences max.poll.records? If I have two partitions, say p1 and p2. p1 has number local fetched records > max.poll.records. p2 has no data in local but in the server (not fetched yet). Now, if I pause P1. Does the next poll send fetch request to the server so that I can get data

Re: [VOTE] 0.11.0.0 RC2

Hi, As previously, Heroku has been doing performance testing of 0.11.0.0 RC2. Now that the ProducerPerformance.java tool supports it, we've even been doing testing with the new transactional/idempotence features in KIP-98. We've tested with idempotency and read_committed consumers and note no not

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

Hi Karan, I think what you are seeing with `--time -1` and '--time -2` confirms that the messages are deleted from the log. The offset returned in both cases is the same, which means that the offset start and offset end are both the same (i.e. the log is empty). When messages are removed from th

Re: Kafka 10.2.1 KStreams rebalancing issue

Hi Sameer, Could you elaborate on your question? Are you concerned that machine2 does not have any tasks in the beginning? Could you share your streams configuration? In particular how many threads does each stream instance have? Also how many topics and partitions do you have? Thanks, Eno

Kafka 10.2.1 KStreams rebalancing issue

Hi, Came across a rebalancing issue in using KafkaStreams. I have two machines, Machine1 and Machine2, machine1 is consuming all partitions and machine2 is completely free and not processing any partitions. If I shutdown machine1, then machine2 will take over and would start consuming all partitio

Re: How does Zookeeper node failure impact Kafka cluster?

Losing one out of three should not impact the cluster. Losing more than a majority means certain Kafka operations won't work. Anything that requires the zookeeper data like electing a new leader for example. Dave Sent using OWA for iPhone From: mayank r

Re: Aggregation operations and Joins not working as I would expect.

Hi Matthias, Thanks for your reply. I have a few questions on your comments: 1) I understand the records are updated as Ktables work as changelogs. However there is a single record in my source stream which means that even if the aggregate is updated 2+ times the output should still be the same.

Re: ZookeeperLeaderElector broker shutdown rebalance time

It seems this doesn’t just happen when the broker being restarted is the one that has the leadership ('1002 successfully elected as leader’). I have now turned off the auto leadership rebalance. What I don’t understand is that I can run the manual leadership rebalance when there are messages b

Exposing Kafka Topic to External Parties

Dear Kafka Users, Would you consider it a good practice to expose the Kafka topic directly to a 3rd party application? While doing this, I need to satisfy the following: 1. I will have say 10 topics and I would need to make sure that only authorized parties are able to write into the Topic 2. If

Re: Kafka 0.11.0 release

Thanks for the update, Guozhang. On Thu, Jun 22, 2017 at 9:52 PM, Guozhang Wang wrote: > Raghav, > > We are going through the voting process now, expecting to have another RC > and release in a few more days. > > > Guozhang > > On Thu, Jun 22, 2017 at 3:59 AM, Raghav wrote: > > > Hi > > > > Wou