How frequent does the follower replica issue a fetch request to leader replica?

2018-04-10 Thread Yu Watanabe
Hello. I would like to ask question regarding to fetch request from follower to leader replica. According to on line document it describes the flow of how fetch requests are fetched for sync operation . https://kafka.apache.org/documentation/#replication "Followers consume messages from the lea

Kafka Cluster - Partition offset reset to 0. How is this possible?

2018-04-10 Thread Yi Yin
Under what circumstance will a Kafka Partition (not consumer) lose its offset, and reset to 0? We are running Kafka 0.10.2.1 We have a Kafka topic with 8 evenly distributed partitions. Each partition's offset is relatively in line with each other. Partition Latest Offset Leader Replicas In Sync R

答复: kafka streams consumer ERROR The coordinator is not aware of this member.

2018-04-10 Thread 杰 杨
Kafka 1.0.0 Kafka-stream 1.0.0 发送自 Windows 10 版邮件应用 发件人: Guozhang Wang 发送时间: Wednesday, April 11, 2018 1:39:17 AM 收件人: users@kafka.apache.org 主题: Re: kafka streams consumer ERROR The coordinator is not aware

Re: Apache Kafka / Spark Integration - Exception - The server disconnected before a response was received.

2018-04-10 Thread M Singh
Hi Daniel: Yes I am working with Spark Structured Streaming. The exception is emanating from spark kafka connector but I was wondering if someone has encountered this issue and resolved it by some configuration parameter in kafka client/broker or OS settings. Thanks Mans On Tuesday, April

Re: kafka streams consumer ERROR The coordinator is not aware of this member.

2018-04-10 Thread Guozhang Wang
1) It means the consumer has not call poll() for quite some time and hence be kicked out by the consumer coordinator (the new consumer use heartbeat as the failure detection protocol, you can read more on the wiki docs about its design). If you have high variance of record processing latency, e.g.

Re: Kafka-streams: mix Processor API with windowed grouping

2018-04-10 Thread Dmitriy Vsekhvalnov
Thanks Michael, unfortunately we currently only considering exact counting. But will take a look to your example for sure. On Tue, Apr 10, 2018 at 12:16 PM, Michael Noll wrote: > Also, if you want (or can tolerate) probabilistic counting, with the option > to also do TopN in that manner, we als

Re: Apache Kafka / Spark Integration - Exception - The server disconnected before a response was received.

2018-04-10 Thread Daniel Hinojosa
This looks more like a spark issue than it does a Kafka judging by the stack trace, are you using Spark structured streaming with Kafka integration by chance? On Mon, Apr 9, 2018 at 8:47 AM, M Singh wrote: > Hi Folks: > Just wanted to see if anyone has any suggestions on this issue. > Thanks > >

kafka streams consumer ERROR The coordinator is not aware of this member.

2018-04-10 Thread 杰 杨
Hi: when I run kafka streams applications for a period of time and will be throw ERROR .1) questions org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.error.301|[Consumer clientId=filter-StreamThread-1-consumer, groupId=filter] Offset commit failed on partition sample-1 at offset

Re: Kafka-streams: mix Processor API with windowed grouping

2018-04-10 Thread Michael Noll
Also, if you want (or can tolerate) probabilistic counting, with the option to also do TopN in that manner, we also have an example that uses Count Min Sketch: https://github.com/confluentinc/kafka-streams-examples/blob/4.0.0-post/src/test/scala/io/confluent/examples/streams/ProbabilisticCountingSc