Assuming, that there are no duplicates in your input topic, as long as
no failure occurs, the consumer will read every message exactly-once by
default.
Only in case of failure, when the consumer "falls back" to an older
offset, you might see some duplicates. You will need to write custom
code to h
Hello,
We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10
partitions each. We have an application which spawns threads as consumers.
We spawn 5 consumers for each topic. I am observing that consider group
randomly keeps rebalancing. Then many times we see logs saying "Revoki
We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10
partitions each. We have an application which spawns threads as consumers.
We spawn 5 consumers for each topic. I am observing that consider group
randomly keeps rebalancing. Then many times we see logs saying "Revoking
part
Hello all,
I have a set of brokers and I want to know 'how much time do my brokers
spend in replication of a message?'.
I have two questions:
1. I did not find any available metric for replication latency (avg, max).
2. Why is there no metric for understanding broker's replication behavior?
Uddha
Hi,
I'm collecting producer metrics via JMX, but I'm not sure how to interpret
some values.
For example, connection-creation-rate. When I query this value, what is the
time window it relates too? Is the value reset every time it's queried?
Appreciate the help,
thanks
We have a Kafka Streams app that fails to start correctly because somehow
the consumer doesn't get assigned any partition and the Queryable stores
are not available when the app starts.
Is there a way to force release the assignment that consumers have on
specific partitions?
Edmondo
We need to perform a Kstream - Kstream join with a very large window, where
a tick on the left would trigger a join only with the most recent record on
the right, and viceversa.
This is not how the default window works, since the WindowStoreIterator
returned by window.fetch inside the KStreamKStre
Hi,
Thanks, I will try it.
I just don’t understand a couple of things:
1. how could reducing the buffer size can help here.
2. What is using a dummy topic relevant? Can’t I just continue to send to the
existing topic?
Thanks
On 23/05/2018, 9:33 AM, "M. Manna" wrote:
Can you try following?
Question cross-posted at SO:
https://stackoverflow.com/questions/50492491/customize-window-store-implementation-in-kstream-kstream-join
I did put an answer there.
-Matthias
On 5/23/18 8:56 AM, Edmondo Porcu wrote:
> We need to perform a Kstream - Kstream join with a very large window, where
> a
Hi Edmondo,
While technically you can provide a custom KafkaClientProvider and
overwrite the configs for the consumer, I don't recommend you do so.
Partition and task assignment is much more involved in Kafka Streams than
doing so for plain consumers.
Why do you want to force partition assignment
Hi,
Trying to create kafka-spark-stream-cassandra process. Found one solution in
jupiter notebook. Trying to simulate the same using python. However Kafka
producer terminates with below error messages.
Traceback (most recent call last): File "kafkaSendData.py", line 13, in
producer.send('test
Dropping in late on this thread, but we just released our Kafka metrics
reporter for AppOptics:
https://github.com/appoptics/kafka-appoptics-reporter
It ties into the Yammer Metrics reporting interface to capture metrics and
publish to Appoptics. Just build and drop the Jar into your Kafka libs
di
Hi,
I am writing a service consuming messages with Kafka, and I sometimes notice
the following INFO log:
[Consumer clientId=consumer-1, groupId=my_group] (Re-)joining group
After a deployment of my service, this message was logged every 5 seconds,
until I re-deployed the service which seemed to
13 matches
Mail list logo
