Hello,
If I have a consumer group with more members than the number of partition
of a topic, adding a consumer to the group will still trigger a rebalancing
of partitions to the group?
Imagine the partitions are already perfectly balanced, ie each consumer has
1 partition. Then reblancing won't b
While I haven't reached that point because with repartitioning and aggregation
and various other things happening with in the stream having additional topics
a simple 5 input topics with 5 partitions can end up with hundreds of total
topic partitions overall.
But I'm 99% confident that it will
Hi Sebastian,
Thanks for your question! A consumer joining a group should always notify
the group coordinator and the leader consumer, and the
partition.assignment.strategy [1] will determine the rebalance protocol,
and which partitions are affected by the rebalance.
For example, I expect that th
Hello,
we have a usecase where we use kafkaConsumer in a SourceTask of a source
connector to poll messages from an aws msk.
if we try to produce data into the source topic immediately after the
connector gets into the running state we sometimes notice that
kafkaconsumer misses some of the records
Hello,
we have a requirement to make kafka connector more fault tolerant for our
use, where in we don't want them to fail for some kinds of errors, i.e
error where bootstrap broker are missing or if we don't have sufficient
permission to read data from topic (we are reading from aws msk).
we tried
Hi Akash,
Thanks for your question. When you say:
> kafkaconsumer misses some of the records written into the kafka source
topic.
Do you mean that there are records in the source topic that are never
returned by Consumer#poll(), SourceTask#poll(), or written to the target
topic by KafkaConnect?
Hi Akash, if your connector doesn't have the appropriate permissions for a
topic then it can't run - so I'm not sure what value there is in trying to
handle or tolerate that kind of exception. If someone is changing ACLs in
such a way that it breaks a connector, then you probably want that
connect
