For sink connectors, I believe you can scale up the tasks to match the
partitions on the topic. But I don't believe this is the case for source
connectors; the number of partitions on the topic you're producing to has
nothing to do with the number of connector tasks. It really depends on the
indi
Hello
Confirmed. Partition is the minimal granularity level, so having more
consumers than the number of partitions of a topic for a same consumer
group is useless, having P partitions means maximum parallelism is reached
using P consumers.
Regards,
Sébastien.
Le jeu. 30 mai 2024 à 14:43, Yeike
The docs say: “Each task is assigned to a thread. Each task is capable of
handling multiple Kafka partitions, but a single partition must be handled by
only one task.”From what I understand additional tasks would sit idle.
From: Yeikel Santana
Date: Thursday, May 30, 2024 at 7:43 AM
To:
Hi everyone,
>From my understanding, if a topic has n partitions, we can create up to n
>tasks for both the source and sink connectors to achieve the maximum
>parallelism. Adding more tasks would not be beneficial, as they would remain
>idle and be limited to the number of partitions of the t
