For sink connectors, I believe you can scale up the tasks to match the partitions on the topic. But I don't believe this is the case for source connectors; the number of partitions on the topic you're producing to has nothing to do with the number of connector tasks. It really depends on the individual source connector and if the source data-type could benefit from multiple tasks. For example, the JDBC source connector (a very popular connector) only supports 1 task - even if you're querying multiple tables.
Bottom line: you'll need to check the documentation for the connector in question to see if it supports multiple tasks. Alex On Thu, May 30, 2024 at 7:51 AM Sébastien Rebecchi <srebec...@kameleoon.com> wrote: > Hello > > Confirmed. Partition is the minimal granularity level, so having more > consumers than the number of partitions of a topic for a same consumer > group is useless, having P partitions means maximum parallelism is reached > using P consumers. > > Regards, > > Sébastien. > > Le jeu. 30 mai 2024 à 14:43, Yeikel Santana <em...@yeikel.com> a écrit : > > > Hi everyone, > > > > > > From my understanding, if a topic has n partitions, we can create up to > n > > tasks for both the source and sink connectors to achieve the maximum > > parallelism. Adding more tasks would not be beneficial, as they would > > remain idle and be limited to the number of partitions of the topic > > > > > > Could you please confirm if this understanding is correct? > > > > > > If this understanding is incorrect could you please explain the > > relationship if any? > > > > > > Thank you! > > > > > > >
