Hi Sarah, Thanks for the KIP! I have two major thoughts:
1. Adding new methods like this to the Connect API comes with the risk that connectors that invoke them become incompatible with older versions of Connect. For example, if I updated my connector to use the newly-proposed SourceTaskContext::connector method, it would fail with a NoSuchMethodError when run on Connect version 3.1 (which will not have this feature). We've been careful to document this limitation in any newly-introduced methods in the past (see KIP-610 [1] and KIP-618 [2], for example), but even then, there's still risk with anything like this and we may not want to expand the API if the benefits don't outweigh the costs. 2. It's already possible to implement this logic directly in a connector today, without any changes to the Connect framework. Every connector can learn its own name in Connector::start by reading the "name" configuration property, and can then choose to pass that information along to its tasks as part of the configs it generates in Connector::taskConfigs. And, in the same way, connectors can choose to provide IDs for each task in the task configs that they generate. If this isn't sufficient for your use cases, it should be documented as a rejected alternative. BTW, it looks like this aims to accomplish something very similar or even identical to KIP-438 [3]. Ryanne Dolan (the author of that KIP) may want to weigh in here; I've CC'd them. [1] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors#KIP610:ErrorReportinginSinkConnectors-Method (see javadocs for SinkTaskContext::errantRecordReporter, paragraph starting with "This method was added in Apache Kafka 2.6") [2] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-ConnectorAPIexpansions (see javadocs for SourceTaskContext::transactionContext, paragraph starting with "This method was added in Apache Kafka 3.0") [3] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-438%3A+Expose+task%2C+connector+IDs+in+Connect+API Cheers, Chris On Tue, Dec 7, 2021 at 10:07 AM Sarah Story <sa...@sarahstoryengineering.com> wrote: > Hi all! Just wanted to bump this KIP for adding Task ID and Connector Name > to the task context. It's a small change, and I'd love some feedback on it! > > Thanks! > Sarah > > On Wed, Nov 24, 2021 at 10:26 AM Sarah Story < > sa...@sarahstoryengineering.com> wrote: > > > I have written a KIP for adding Task ID and Connector name to the Connect > > API task context. > > > > Here is the doc: https://cwiki.apache.org/confluence/x/4pKqCw > > > > Looking forward to hearing all of your thoughts! > > >
