Hey Snehashsih, Thanks for the KIP. It looks like a very useful feature. Couple of small-ish points, let me know what you think:
1) Should we update the GET /connectors endpoint to include the version of the plugin that is running? It could be useful to figure out the version of the plugin or I am assuming it gets returned by the expand=info call? 2) I am not aware of this and hence asking, can 2 connectors with different versions have the same name? Does the plugin isolation allow this? This could have a bearing when using the lifecycle endpoints for connectors like DELETE etc. Thanks! Sagar. On Tue, Nov 22, 2022 at 2:10 PM Ashwin <[email protected]> wrote: > Hi Snehasis, > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > a versioning scheme for a connector config for the same connector (and not > different versions of a connector plugin). > > Sorry for not being more precise in my wording - I meant registering > versions of schema for connector config. > > Let's take the example of a fictional connector which uses a fictional AWS > service. > > Fictional Connector Config schema version:2.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#";, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "aws_access_key": { > "type": "string" > }, > "aws_secret_key": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "aws_access_key", > "aws_secret_key" > ] > } > > Fictional Connector config schema version:3.0 > --- > { > "$schema": "http://json-schema.org/draft-04/schema#";, > "type": "object", > "properties": { > "name": { > "type": "string" > }, > "schema_version": { > "type": "string" > }, > "iam_role": { > "type": "string" > } > }, > "required": [ > "name", > "schema_version", > "iam_role" > ] > } > > The connector which supports Fictional config schema 2.0 will validate the > access key and secret key. > Whereas a connector which supports config with schema version 3.0 will only > validate the IAM role. > > This is the alternative which I wanted to suggest. Each plugin will > register the schema versions of connector config which it supports. > > The plugin paths may be optionally different i.e we don't have to > mandatorily add a new plugin path to support a new schema version. > > Thanks, > Ashwin > > On Tue, Nov 22, 2022 at 12:47 PM Snehashis <[email protected]> > wrote: > > > Thanks for the input Ashwin. > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > config is versioned and has a schema. Then a single plugin (whose > > > dependencies have not changed) can handle multiple config versions for > > the > > > same connector class. > > > > IIUC (please correct me if I am wrong here), what you highlighted above, > is > > a versioning scheme for a connector config for the same connector (and > not > > different versions of a connector plugin). That is a somewhat tangential > > problem. While it is definitely a useful feature to have, like a log to > > check what changes were made over time to the config which might make it > > easier to do rollbacks, it is not the focus here. Here by version we mean > > to say what underlying version of the plugin should the given > configuration > > of the connector use. Perhaps it is better to change the name of the > > parameter from connector.version to connector.plugin.version or > > plugin.version if it was confusing. wdyt? > > > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > > connector/config?migrate=latest", the latest version __attempts__ to > > > transform the existing config to the newer version. This would require > > > adding a method like "boolean migrate(Version fromVersion)" to the > > > connector interface. > > > > This is an enhancement we can think of doing in future. Users can simply > do > > a PUT call with the updated config which has the updated version number. > > The assisted mode could be handy as the user does not need to know the > > config but beyond this it does not seem to justify its existence. > > > > Regards > > Snehashis > > > > On Tue, Nov 22, 2022 at 10:50 AM Ashwin <[email protected]> > > wrote: > > > > > Hi Snehasis, > > > > > > This is a really useful feature and thanks for initiating this > > discussion. > > > > > > I had the following questions - > > > > > > > > > 1. Can you elaborate on the rejected alternatives ? Suppose connector > > > config is versioned and has a schema. Then a single plugin (whose > > > dependencies have not changed) can handle multiple config versions for > > the > > > same connector class. > > > > > > 2. Any plans to support assisted migration e.g if a user invokes "POST > > > connector/config?migrate=latest", the latest version __attempts__ to > > > transform the existing config to the newer version. This would require > > > adding a method like "boolean migrate(Version fromVersion)" to the > > > connector interface. > > > > > > Thanks, > > > Ashwin > > > > > > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <[email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a discussion thread on KIP-891: Running multiple > > > versions > > > > of a connector. > > > > > > > > The KIP aims to add the ability for the connect runtime to run > multiple > > > > versions of a connector. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector > > > > > > > > Please take a look and let me know what you think. > > > > > > > > Thank you > > > > Snehashis Pal > > > > > > > > > >
