Hi Snehasis,
> IIUC (please correct me if I am wrong here), what you highlighted above,
is
a versioning scheme for a connector config for the same connector (and not
different versions of a connector plugin).
Sorry for not being more precise in my wording - I meant registering
versions of schema for connector config.
Let's take the example of a fictional connector which uses a fictional AWS
service.
Fictional Connector Config schema version:2.0
---
{
"$schema": "http://json-schema.org/draft-04/schema#";,
"type": "object",
"properties": {
"name": {
"type": "string"
},
"schema_version": {
"type": "string"
},
"aws_access_key": {
"type": "string"
},
"aws_secret_key": {
"type": "string"
}
},
"required": [
"name",
"schema_version",
"aws_access_key",
"aws_secret_key"
]
}
Fictional Connector config schema version:3.0
---
{
"$schema": "http://json-schema.org/draft-04/schema#";,
"type": "object",
"properties": {
"name": {
"type": "string"
},
"schema_version": {
"type": "string"
},
"iam_role": {
"type": "string"
}
},
"required": [
"name",
"schema_version",
"iam_role"
]
}
The connector which supports Fictional config schema 2.0 will validate the
access key and secret key.
Whereas a connector which supports config with schema version 3.0 will only
validate the IAM role.
This is the alternative which I wanted to suggest. Each plugin will
register the schema versions of connector config which it supports.
The plugin paths may be optionally different i.e we don't have to
mandatorily add a new plugin path to support a new schema version.
Thanks,
Ashwin
On Tue, Nov 22, 2022 at 12:47 PM Snehashis <snehashisp1...@gmail.com> wrote:
> Thanks for the input Ashwin.
>
> > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > config is versioned and has a schema. Then a single plugin (whose
> > dependencies have not changed) can handle multiple config versions for
> the
> > same connector class.
>
> IIUC (please correct me if I am wrong here), what you highlighted above, is
> a versioning scheme for a connector config for the same connector (and not
> different versions of a connector plugin). That is a somewhat tangential
> problem. While it is definitely a useful feature to have, like a log to
> check what changes were made over time to the config which might make it
> easier to do rollbacks, it is not the focus here. Here by version we mean
> to say what underlying version of the plugin should the given configuration
> of the connector use. Perhaps it is better to change the name of the
> parameter from connector.version to connector.plugin.version or
> plugin.version if it was confusing. wdyt?
>
> > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > connector/config?migrate=latest", the latest version __attempts__ to
> > transform the existing config to the newer version. This would require
> > adding a method like "boolean migrate(Version fromVersion)" to the
> > connector interface.
>
> This is an enhancement we can think of doing in future. Users can simply do
> a PUT call with the updated config which has the updated version number.
> The assisted mode could be handy as the user does not need to know the
> config but beyond this it does not seem to justify its existence.
>
> Regards
> Snehashis
>
> On Tue, Nov 22, 2022 at 10:50 AM Ashwin <apan...@confluent.io.invalid>
> wrote:
>
> > Hi Snehasis,
> >
> > This is a really useful feature and thanks for initiating this
> discussion.
> >
> > I had the following questions -
> >
> >
> > 1. Can you elaborate on the rejected alternatives ? Suppose connector
> > config is versioned and has a schema. Then a single plugin (whose
> > dependencies have not changed) can handle multiple config versions for
> the
> > same connector class.
> >
> > 2. Any plans to support assisted migration e.g if a user invokes "POST
> > connector/config?migrate=latest", the latest version __attempts__ to
> > transform the existing config to the newer version. This would require
> > adding a method like "boolean migrate(Version fromVersion)" to the
> > connector interface.
> >
> > Thanks,
> > Ashwin
> >
> > On Mon, Nov 21, 2022 at 2:27 PM Snehashis <snehashisp1...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a discussion thread on KIP-891: Running multiple
> > versions
> > > of a connector.
> > >
> > > The KIP aims to add the ability for the connect runtime to run multiple
> > > versions of a connector.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > >
> > > Please take a look and let me know what you think.
> > >
> > > Thank you
> > > Snehashis Pal
> > >
> >
>
