Hey Snehashis,
I updated the KIP to remove some stale mentions of the soft version
requirements, and the crashing workers on startup. I also added more detail
to the REST API.
IMHO we're ready to move to voting, so please open the thread if you also
believe it is ready.
Thanks!
Greg
On Fri, Aug 23, 2024 at 7:21 AM Snehashis <snehashisp1...@gmail.com> wrote:
> Hi Greg,
>
> Thanks for the clarification!
>
> I agree with not breaking compatibility and opting to not fail worker
> startup by validating converter versions.
>
> Let's also go with the unclosed variation of the requirements as a hard
> requirement i.e. 3.8 instead of [3.8]. I have updated the KIP and added a
> line there highlighting this.
>
> Regards
> Snehashis
>
> On Wed, Aug 21, 2024 at 9:24 PM Greg Harris <greg.har...@aiven.io.invalid>
> wrote:
>
> > Hey Snehashis,
> >
> > Thanks for your reply!
> >
> > > Deviating
> > > it from the spec seems unnecessary if we document it accordingly,
> however
> > > It's probably less intuitive and can lead to confusion. I would just
> keep
> > > it as is but making it simpler is also fine.
> >
> > It sounds like you don't have a strong opinion on this, similar to me.
> > Chris had a more firm stance on this. I think that you're right that we
> > could document this, but it would still be the single biggest foot-gun of
> > the feature, and I think we would regret it later.
> >
> > > Also for converter versions
> > > specified as part of the worker configs I believe we concluded that
> this
> > > step need not be fatal during worker startup if the required version is
> > not
> > > found but LMK if otherwise.
> >
> > Re-reading our earlier discussion, I think Chris had a strong opinion
> that
> > we shouldn't fail on startup because it would be inconsistent. I made an
> > offhand comment that if this was released in 4.0, we could change it so
> > both invalid classes and invalid versions cause the worker to fail, which
> > would be inconsistent but backwards incompatible.
> > I think in the interest of not breaking compatibility needlessly and
> > keeping consistent behavior, we should ignore invalid versions in the
> > worker config.
> >
> > Thanks,
> > Greg
> >
> > On Wed, Aug 21, 2024 at 1:58 AM Snehashis <snehashisp1...@gmail.com>
> > wrote:
> >
> > > Hi Greg,
> > >
> > > No issues, I have been caught up in a few things myself.
> > >
> > > I have added the points we discussed. In addition, I have added config
> > > providers as part of the set of plugins which will not support
> > versioning,
> > > for the same reason as to why it is not supported in the other plugins
> > that
> > > are initiated on startup.
> > >
> > > On whether to deviate from maven versioning for hard requirements as
> > > discussed between you and Chris.Whether to simply simply specify
> > > connector.version=1.1.1 as a hard requirement instead of [1.1.1].
> > Deviating
> > > it from the spec seems unnecessary if we document it accordingly,
> however
> > > It's probably less intuitive and can lead to confusion. I would just
> keep
> > > it as is but making it simpler is also fine. Also for converter
> versions
> > > specified as part of the worker configs I believe we concluded that
> this
> > > step need not be fatal during worker startup if the required version is
> > not
> > > found but LMK if otherwise.
> > >
> > > Regards
> > > Snehashis
> > >
> > >
> > >
> > > On Mon, Aug 19, 2024 at 9:24 PM Greg Harris
> <greg.har...@aiven.io.invalid
> > >
> > > wrote:
> > >
> > > > Hi Snehashis,
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > > Heterogeneous dependencies in a multi cluster deployment is highly
> > > > discouraged
> > > >
> > > > Right, this remains unchanged in this KIP.
> > > >
> > > > > Let's add the version information for both connector and tasks in
> the
> > > > connector status itself
> > > > > once we add these two additions to the KIP (LMK if you want me to
> > take
> > > > that up).
> > > >
> > > > Could you make these additions?
> > > >
> > > > I'm interested to see if we can include this in 4.0.
> > > >
> > > > Thanks,
> > > > Greg
> > > >
> > > > On Tue, Jul 2, 2024 at 2:52 AM Snehashis <snehashisp1...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Greg,
> > > > >
> > > > > Thanks for taking a look at this, to conclude on the two points
> > above.
> > > > >
> > > > > 1. I'm okay with the status quo of leaving the dependency
> management
> > of
> > > > > plugins to systems outside of the connect runtime as it is now.
> Given
> > > > that
> > > > > the dependencies are homogenous across a connect cluster, it should
> > > > ensure
> > > > > that task and connector versions are uniform across a deployment,
> > after
> > > > it
> > > > > has stabilised post an upgrade operation. Heterogeneous
> dependencies
> > > in a
> > > > > multi cluster deployment is highly discouraged and we should point
> > out
> > > > this
> > > > > can lead to inconsistent behaviour with connectors and even
> > > validations.
> > > > >
> > > > > 2. Let's add the version information for both connector and tasks
> in
> > > the
> > > > > connector status itself, I don't see another endpoint being
> required
> > > for
> > > > > this.
> > > > >
> > > > > I don't have any further points, so if you are okay we can put this
> > to
> > > a
> > > > > vote, once we add these two additions to the KIP (LMK if you want
> me
> > to
> > > > > take that up).
> > > > >
> > > > > Regards
> > > > > Snehashis
> > > > >
> > > > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris
> > > <greg.har...@aiven.io.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Snehashis,
> > > > > >
> > > > > > Sorry for the late reply, and thanks for helping close out the
> > > > > discussion.
> > > > > >
> > > > > > > Note that if my assumptions are correct then
> > > > > > > this can happen with the existing framework as well, or is
> there
> > > some
> > > > > > > safeguard from this happening?
> > > > > >
> > > > > > Currently, if a cluster has a heterogeneous plugin installation,
> > each
> > > > > > worker may have a different definition of "latest". If you call
> to
> > > > > > validate a configuration, it will validate against whatever
> latest
> > > > > version
> > > > > > is present locally.
> > > > > > With this KIP, that behavior will continue when the version isn't
> > > > > > specified, when the version is a soft requirement, or when the
> > > version
> > > > > is a
> > > > > > range. For users that do not want to tolerate their tasks having
> > > > > > heterogeneous versions, the single-version hard requirements are
> > > there,
> > > > > but
> > > > > > come at the cost of micromanagement of upgrades.
> > > > > > Perhaps there is some room in the future for a connector
> configured
> > > > with
> > > > > a
> > > > > > range to "pin" the version that all of its tasks should use. Or
> > > > > > version-aware-scheduling can schedule connectors and tasks to
> > workers
> > > > > that
> > > > > > will resolve the same versions. The important primitive is being
> > able
> > > > to
> > > > > > enforce which version is being used, everything else looks more
> > like
> > > a
> > > > > > convenience feature to me.
> > > > > >
> > > > > > > So far, we could have pointed to the
> > > > > > > misconfigured cluster configuration and somewhat differ this
> > > problem
> > > > to
> > > > > > > something outside of connect runtime.
> > > > > >
> > > > > > I think this will still be the case. Even with this feature, we
> > will
> > > > > still
> > > > > > recommend homogeneous clusters that have the same set of plugins
> > > > > installed
> > > > > > on every worker. Version-aware scheduling is a rejected
> > alternative,
> > > > and
> > > > > > your example for performing validation requests on an arbitrary
> > > worker
> > > > > is a
> > > > > > great example of a complication for heterogeneous clusters that
> > we're
> > > > not
> > > > > > ready to solve.
> > > > > >
> > > > > > > We can introduce
> > > > > > > a new path under this for version
> > > > (/connector/connector-name/version),
> > > > > > but
> > > > > > > perhaps adding this as part of the status is a valid
> alternative.
> > > > > >
> > > > > > I think either is fine. Adding a new field to the existing
> > endpoints
> > > is
> > > > > > fine, and shouldn't require special compatibility such as a query
> > > > > > parameter.
> > > > > >
> > > > > > > Also, to go further I think
> > > > > > > version information for tasks could also be available,
> > > > > >
> > > > > > I completely agree. The version of each connector and each task
> > > should
> > > > be
> > > > > > visible individually, since they may be different during a
> rolling
> > > > > upgrade.
> > > > > >
> > > > > > Thanks!
> > > > > > Greg
> > > > > >
> > > > > > On Tue, Jun 18, 2024 at 5:18 AM Snehashis <
> > snehashisp1...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg, Chris
> > > > > > >
> > > > > > > Thanks for the in-depth discussion, I have a couple of
> discussion
> > > > > points
> > > > > > > and would like your thoughts on this.
> > > > > > >
> > > > > > > 1) One concern I have with the new addition of 'soft' and
> 'hard'
> > > > > version
> > > > > > > requirements is that there could be a mismatch in the plugin
> > > version
> > > > > that
> > > > > > > two different tasks are running, if a soft requirement is
> > provided
> > > > and
> > > > > > the
> > > > > > > nodes a multi cluster deployment are not in sync w.r.t the
> plugin
> > > > > > versions
> > > > > > > that they are configured with. Note that if my assumptions are
> > > > correct
> > > > > > then
> > > > > > > this can happen with the existing framework as well, or is
> there
> > > some
> > > > > > > safeguard from this happening? So far, we could have pointed to
> > the
> > > > > > > misconfigured cluster configuration and somewhat differ this
> > > problem
> > > > to
> > > > > > > something outside of connect runtime. With this feature in
> place
> > > > > perhaps
> > > > > > > the expectation is more on connect to not be running with such
> > > > > > > inconsistency, especially if a connector version is specified.
> > This
> > > > is
> > > > > > also
> > > > > > > a problem with validation if different cluster have different
> > > > > > > configurations, as IIRC validations are local to the worker
> which
> > > > > > receives
> > > > > > > the rest call for validate. So, we might be validating with a
> > > certain
> > > > > > > version which is different from the one that will be used to
> > create
> > > > > > > connector and tasks. Again, this is likely how the current
> state
> > > is,
> > > > > but
> > > > > > > perhaps such inconsistencies warrant a deeper look with the
> > > addition
> > > > of
> > > > > > > this feature. The problems associated with them can be somewhat
> > > > > insidious
> > > > > > > and hard to diagnose.
> > > > > > >
> > > > > > > 2) There was some discussion on the need for a new REST
> endpoint
> > to
> > > > > > provide
> > > > > > > information on the versions of running connectors, and I think
> > > adding
> > > > > > this
> > > > > > > information via REST is a valuable addition. The way I see it
> the
> > > > > version
> > > > > > > is an intrinsic property of an instance of a running connector
> > and
> > > > > hence
> > > > > > > this should be part of the set of APIs under
> > > > > /connector/<connector-name>
> > > > > > > (also the /connectors API should also have this information as
> it
> > > is
> > > > an
> > > > > > > amalgamation of all the individual connector information). We
> can
> > > > > > introduce
> > > > > > > a new path under this for version
> > > > (/connector/connector-name/version),
> > > > > > but
> > > > > > > perhaps adding this as part of the status is a valid
> alternative.
> > > > This
> > > > > is
> > > > > > > mentioned as a rejected alternative right now. Also, to go
> > further
> > > I
> > > > > > think
> > > > > > > version information for tasks could also be available,
> especially
> > > if
> > > > we
> > > > > > > choose to not address the pitfalls discussed in my point 1),
> this
> > > > will
> > > > > > > at-least provide admins a quick and easy way to determine if
> such
> > > and
> > > > > > > inconsistent state exist in any of the connectors.
> > > > > > >
> > > > > > > Thanks again for reviving my original KIP and working to
> improve
> > > it.
> > > > > > > Looking forward to your thoughts on the points mentioned above.
> > > > > > > Regards
> > > > > > > Snehashis
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 9:59 PM Chris Egerton
> > > > <chr...@aiven.io.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > First, an apology! I mistakenly assumed that each plugin
> > appeared
> > > > > only
> > > > > > > once
> > > > > > > > in the responses from GET
> > > /connector-plugins?connectorsOnly=false.
> > > > > > Thank
> > > > > > > > you for correcting me and pointing out that all versions of
> > each
> > > > > plugin
> > > > > > > > appear in that response, which does indeed satisfy my desire
> > for
> > > > > users
> > > > > > to
> > > > > > > > discover this information in at most two REST requests (and
> in
> > > > fact,
> > > > > > does
> > > > > > > > it in only one)!
> > > > > > > >
> > > > > > > > And secondly, with the revelation about recommenders, I agree
> > > that
> > > > > it's
> > > > > > > > best to leave the "version" property out of the lists of
> > > properties
> > > > > > > > returned from the GET /connector-plugins/<plugin>/config
> > > endpoint.
> > > > > > > >
> > > > > > > > With those two points settled, I think the only unresolved
> item
> > > is
> > > > > the
> > > > > > > > small change to version parsing added to the KIP (where raw
> > > version
> > > > > > > numbers
> > > > > > > > are treated as an exact match, instead of a best-effort match
> > > with
> > > > a
> > > > > > > > fallback on the default version). If the KIP is updated with
> > that
> > > > > then
> > > > > > > I'd
> > > > > > > > be ready to vote on it.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Chris
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 12:00 PM Greg Harris
> > > > > > > <greg.har...@aiven.io.invalid
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Chris,
> > > > > > > > >
> > > > > > > > > Thanks for your thoughts.
> > > > > > > > >
> > > > > > > > > > Won't it still only expose the
> > > > > > > > > > latest version for each plugin, instead of the range of
> > > > versions
> > > > > > > > > available?
> > > > > > > > >
> > > > > > > > > Here is a snippet of the current output of the GET
> > > > > > > > > /connector-plugins?connectorsOnly=false endpoint, after I
> > > > installed
> > > > > > two
> > > > > > > > > versions of the debezium PostgresConnector:
> > > > > > > > >
> > > > > > > > > {
> > > > > > > > > "class":
> > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > "type": "source",
> > > > > > > > > "version": "2.0.1.Final"
> > > > > > > > > },
> > > > > > > > > {
> > > > > > > > > "class":
> > > > "io.debezium.connector.postgresql.PostgresConnector",
> > > > > > > > > "type": "source",
> > > > > > > > > "version": "2.6.1.Final"
> > > > > > > > > },
> > > > > > > > >
> > > > > > > > > I think this satisfies your requirement to learn about all
> > > > plugins
> > > > > > and
> > > > > > > > all
> > > > > > > > > versions in two or fewer REST calls.
> > > > > > > > >
> > > > > > > > > I tried to get an example of the output of `/config` by
> > > > hardcoding
> > > > > > the
> > > > > > > > > Recommender, and realized that Recommenders aren't executed
> > on
> > > > the
> > > > > > > > > `/config` endpoint at all: only during validation, when a
> > > > > > configuration
> > > > > > > > is
> > > > > > > > > actually present.
> > > > > > > > > And this led me to discover that the `/config` endpoint
> > > returns a
> > > > > > > > > List<ConfigKeyInfo>, and ConfigKeyInfo does not contain a
> > > > > > > > recommendedValues
> > > > > > > > > field. The ConfigValue field is the object which contains
> > > > > > > > > recommendedValues, and it is only generated during
> > validation.
> > > > > > > > > I think it's out of scope to start calling recommenders on
> > > empty
> > > > > > > configs
> > > > > > > > > that might throw exceptions, changing the existing REST
> > > entities,
> > > > > or
> > > > > > > > > changing the core ConfigDef implementation.
> > > > > > > > > Someone could add this functionality later, I don't think
> > it's
> > > > > > > necessary
> > > > > > > > > here.
> > > > > > > > >
> > > > > > > > > Then the question is: should "version" without recommenders
> > > > appear
> > > > > in
> > > > > > > > > non-connector plugins? I think I'd rather be consistent
> with
> > > > > > > "predicate"
> > > > > > > > > and "negate" on release, and let a later improvement add
> > them.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Wed, May 29, 2024 at 8:06 AM Chris Egerton
> > > > > > <chr...@aiven.io.invalid
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > I'm confused about the behavior for GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false. Won't it still
> > only
> > > > > expose
> > > > > > > the
> > > > > > > > > > latest version for each plugin, instead of the range of
> > > > versions
> > > > > > > > > available?
> > > > > > > > > >
> > > > > > > > > > I'm hoping we can provide a flow where people need at
> most
> > > two
> > > > > REST
> > > > > > > > calls
> > > > > > > > > > to discover 1) the complete set of plugins available on
> the
> > > > > worker
> > > > > > > > (which
> > > > > > > > > > is already possible with the endpoint under discussion)
> and
> > > 2)
> > > > > the
> > > > > > > set
> > > > > > > > of
> > > > > > > > > > versions available for a specific plugin on the worker
> > (which
> > > > > > doesn't
> > > > > > > > > > appear to be possible, at least for some plugin types).
> > This
> > > > > > wouldn't
> > > > > > > > > > require any out-of-band knowledge and would be valuable
> for
> > > > > > connector
> > > > > > > > > users
> > > > > > > > > > (who may want to, for example, know what their options
> are
> > > when
> > > > > > > > > considering
> > > > > > > > > > a plugin upgrade) and cluster administrators (who could
> use
> > > it
> > > > > as a
> > > > > > > > > sanity
> > > > > > > > > > check for the setup of their Kafka Connect clusters
> without
> > > > > having
> > > > > > to
> > > > > > > > > pore
> > > > > > > > > > over log files).
> > > > > > > > > >
> > > > > > > > > > As far as modifying the content of the GET
> > > > > > > > > > /connector-plugins/<plugin>/config endpoint to include a
> > > > > "version"
> > > > > > > > > property
> > > > > > > > > > goes, I think your point about returning that property
> from
> > > > > > requests
> > > > > > > to
> > > > > > > > > > that endpoint that include a version query parameter is
> > > > salient,
> > > > > > but
> > > > > > > it
> > > > > > > > > > also unfortunately applies to all types of plugin. I
> don't
> > > > think
> > > > > it
> > > > > > > > > should
> > > > > > > > > > completely disqualify that option. I also wasn't
> imagining
> > > > adding
> > > > > > > > > anything
> > > > > > > > > > besides that single property to that endpoint, so no new
> > > > > > "predicate"
> > > > > > > > > > property for SMTs, no new "negate" property for
> predicates,
> > > and
> > > > > no
> > > > > > > new
> > > > > > > > > > "type" property for either. I'm not necessarily opposed
> to
> > > > adding
> > > > > > > > those,
> > > > > > > > > > but it can be done without being pulled into the scope of
> > > this
> > > > > KIP.
> > > > > > > > > >
> > > > > > > > > > So to summarize how I imagine people might use the REST
> API
> > > > after
> > > > > > > this
> > > > > > > > > KIP:
> > > > > > > > > >
> > > > > > > > > > 1. Discover the set of plugins on the worker via GET
> > > > > > > > > > /connector-plugins?connectorsOnly=false
> > > > > > > > > > 2. For any of those plugins, discover the set of
> available
> > > > > versions
> > > > > > > by
> > > > > > > > > > examining the recommended values for the "version"
> property
> > > > after
> > > > > > > > hitting
> > > > > > > > > > the GET GET /connector-plugins/<plugin>/config endpoint
> > > > > > > > > >
> > > > > > > > > > I don't think this is necessarily optimal since plucking
> > the
> > > > > > > "version"
> > > > > > > > > > property out of the response from that endpoint might be
> a
> > > PITA
> > > > > for
> > > > > > > ad
> > > > > > > > > hoc
> > > > > > > > > > queries via, e.g., curl, but I think it'd be enough for
> > this
> > > > KIP.
> > > > > > > Still
> > > > > > > > > > open to other options though, and if I've misunderstood
> > > > anything
> > > > > in
> > > > > > > the
> > > > > > > > > > proposal, please let me know!
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > >
> > > > > > > > > > Chris
> > > > > > > > > >
> > > > > > > > > > On Wed, May 22, 2024 at 4:05 PM Greg Harris
> > > > > > > > <greg.har...@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hey Chris,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your comments, and I'm glad that it seems
> like
> > > > we're
> > > > > > > > > > > aligning on the vision here.
> > > > > > > > > > >
> > > > > > > > > > > > An
> > > > > > > > > > > > alternative could be to change existing behavior to
> > fail
> > > > fast
> > > > > > on
> > > > > > > > any
> > > > > > > > > > > > invalid default converter configuration instead of
> just
> > > for
> > > > > > > invalid
> > > > > > > > > > > > versions
> > > > > > > > > > >
> > > > > > > > > > > I suppose if this is landing in 4.0, we have the
> > > opportunity
> > > > to
> > > > > > > break
> > > > > > > > > > > compatibility and strictly validate the worker class
> > > configs,
> > > > > and
> > > > > > > > > > > could have a new consistent behavior instead of
> > > inconsistent
> > > > > but
> > > > > > > > > > > backwards-compatible behavior.
> > > > > > > > > > >
> > > > > > > > > > > > no other part of the KIP requires this change.
> > > > > > > > > > >
> > > > > > > > > > > This is correct, and a compelling argument. I'm fine
> > > leaving
> > > > > the
> > > > > > > > > > > strict worker validation off of this configuration to
> > > > > potentially
> > > > > > > be
> > > > > > > > > > > added later. If a KIP was raised to perform strict
> > > validation
> > > > > of
> > > > > > > the
> > > > > > > > > > > worker config, it would include the version config, and
> > can
> > > > > > address
> > > > > > > > > > > backwards compatibility for both configs together.
> > > > > > > > > > >
> > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > > types.
> > > > > > > > > > >
> > > > > > > > > > > Yes it is, and that started in a bugfix, once people
> > > started
> > > > > > using
> > > > > > > > > > > this endpoint:
> > > > > https://issues.apache.org/jira/browse/KAFKA-14843
> > > > > > > . I
> > > > > > > > > > > wasn't planning on converging the behavior in this KIP,
> > as
> > > I
> > > > > > > > > > > considered it out-of-scope, and was going to follow the
> > > > current
> > > > > > > > > > > behavior. To summarize:
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<connector>/config will emit:
> > > > > > > > > > > * connector.class (already implemented)
> > > > > > > > > > > * connector.version (new)
> > > > > > > > > > > * key.converter (already implemented)
> > > > > > > > > > > * key.converter.version (new)
> > > > > > > > > > > * value.converter (already implemented)
> > > > > > > > > > > * value.converter.version (new)
> > > > > > > > > > > * header.converter (already implemented)
> > > > > > > > > > > * header.converter.version (new)
> > > > > > > > > > > But will NOT emit:
> > > > > > > > > > > * transforms.<alias>.type
> > > > > > > > > > > * transforms.<alias>.version
> > > > > > > > > > > * transforms.<alias>.predicate
> > > > > > > > > > > * predicates.<alias>.type
> > > > > > > > > > > * predicates.<alias>.version
> > > > > > > > > > > * predicates.<alias>.negate
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<converter>/config will NOT
> emit:
> > > > > > > > > > > * "" (there's not even a ".class" prefix!)
> > > > > > > > > > > * version
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<transform>/config will NOT
> emit:
> > > > > > > > > > > * type
> > > > > > > > > > > * version
> > > > > > > > > > > * predicate
> > > > > > > > > > >
> > > > > > > > > > > GET /connector-plugins/<predicate>/config will NOT
> emit:
> > > > > > > > > > > * type
> > > > > > > > > > > * version
> > > > > > > > > > > * negate
> > > > > > > > > > >
> > > > > > > > > > > Do you want the converter, transform, and predicate
> > > endpoints
> > > > > > > > changed?
> > > > > > > > > > > Do you want just "version", or do you want all of the
> > > > prefixed
> > > > > > > > configs
> > > > > > > > > > > including "type", "predicate" and "negate"? How would
> you
> > > > want
> > > > > to
> > > > > > > > > > > handle the converters?
> > > > > > > > > > > And when I say the configs are "not part of the plugin
> > > config
> > > > > > > itself"
> > > > > > > > > > > I mean that saying that GET
> > > > > > > > > > > /connector-plugins/Flatten$Key/config?version=3.8.0
> has a
> > > > > > "version"
> > > > > > > > > > > config that must be "3.8.0" is a little bit nonsense,
> as
> > > the
> > > > > > > version
> > > > > > > > > > > is already specified.
> > > > > > > > > > >
> > > > > > > > > > > > IMO
> > > > > > > > > > > > it's worth including this information somewhere
> > directly
> > > > > > > accessible
> > > > > > > > > > > without
> > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> be
> > > fine
> > > > > > with
> > > > > > > > GET
> > > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > > endpoint
> > > > > > > > > > >
> > > > > > > > > > > You don't have to provide a configuration to call GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false , is that
> > endpoint
> > > > not
> > > > > > > close
> > > > > > > > > > > enough to what you have in mind? See also the Rejected
> > > > > > Alternative
> > > > > > > > > > > "Adding new REST API endpoints"
> > > > > > > > > > >
> > > > > > > > > > > If you're calling
> /connector-plugins/<transform>/config,
> > > you
> > > > > know
> > > > > > > the
> > > > > > > > > > > name of a plugin right? That either comes from
> > out-of-band
> > > > > > > knowledge,
> > > > > > > > > > > validating a connector config, or calling GET
> > > > > > > > > > > /connector-plugins?connectorsOnly=false.
> > > > > > > > > > > * If you have out-of-band knowledge of plugin classes,
> > > > perhaps
> > > > > > you
> > > > > > > > > > > have out-of-band knowledge of versions too.
> > > > > > > > > > > * If you've just validated a connector config, there
> > should
> > > > be
> > > > > an
> > > > > > > > > > > accompanying "version" field there with an accurate
> > default
> > > > > value
> > > > > > > and
> > > > > > > > > > > recommenders.
> > > > > > > > > > > * If you've called GET
> > > > /connector-plugins?connectorsOnly=false,
> > > > > > > that
> > > > > > > > > > > endpoint includes version information.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Greg
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 22, 2024 at 11:05 AM Chris Egerton
> > > > > > > > <chr...@aiven.io.invalid
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > >
> > > > > > > > > > > > Hope you had a nice weekend! Gonna try to keep things
> > > > > concise.
> > > > > > > > > > > >
> > > > > > > > > > > > Concluded points:
> > > > > > > > > > > >
> > > > > > > > > > > > RE version recommenders, I agree it's likely that
> > > > > programmatic
> > > > > > > UIs
> > > > > > > > > will
> > > > > > > > > > > > already be able to handle dynamic configuration
> > > > definitions,
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > detail
> > > > > > > > > > > > about SMTs is a great point. I still anticipate some
> > > > > > awkwardness
> > > > > > > > with
> > > > > > > > > > > > connector versions, though: if the latest version
> > > supports
> > > > > some
> > > > > > > new
> > > > > > > > > > > > properties, then a user switches to an earlier
> > version, a
> > > > UI
> > > > > > may
> > > > > > > > > > respond
> > > > > > > > > > > by
> > > > > > > > > > > > wiping values for these properties. I guess we can
> bite
> > > the
> > > > > > > bullet,
> > > > > > > > > > > though.
> > > > > > > > > > > >
> > > > > > > > > > > > RE double-dinging during preflight validation for
> > invalid
> > > > > > > > versions, I
> > > > > > > > > > > like
> > > > > > > > > > > > the analogy with login credentials. I'm convinced
> that
> > > the
> > > > > > > proposal
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > KIP is best 👍
> > > > > > > > > > > >
> > > > > > > > > > > > Continued points:
> > > > > > > > > > > >
> > > > > > > > > > > > RE failing on worker startup, sorry, I should be
> > clearer:
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > > _new_
> > > > > > > > > > > > justification for it that doesn't also apply to
> > existing
> > > > > > > behavior.
> > > > > > > > We
> > > > > > > > > > > > shouldn't diverge from existing behavior solely for
> > this
> > > > new
> > > > > > > case.
> > > > > > > > An
> > > > > > > > > > > > alternative could be to change existing behavior to
> > fail
> > > > fast
> > > > > > on
> > > > > > > > any
> > > > > > > > > > > > invalid default converter configuration instead of
> just
> > > for
> > > > > > > invalid
> > > > > > > > > > > > versions, but I'd vote to just stick to existing
> > behavior
> > > > and
> > > > > > not
> > > > > > > > > > > > complicate things, especially since no other part of
> > the
> > > > KIP
> > > > > > > > requires
> > > > > > > > > > > this
> > > > > > > > > > > > change.
> > > > > > > > > > > >
> > > > > > > > > > > > RE exposing the version property in the
> > > > > > > > > > > /connector-plugins/<plugin>/config
> > > > > > > > > > > > endpoint, the behavior is inconsistent across plugin
> > > types.
> > > > > > > Hitting
> > > > > > > > > the
> > > > > > > > > > > > endpoint for the FileStreamSinkConnector on version
> > 3.7.0
> > > > > > yields
> > > > > > > a
> > > > > > > > > > > response
> > > > > > > > > > > > that includes, among other things, the "topics",
> > > > > > "topics.regex",
> > > > > > > > and
> > > > > > > > > > > > "errors.tolerance" properties. I see that we don't do
> > > this
> > > > > > > > everywhere
> > > > > > > > > > > (the
> > > > > > > > > > > > examples you cite for SMT and converter properties
> are
> > > > > > accurate),
> > > > > > > > but
> > > > > > > > > > IMO
> > > > > > > > > > > > it's worth including this information somewhere
> > directly
> > > > > > > accessible
> > > > > > > > > > > without
> > > > > > > > > > > > having to provide a full connector config. FWIW I'd
> be
> > > fine
> > > > > > with
> > > > > > > > GET
> > > > > > > > > > > > /connector-plugins/<plugin>/versions as a first-class
> > > > > endpoint
> > > > > > > > either
> > > > > > > > > > > > instead of or in addition to adding recommended
> values
> > > for
> > > > > all
> > > > > > > > plugin
> > > > > > > > > > > > versions.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your continued work on this KIP, and with
> > the
> > > > > > progress
> > > > > > > > > we're
> > > > > > > > > > > > making I'm optimistic about its chances of appearing
> in
> > > > > 4.0.0.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > >
> > > > > > > > > > > > Chris
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, May 15, 2024 at 1:22 PM Greg Harris
> > > > > > > > > > <greg.har...@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your quick follow up.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > But this risk is already present with
> > > > > > > > > > > > > > existing error cases, and I don't see anything
> that
> > > > > > justifies
> > > > > > > > > > > changing
> > > > > > > > > > > > > > existing behavior with an invalid converter
> class,
> > or
> > > > > > > diverging
> > > > > > > > > > from
> > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the case of invalid converter versions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The justification is to fail-fast, and prevent REST
> > API
> > > > > users
> > > > > > > > from
> > > > > > > > > > > > > receiving errors from bad configs that they didn't
> > > write,
> > > > > or
> > > > > > > > maybe
> > > > > > > > > > > > > don't even know apply to them.
> > > > > > > > > > > > > Up until recently errors in these configurations
> > > surfaced
> > > > > as
> > > > > > > > > failures
> > > > > > > > > > > > > to create the connector, or failures to start, and
> > you
> > > > made
> > > > > > > them
> > > > > > > > > > > > > fail-fast during validation. I think this change is
> > in
> > > > the
> > > > > > same
> > > > > > > > > > > > > spirit, pulling the validation further forward and
> > not
> > > > > > letting
> > > > > > > > > errors
> > > > > > > > > > > > > lie dormant.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And to call back to your original concern about
> > > > > interrupting
> > > > > > > > > > > > > connectors that explicitly provide these
> > configurations
> > > > and
> > > > > > > don't
> > > > > > > > > use
> > > > > > > > > > > > > the worker configs: I expect that operators with a
> > > > majority
> > > > > > of
> > > > > > > > > these
> > > > > > > > > > > > > sorts of clients aren't going to be setting the
> > worker
> > > > > > .version
> > > > > > > > > > > > > properties, because it would have no effect on the
> > > > majority
> > > > > > of
> > > > > > > > > their
> > > > > > > > > > > > > connectors. They would be able to rely on
> > > > > > > backwards-compatibility
> > > > > > > > > and
> > > > > > > > > > > > > continue to ignore the class properties.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > CLI
> > > > > > > > > > > > > > and programmatic UI developers will want to
> develop
> > > > their
> > > > > > own
> > > > > > > > > > tooling
> > > > > > > > > > > > > > layers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a very compelling argument. Snehashis do
> you
> > > want
> > > > > to
> > > > > > > > figure
> > > > > > > > > > > > > out a REST API design for this use-case?
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the GET
> > /connector-plugins/<plugin>/config
> > > > > > > endpoint,
> > > > > > > > I
> > > > > > > > > > was
> > > > > > > > > > > > > > thinking about the response for non-connector
> > > plugins,
> > > > > > e.g.,
> > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would
> a
> > > > > > "version"
> > > > > > > > > > property
> > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I intended for the ".version" property to be like
> > other
> > > > > > > framework
> > > > > > > > > > > > > configs (".class", ".type", ".predicate",
> ".negate")
> > > > where
> > > > > > they
> > > > > > > > are
> > > > > > > > > > > > > inside the plugin namespace, but not part of the
> > plugin
> > > > > > config
> > > > > > > > > > itself.
> > > > > > > > > > > > > Perhaps we can deviate from those configs because
> it
> > > > would
> > > > > > aid
> > > > > > > in
> > > > > > > > > > > > > discovering other valid `GET
> > > > > > > > > > > > > /connector-plugins/<plugin>/config?version=` calls,
> > > > without
> > > > > > > > calling
> > > > > > > > > > > > > `GET /connector-plugins?connectorsOnly=false`.
> > > > > > > > > > > > > I don't really feel strongly either way.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > if a user changes the
> > > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> > the
> > > > UI
> > > > > > > either
> > > > > > > > > has
> > > > > > > > > > > to
> > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> risk
> > > > > > operating
> > > > > > > > on
> > > > > > > > > > > stale
> > > > > > > > > > > > > > information
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is a really interesting situation, thanks for
> > > > finding
> > > > > > > that!
> > > > > > > > > This
> > > > > > > > > > > > > is already a footgun with transformations and
> > > predicates;
> > > > > > Once
> > > > > > > > you
> > > > > > > > > > > > > fill out a transformation class (which includes a
> > > > > > recommender)
> > > > > > > > you
> > > > > > > > > > > > > will then see all of the transformations
> > > configurations.
> > > > > > > > > > > > > I think this means that UI developers should have
> > > already
> > > > > > > > developed
> > > > > > > > > > > > > the infrastructure for handling dynamic
> recommenders,
> > > or
> > > > > else
> > > > > > > > > they've
> > > > > > > > > > > > > had this bug since KIP-66 in 2017. It may require
> > some
> > > > > manual
> > > > > > > > > > > > > attention to roll-out support for the ".version"
> > > > properties
> > > > > > > > > > > > > specifically, but I don't think that should prevent
> > us
> > > > from
> > > > > > > > > providing
> > > > > > > > > > > > > this information in a natural place.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > but will it be possible to configure
> > > > > > > > > > > > > > a connector to use two instances of the same
> > > transform
> > > > or
> > > > > > > > > > predicate,
> > > > > > > > > > > but
> > > > > > > > > > > > > > with different versions for each?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, and this is included in the example in the
> KIP,
> > > > > > > > > > > > > `transforms.flatten-latest` and
> > > `transforms.flatten-old`.
> > > > > > This
> > > > > > > > will
> > > > > > > > > > > > > work in the natural way, with the two instances
> > having
> > > a
> > > > > > > > different
> > > > > > > > > > > > > version if configured for that. If a user wants to
> > pin
> > > > the
> > > > > > same
> > > > > > > > > > > > > version of a plugin for every instance, they will
> > need
> > > to
> > > > > > > provide
> > > > > > > > > the
> > > > > > > > > > > > > version config multiple times and keep them in-sync
> > > > > > themselves.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I would have expected the error
> > > > > > > > > > > > > > to only be attributed to the version property,
> and
> > > for
> > > > > the
> > > > > > > > class
> > > > > > > > > > > property
> > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I considered this, and then went back and changed
> it.
> > > To
> > > > > > quote
> > > > > > > > > > someone
> > > > > > > > > > > > > else, this is to catch "misspelling cat as dog".
> For
> > > > > example
> > > > > > a
> > > > > > > > user
> > > > > > > > > > > > > types DogConverter and meant to type CatConverter,
> > and
> > > > then
> > > > > > > > types a
> > > > > > > > > > > > > version which is valid for CatConverter,
> conceptually
> > > the
> > > > > > error
> > > > > > > > is
> > > > > > > > > in
> > > > > > > > > > > > > the class name and not the version.
> > > > > > > > > > > > > That's a very contrived scenario, but I think
> similar
> > > > > > arguments
> > > > > > > > are
> > > > > > > > > > > > > used for attributing validation errors to
> > > > > > > > endpoints/urls/hostnames
> > > > > > > > > > > > > when the associated credentials are unable to log
> > into
> > > > the
> > > > > > > remote
> > > > > > > > > > > > > system. Did the user provide the correct testing
> > > > > credentials,
> > > > > > > but
> > > > > > > > > > > > > accidentally typed the production endpoint? Even if
> > the
> > > > > > > > production
> > > > > > > > > > > > > endpoint looks valid (it's a real hostname that is
> > > > > reachable)
> > > > > > > the
> > > > > > > > > > > > > conceptual error is still in the hostname and
> should
> > > have
> > > > > the
> > > > > > > > error
> > > > > > > > > > > > > attributed to it to draw the user's attention.
> > > > > > > > > > > > > If that's not convincing, I think the alternative
> of
> > > only
> > > > > > > > > attributing
> > > > > > > > > > > > > version errors to the version property is also
> > > > acceptable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, May 15, 2024 at 8:06 AM Chris Egerton
> > > > > > > > > > <chr...@aiven.io.invalid
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your responses! Continuations of
> > existing
> > > > > > > > discussions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding crashing the worker on startup--yes,
> > there
> > > is
> > > > > > also
> > > > > > > a
> > > > > > > > > risk
> > > > > > > > > > > to
> > > > > > > > > > > > > > allowing it to join the cluster. But this risk is
> > > > already
> > > > > > > > present
> > > > > > > > > > > with
> > > > > > > > > > > > > > existing error cases, and I don't see anything
> that
> > > > > > justifies
> > > > > > > > > > > changing
> > > > > > > > > > > > > > existing behavior with an invalid converter
> class,
> > or
> > > > > > > diverging
> > > > > > > > > > from
> > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the case of invalid converter versions. I think
> we
> > > > should
> > > > > > > keep
> > > > > > > > > this
> > > > > > > > > > > > > simple
> > > > > > > > > > > > > > and not do anything different for worker startup
> > than
> > > > we
> > > > > do
> > > > > > > > > > already.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As far as REST API vs. metrics go--I think you're
> > > right
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > original
> > > > > > > > > > > > > > version metrics were added as a "monitoring"
> > detail.
> > > > > > However,
> > > > > > > > > this
> > > > > > > > > > > was
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > when plugin versions were managed solely by
> cluster
> > > > > > > > > administrators.
> > > > > > > > > > > With
> > > > > > > > > > > > > > this KIP, connector users will be able to manage
> > > plugin
> > > > > > > > versions,
> > > > > > > > > > > and CLI
> > > > > > > > > > > > > > and programmatic UI developers will want to
> develop
> > > > their
> > > > > > own
> > > > > > > > > > tooling
> > > > > > > > > > > > > > layers. I think focusing on the REST API as the
> > > primary
> > > > > > > > interface
> > > > > > > > > > for
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > KIP would be best for these users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > (All that said, I don't object to the metrics
> that
> > > are
> > > > > > > proposed
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > KIP;
> > > > > > > > > > > > > > I just think they make more sense in addition to
> > new
> > > > REST
> > > > > > API
> > > > > > > > > > > > > > functionality, as opposed to instead of it.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the GET
> > /connector-plugins/<plugin>/config
> > > > > > > endpoint,
> > > > > > > > I
> > > > > > > > > > was
> > > > > > > > > > > > > > thinking about the response for non-connector
> > > plugins,
> > > > > > e.g.,
> > > > > > > > > > > > > > GET /connector-plugins/RegexRouter/config. Would
> a
> > > > > > "version"
> > > > > > > > > > property
> > > > > > > > > > > > > > appear with recommended values?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And new thoughts:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) Regarding the recommended values for
> > > > > > "connector.version",
> > > > > > > > this
> > > > > > > > > > > might
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > confusing since there could be differences
> between
> > > the
> > > > > > > > ConfigDefs
> > > > > > > > > > > for the
> > > > > > > > > > > > > > latest version of the connector and prior
> versions.
> > > It
> > > > > also
> > > > > > > > makes
> > > > > > > > > > the
> > > > > > > > > > > > > flow
> > > > > > > > > > > > > > a bit awkward for programmatic UI developers: if
> a
> > > user
> > > > > > > changes
> > > > > > > > > the
> > > > > > > > > > > > > > connector version in, e.g., a dropdown menu, then
> > the
> > > > UI
> > > > > > > either
> > > > > > > > > has
> > > > > > > > > > > to
> > > > > > > > > > > > > > re-fetch the ConfigDef for the new version, or
> risk
> > > > > > operating
> > > > > > > > on
> > > > > > > > > > > stale
> > > > > > > > > > > > > > information. I'm starting to doubt that exposing
> > the
> > > > > range
> > > > > > of
> > > > > > > > > > > available
> > > > > > > > > > > > > > versions via recommended values is the best way
> to
> > > > > proceed,
> > > > > > > > > instead
> > > > > > > > > > > of a
> > > > > > > > > > > > > > more explicit approach like GET
> > > > > > > > > > > /connector-plugins/<plugin>/versions, or
> > > > > > > > > > > > > > the "Adding new REST API endpoints" rejected
> > > > alternative.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) I know that this is a bit heinous, but will it
> > be
> > > > > > possible
> > > > > > > > to
> > > > > > > > > > > > > configure
> > > > > > > > > > > > > > a connector to use two instances of the same
> > > transform
> > > > or
> > > > > > > > > > predicate,
> > > > > > > > > > > but
> > > > > > > > > > > > > > with different versions for each? (I don't think
> > this
> > > > is
> > > > > > > worth
> > > > > > > > > > > > > significant
> > > > > > > > > > > > > > design/implementation effort, so if it would
> > inflate
> > > > > either
> > > > > > > of
> > > > > > > > > > those,
> > > > > > > > > > > > > > please don't feel obligated to support it.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 3) Just out of curiosity, why double-ding during
> > > config
> > > > > > > > > validation
> > > > > > > > > > > if the
> > > > > > > > > > > > > > version for a plugin class can't be found? I
> would
> > > have
> > > > > > > > expected
> > > > > > > > > > the
> > > > > > > > > > > > > error
> > > > > > > > > > > > > > to only be attributed to the version property,
> and
> > > for
> > > > > the
> > > > > > > > class
> > > > > > > > > > > property
> > > > > > > > > > > > > > to be reported as valid.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 14, 2024 at 1:11 PM Greg Harris
> > > > > > > > > > > <greg.har...@aiven.io.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey Chris,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your comments. I'm glad you like the
> > > > > > > motivations,
> > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > wrote that part!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > the configuration syntax for the most basic
> use
> > > > case
> > > > > of
> > > > > > > > > > > > > > > > specifying a single desired version is pretty
> > > > > > > > > counterintuitive.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I agree, and the "soft" requirement scheme is
> > > > > something I
> > > > > > > > > wasn't
> > > > > > > > > > > > > > > explicitly looking for, but would be inherited
> > from
> > > > the
> > > > > > > > > library.
> > > > > > > > > > > I'm
> > > > > > > > > > > > > > > fine with eliminating the soft requirement
> > > semantics,
> > > > > and
> > > > > > > > > having
> > > > > > > > > > > > > > > "1.0.0" behave the same as "[1.0.0]". I'm less
> > > > inclined
> > > > > > to
> > > > > > > > > > include
> > > > > > > > > > > > > > > "range" in the property name, or have two
> > > properties.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Failing startup is drastic and has the
> > potential
> > > to
> > > > > > > disrupt
> > > > > > > > > the
> > > > > > > > > > > > > > > > availability of connectors that would
> otherwise
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > because they were explicitly configured to
> use
> > > > valid
> > > > > > > > > converters
> > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think this argument cuts both ways. If
> someone
> > > > > > > > reconfigures a
> > > > > > > > > > > worker
> > > > > > > > > > > > > > > and adds an invalid ".version" string to the
> > worker
> > > > (or
> > > > > > > > changes
> > > > > > > > > > the
> > > > > > > > > > > > > > > plugin.path to make it invalid), it would be
> > > > permitted
> > > > > to
> > > > > > > > enter
> > > > > > > > > > the
> > > > > > > > > > > > > > > group, and accept work assignments. If those
> work
> > > > > > > assignments
> > > > > > > > > > used
> > > > > > > > > > > > > > > these configurations, a set of tasks could
> > > transition
> > > > > to
> > > > > > > > FAILED
> > > > > > > > > > and
> > > > > > > > > > > > > > > not be able to recover, because they would be
> > > > restarted
> > > > > > > again
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > > > same worker.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Why are metrics utilized to report
> information
> > > > about
> > > > > > > plugin
> > > > > > > > > > > versions
> > > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > > publishing
> > > > > > > > this
> > > > > > > > > > > info in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > REST API
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I was following the existing pattern for
> exposing
> > > > > runtime
> > > > > > > > > > versions
> > > > > > > > > > > for
> > > > > > > > > > > > > > > connectors, and it did seem like a "monitoring"
> > > > > feature.
> > > > > > If
> > > > > > > > > that
> > > > > > > > > > > > > > > approach is flawed and should be phased out, I
> > > think
> > > > it
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > a
> > > > > > > > > > > > > > > good idea to reconsider the REST API rejected
> > > > > > alternative.
> > > > > > > > > > > > > > > We would need some additional design work to
> spec
> > > out
> > > > > the
> > > > > > > > REST
> > > > > > > > > > API
> > > > > > > > > > > > > > > interface, as I don't have anything in mind
> > > > currently.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm unclear on whether or not it'll be
> possible
> > > to
> > > > > see
> > > > > > > this
> > > > > > > > > > > > > information
> > > > > > > > > > > > > > > via the
> > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> endpoint
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There is room in the API to add recommenders
> for
> > > > > > > > > "key.converter",
> > > > > > > > > > > > > > > "value.converter", and "header.converter", but
> > not
> > > > for
> > > > > > > > > transforms
> > > > > > > > > > > and
> > > > > > > > > > > > > > > predicates, as they include aliases that depend
> > on
> > > an
> > > > > > > actual
> > > > > > > > > > > > > > > configuration. We could explicitly say we're
> > going
> > > to
> > > > > do
> > > > > > > > that,
> > > > > > > > > or
> > > > > > > > > > > do
> > > > > > > > > > > > > > > whatever is convenient during the
> implementation
> > > > phase,
> > > > > > or
> > > > > > > > > leave
> > > > > > > > > > it
> > > > > > > > > > > > > > > open to be improved later.
> > > > > > > > > > > > > > > There will not be any recommenders for
> ".version"
> > > > > > > properties
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > > > `/config` endpoint, because those recommenders
> > are
> > > > > > dynamic
> > > > > > > > and
> > > > > > > > > > > depend
> > > > > > > > > > > > > > > on an actual configuration.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 5) There are two relevant lines in the KIP:
> "If a
> > > > > > .version
> > > > > > > > > > property
> > > > > > > > > > > > > > > contains a hard requirement, select the latest
> > > > > installed
> > > > > > > > > version
> > > > > > > > > > > which
> > > > > > > > > > > > > > > satisfies the requirement." and "This
> > configuration
> > > > is
> > > > > > > > > > re-evaluated
> > > > > > > > > > > > > > > each time the connector or task are assigned
> to a
> > > new
> > > > > > > > worker".
> > > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > > > > call this "eager" upgrade behavior, rather
> than a
> > > > > > "sticky"
> > > > > > > or
> > > > > > > > > > > "lazy"
> > > > > > > > > > > > > > > upgrade behavior.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 6) Updated!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, May 14, 2024 at 9:14 AM Chris Egerton
> > > > > > > > > > > <chr...@aiven.io.invalid
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks Greg for updating the KIP, and thanks
> > > > > Snehashis
> > > > > > > for
> > > > > > > > > > > starting
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > work on this originally.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The motivation section makes a pretty
> > convincing
> > > > case
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > kind
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > feature. My thoughts are mostly about
> specific
> > > > > details:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) I like the support for version ranges (the
> > > > example
> > > > > > > > > > > demonstrating
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > avoid KAFKA-10574 with the header converter
> was
> > > > > > > > particularly
> > > > > > > > > > > > > > > entertaining),
> > > > > > > > > > > > > > > > but the configuration syntax for the most
> basic
> > > use
> > > > > > case
> > > > > > > of
> > > > > > > > > > > > > specifying a
> > > > > > > > > > > > > > > > single desired version is pretty
> > > counterintuitive.
> > > > > > People
> > > > > > > > may
> > > > > > > > > > get
> > > > > > > > > > > > > bitten
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > at least frustrated if they put
> > > > > connector.version=3.8.0
> > > > > > > in
> > > > > > > > a
> > > > > > > > > > > config
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > then version 3.7.5 ends up running. I'd like
> it
> > > if
> > > > we
> > > > > > > could
> > > > > > > > > > > either
> > > > > > > > > > > > > > > > intentionally deviate from Maven ranges when
> a
> > > bare
> > > > > > > version
> > > > > > > > > is
> > > > > > > > > > > > > present,
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > separate things out into two properties:
> > > > foo.version
> > > > > > > would
> > > > > > > > be
> > > > > > > > > > the
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > accepted version for the foo plugin, and
> > > > > > > foo.version.range
> > > > > > > > > > would
> > > > > > > > > > > use
> > > > > > > > > > > > > > > Maven
> > > > > > > > > > > > > > > > range syntax. Open to other options too, just
> > > > > > providing a
> > > > > > > > > > couple
> > > > > > > > > > > to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > ball rolling.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2) Although the current behavior for a worker
> > > with
> > > > an
> > > > > > > > invalid
> > > > > > > > > > > > > > > > key/value/header converter class specified in
> > its
> > > > > > config
> > > > > > > > file
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > strange (I was surprised to learn that it
> > > wouldn't
> > > > > fail
> > > > > > > on
> > > > > > > > > > > startup),
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > don't see a good reason to deviate from this
> > when
> > > > an
> > > > > > > > invalid
> > > > > > > > > > > version
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > specified. Failing startup is drastic and has
> > the
> > > > > > > potential
> > > > > > > > > to
> > > > > > > > > > > > > disrupt
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > availability of connectors that would
> otherwise
> > > be
> > > > > able
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > > > healthily
> > > > > > > > > > > > > > > > because they were explicitly configured to
> use
> > > > valid
> > > > > > > > > converters
> > > > > > > > > > > > > instead
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the worker defaults.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 3) Why are metrics utilized to report
> > information
> > > > > about
> > > > > > > > > plugin
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > utilized by connectors at runtime instead of
> > > > > publishing
> > > > > > > > this
> > > > > > > > > > > info in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > REST API? I saw that this was mentioned as a
> > > > rejected
> > > > > > > > > > > alternative,
> > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > didn't get a sense of why. It seems like the
> > REST
> > > > API
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > > easier to
> > > > > > > > > > > > > > > > access and more intuitive for most users than
> > new
> > > > > > > metrics.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 4) In the "Validation" section it's stated
> that
> > > > > "Users
> > > > > > > can
> > > > > > > > > use
> > > > > > > > > > > these
> > > > > > > > > > > > > > > > recommenders to discover the valid plugin
> > classes
> > > > and
> > > > > > > > > versions,
> > > > > > > > > > > > > without
> > > > > > > > > > > > > > > > requiring an earlier call to GET
> > > > > > > > > > > > > > > /connector-plugins?connectorsOnly=false."
> > > > > > > > > > > > > > > > I really like the creativity and simplicity
> of
> > > > > reusing
> > > > > > > the
> > > > > > > > > > > > > recommender
> > > > > > > > > > > > > > > > mechanism to expose available versions for
> > > plugins
> > > > > via
> > > > > > > the
> > > > > > > > > REST
> > > > > > > > > > > API.
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > unclear on whether or not it'll be possible
> to
> > > see
> > > > > this
> > > > > > > > > > > information
> > > > > > > > > > > > > via
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > GET /connector-plugins/<plugin>/config
> > endpoint,
> > > > > > though.
> > > > > > > > It'd
> > > > > > > > > > be
> > > > > > > > > > > > > great if
> > > > > > > > > > > > > > > > this were supported, since we learned in
> > KIP-769
> > > > [1]
> > > > > > that
> > > > > > > > > > people
> > > > > > > > > > > > > really
> > > > > > > > > > > > > > > > want to be able to see configuration options
> > for
> > > > > > > connectors
> > > > > > > > > and
> > > > > > > > > > > their
> > > > > > > > > > > > > > > > plugins via some kind of GET endpoint without
> > > > having
> > > > > to
> > > > > > > > > > provide a
> > > > > > > > > > > > > > > complete
> > > > > > > > > > > > > > > > connector config for validation.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 5) In the Maven version range docs, it's
> stated
> > > > that
> > > > > > > "Maven
> > > > > > > > > > > picks the
> > > > > > > > > > > > > > > > highest version of each project that
> satisfies
> > > all
> > > > > the
> > > > > > > hard
> > > > > > > > > > > > > requirements
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the dependencies on that project." I'm
> guessing
> > > > this
> > > > > > > > behavior
> > > > > > > > > > > will be
> > > > > > > > > > > > > > > > retained for Connect; i.e., the
> > highest-possible
> > > > > > version
> > > > > > > of
> > > > > > > > > > each
> > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > that satisfies the user-specified version
> > > > constraints
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > run?
> > > > > > > > > > > > > (An
> > > > > > > > > > > > > > > > alternative approach could be to have some
> kind
> > > of
> > > > > > > "sticky"
> > > > > > > > > > logic
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > restarts connectors/tasks when their
> > > currently-used
> > > > > > > version
> > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > incompatible with the configured
> constraints.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 6) (Nit) It'd be nice to add a link to the
> > > > > TestPlugins
> > > > > > > > class
> > > > > > > > > or
> > > > > > > > > > > > > somewhere
> > > > > > > > > > > > > > > > in its neighborhood to the testing plan;
> > > unfamiliar
> > > > > > > readers
> > > > > > > > > > > probably
> > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > get much out of what's there right now.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [1] -
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Chris
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, May 13, 2024 at 2:01 PM Snehashis <
> > > > > > > > > > > snehashisp1...@gmail.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > That is much appreciated. No complaints on
> > the
> > > > > > > additional
> > > > > > > > > > > scope, I
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > make some time out to work on this once we
> > have
> > > > > > > approval.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 9:28 PM Greg Harris
> > > > > > > > > > > > > > > <greg.har...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey Snehashis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm glad to hear you're still interested
> in
> > > > this
> > > > > > KIP!
> > > > > > > > > > > > > > > > > > I'm happy to let you drive this, and I
> > > > apologize
> > > > > > for
> > > > > > > > > > > increasing
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > scope of work so drastically. To make up
> > for
> > > > > that,
> > > > > > > I'll
> > > > > > > > > > > > > volunteer to
> > > > > > > > > > > > > > > > > > be the primary PR reviewer to help get
> this
> > > > done
> > > > > > > > quickly
> > > > > > > > > > once
> > > > > > > > > > > > > the KIP
> > > > > > > > > > > > > > > > > > is approved.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 3:51 AM
> Snehashis <
> > > > > > > > > > > > > snehashisp1...@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Greg,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the follow up to my original
> > > KIP,
> > > > I
> > > > > am
> > > > > > > in
> > > > > > > > > > > favour of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > additions made to expand its scope, the
> > > > > addition
> > > > > > of
> > > > > > > > > range
> > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > specifically make a lot of sense.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Apologies if I have not publicly worked
> > on
> > > > this
> > > > > > KIP
> > > > > > > > > for a
> > > > > > > > > > > long
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > original work was done when the move to
> > > > service
> > > > > > > > loading
> > > > > > > > > > > was in
> > > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > > and I wanted to loop back to this only
> > > after
> > > > > that
> > > > > > > > work
> > > > > > > > > > was
> > > > > > > > > > > > > > > completed.
> > > > > > > > > > > > > > > > > > Post
> > > > > > > > > > > > > > > > > > > its conclusion, I have not been able to
> > > take
> > > > > this
> > > > > > > up
> > > > > > > > > due
> > > > > > > > > > to
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > priorities. If it's okay with you, I
> > would
> > > > > still
> > > > > > > like
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > implemented myself, including the
> > > additional
> > > > > > scope.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks and regards
> > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Fri, May 10, 2024 at 12:45 AM Greg
> > > Harris
> > > > > > > > > > > > > > > > > > <greg.har...@aiven.io.invalid>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'd like to reboot the discussion on
> > > > KIP-891:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I've made some changes, most notably:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1. Specifying versions for all
> plugins
> > in
> > > > > > > Connector
> > > > > > > > > > > configs
> > > > > > > > > > > > > > > > > > > > (converters, header converters,
> > > transforms,
> > > > > and
> > > > > > > > > > > predicates)
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > > > > > > connectors & tasks
> > > > > > > > > > > > > > > > > > > > 2. Specifying a range of versions
> > instead
> > > > of
> > > > > an
> > > > > > > > exact
> > > > > > > > > > > match
> > > > > > > > > > > > > > > > > > > > 3. New metrics to observe what
> versions
> > > are
> > > > > > > in-use
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks to Snehashis for the original
> > KIP
> > > > > idea!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Tue, Jan 2, 2024 at 11:49 AM Greg
> > > > Harris <
> > > > > > > > > > > > > > > greg.har...@aiven.io>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Snehashis,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you for the KIP! This is
> > > something
> > > > > I've
> > > > > > > > > wanted
> > > > > > > > > > > for a
> > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I know the discussion has gone
> cold,
> > > are
> > > > > you
> > > > > > > > still
> > > > > > > > > > > > > interested
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > pursuing this feature? I'll make
> time
> > > to
> > > > > > review
> > > > > > > > the
> > > > > > > > > > > KIP if
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > still accepting comments.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Greg
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:29 PM
> > > > Snehashis
> > > > > <
> > > > > > > > > > > > > > > > > snehashisp1...@gmail.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the points Sagar.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > /connectors
> > > > > > > > > endpoint
> > > > > > > > > > to
> > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > > could
> > > > be
> > > > > > > > useful
> > > > > > > > > to
> > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> > gets
> > > > > > > returned
> > > > > > > > by
> > > > > > > > > > the
> > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think this is good to have and
> > > > possible
> > > > > > > > future
> > > > > > > > > > > > > > > enhancement. The
> > > > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > info will be present in the
> config
> > of
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > if
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > > specified the version. Otherwise
> it
> > > is
> > > > > the
> > > > > > > > latest
> > > > > > > > > > > version
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > > > can find out from the
> > > connector-plugin
> > > > > > > > endpoint.
> > > > > > > > > > The
> > > > > > > > > > > > > > > information
> > > > > > > > > > > > > > > > > > can be
> > > > > > > > > > > > > > > > > > > > > > introduced to the response of the
> > GET
> > > > > > > > /connectors
> > > > > > > > > > > > > endpoint
> > > > > > > > > > > > > > > > > itself,
> > > > > > > > > > > > > > > > > > > > however
> > > > > > > > > > > > > > > > > > > > > > the most ideal way of doing this
> > > would
> > > > be
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > instance of the connector and get
> > the
> > > > > > version
> > > > > > > > > > > directly
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > > > > > This is
> > > > > > > > > > > > > > > > > > > > > > slightly tricky as the connector
> > > could
> > > > be
> > > > > > > > running
> > > > > > > > > > in
> > > > > > > > > > > a
> > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > node.
> > > > > > > > > > > > > > > > > > > > > > One way to do this would be to
> > > persist
> > > > > the
> > > > > > > > > version
> > > > > > > > > > > > > > > information in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > status backing store during
> > > > instantiation
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > > > > > > > some more thought and since the
> > > version
> > > > > is
> > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > > configs if
> > > > > > > > > > > > > > > > > > > > provided
> > > > > > > > > > > > > > > > > > > > > > and evident otherwise, I have not
> > > > > included
> > > > > > it
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > hence
> > > > > > asking,
> > > > > > > > > can 2
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> Does
> > > the
> > > > > > > plugin
> > > > > > > > > > > isolation
> > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> > the
> > > > > > > lifecycle
> > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > All connectors in a cluster need
> to
> > > > have
> > > > > > > > uniquire
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > names
> > > > > > > > > > > > > > > > > > > > > > regardless of what version of the
> > > > plugin
> > > > > > the
> > > > > > > > > > > connector is
> > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > > > > underneath. This is something
> > > enforced
> > > > by
> > > > > > the
> > > > > > > > > > connect
> > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > itself.
> > > > > > > > > > > > > > > > > > > > All
> > > > > > > > > > > > > > > > > > > > > > connect CRUD operations are keyed
> > on
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > name
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > be an issue.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 3:16 PM
> > > Sagar <
> > > > > > > > > > > > > > > sagarmeansoc...@gmail.com
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hey Snehashsih,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP. It looks
> > like a
> > > > > very
> > > > > > > > useful
> > > > > > > > > > > > > feature.
> > > > > > > > > > > > > > > Couple
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > small-ish points, let me know
> > what
> > > > you
> > > > > > > think:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1) Should we update the GET
> > > > /connectors
> > > > > > > > > endpoint
> > > > > > > > > > to
> > > > > > > > > > > > > > > include the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin that is running? It
> > > could
> > > > be
> > > > > > > > useful
> > > > > > > > > to
> > > > > > > > > > > > > figure
> > > > > > > > > > > > > > > out
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > version of
> > > > > > > > > > > > > > > > > > > > > > > the plugin or I am assuming it
> > gets
> > > > > > > returned
> > > > > > > > by
> > > > > > > > > > the
> > > > > > > > > > > > > > > expand=info
> > > > > > > > > > > > > > > > > > call?
> > > > > > > > > > > > > > > > > > > > > > > 2) I am not aware of this and
> > hence
> > > > > > asking,
> > > > > > > > > can 2
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > different
> > > > > > > > > > > > > > > > > > > > > > > versions have the same name?
> Does
> > > the
> > > > > > > plugin
> > > > > > > > > > > isolation
> > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > this?
> > > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > > > could have a bearing when using
> > the
> > > > > > > lifecycle
> > > > > > > > > > > > > endpoints for
> > > > > > > > > > > > > > > > > > > > connectors like
> > > > > > > > > > > > > > > > > > > > > > > DELETE etc.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > > > > > > > > > Sagar.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 2:10 PM
> > > > Ashwin
> > > > > > > > > > > > > > > > > > <apan...@confluent.io.invalid
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if
> I
> > am
> > > > > wrong
> > > > > > > > > here),
> > > > > > > > > > > what
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > connector
> > > > > > > config
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > (and
> > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > connector
> > > > > > > plugin).
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Sorry for not being more
> > precise
> > > in
> > > > > my
> > > > > > > > > wording
> > > > > > > > > > > - I
> > > > > > > > > > > > > meant
> > > > > > > > > > > > > > > > > > > > registering
> > > > > > > > > > > > > > > > > > > > > > > > versions of schema for
> > connector
> > > > > > config.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Let's take the example of a
> > > > fictional
> > > > > > > > > connector
> > > > > > > > > > > which
> > > > > > > > > > > > > > > uses a
> > > > > > > > > > > > > > > > > > > > fictional
> > > > > > > > > > > > > > > > > > > > > > > AWS
> > > > > > > > > > > > > > > > > > > > > > > > service.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector Config
> > schema
> > > > > > > > version:2.0
> > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > http://json-schema.org/draft-04/schema#";,
> > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > "aws_access_key",
> > > > > > > > > > > > > > > > > > > > > > > > "aws_secret_key"
> > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Fictional Connector config
> > schema
> > > > > > > > version:3.0
> > > > > > > > > > > > > > > > > > > > > > > > ---
> > > > > > > > > > > > > > > > > > > > > > > > {
> > > > > > > > > > > > > > > > > > > > > > > > "$schema": "
> > > > > > > > > > > > > http://json-schema.org/draft-04/schema#";,
> > > > > > > > > > > > > > > > > > > > > > > > "type": "object",
> > > > > > > > > > > > > > > > > > > > > > > > "properties": {
> > > > > > > > > > > > > > > > > > > > > > > > "name": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "iam_role": {
> > > > > > > > > > > > > > > > > > > > > > > > "type": "string"
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > > },
> > > > > > > > > > > > > > > > > > > > > > > > "required": [
> > > > > > > > > > > > > > > > > > > > > > > > "name",
> > > > > > > > > > > > > > > > > > > > > > > > "schema_version",
> > > > > > > > > > > > > > > > > > > > > > > > "iam_role"
> > > > > > > > > > > > > > > > > > > > > > > > ]
> > > > > > > > > > > > > > > > > > > > > > > > }
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The connector which supports
> > > > > Fictional
> > > > > > > > config
> > > > > > > > > > > schema
> > > > > > > > > > > > > 2.0
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > validate
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > access key and secret key.
> > > > > > > > > > > > > > > > > > > > > > > > Whereas a connector which
> > > supports
> > > > > > config
> > > > > > > > > with
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > 3.0
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > only
> > > > > > > > > > > > > > > > > > > > > > > > validate the IAM role.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > This is the alternative
> which I
> > > > > wanted
> > > > > > to
> > > > > > > > > > > suggest.
> > > > > > > > > > > > > Each
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > register the schema versions
> of
> > > > > > connector
> > > > > > > > > > config
> > > > > > > > > > > > > which it
> > > > > > > > > > > > > > > > > > supports.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The plugin paths may be
> > > optionally
> > > > > > > > different
> > > > > > > > > > > i.e we
> > > > > > > > > > > > > > > don't
> > > > > > > > > > > > > > > > > > have to
> > > > > > > > > > > > > > > > > > > > > > > > mandatorily add a new plugin
> > path
> > > > to
> > > > > > > > support
> > > > > > > > > a
> > > > > > > > > > > new
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at 12:47
> > PM
> > > > > > > Snehashis
> > > > > > > > <
> > > > > > > > > > > > > > > > > > > > snehashisp1...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the input
> Ashwin.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> the
> > > > > > rejected
> > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > has a
> > > > > > schema.
> > > > > > > > > Then
> > > > > > > > > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > changed)
> > > > > can
> > > > > > > > handle
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > IIUC (please correct me if
> I
> > am
> > > > > wrong
> > > > > > > > > here),
> > > > > > > > > > > what
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > highlighted
> > > > > > > > > > > > > > > > > > > > > > > above,
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > a versioning scheme for a
> > > > connector
> > > > > > > > config
> > > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > same
> > > > > > > > > > > > > > > > > > > > connector (and
> > > > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > > > > different versions of a
> > > connector
> > > > > > > > plugin).
> > > > > > > > > > > That is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > > > > > > > > > > tangential
> > > > > > > > > > > > > > > > > > > > > > > > > problem. While it is
> > > definitely a
> > > > > > > useful
> > > > > > > > > > > feature to
> > > > > > > > > > > > > > > have,
> > > > > > > > > > > > > > > > > > like a
> > > > > > > > > > > > > > > > > > > > log to
> > > > > > > > > > > > > > > > > > > > > > > > > check what changes were
> made
> > > over
> > > > > > time
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > config
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > easier to do rollbacks, it
> is
> > > not
> > > > > the
> > > > > > > > focus
> > > > > > > > > > > here.
> > > > > > > > > > > > > Here
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > version we
> > > > > > > > > > > > > > > > > > > > > > > mean
> > > > > > > > > > > > > > > > > > > > > > > > > to say what underlying
> > version
> > > of
> > > > > the
> > > > > > > > > plugin
> > > > > > > > > > > > > should the
> > > > > > > > > > > > > > > > > given
> > > > > > > > > > > > > > > > > > > > > > > > configuration
> > > > > > > > > > > > > > > > > > > > > > > > > of the connector use.
> Perhaps
> > > it
> > > > is
> > > > > > > > better
> > > > > > > > > to
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > name of
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > parameter from
> > > connector.version
> > > > to
> > > > > > > > > > > > > > > > > connector.plugin.version
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > > > > plugin.version if it was
> > > > confusing.
> > > > > > > wdyt?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > > assisted
> > > > > > > > > migration
> > > > > > > > > > > e.g
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > connector/config?migrate=latest",
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > config
> > > > to
> > > > > > the
> > > > > > > > > newer
> > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > "boolean
> > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > This is an enhancement we
> can
> > > > think
> > > > > > of
> > > > > > > > > doing
> > > > > > > > > > in
> > > > > > > > > > > > > future.
> > > > > > > > > > > > > > > > > > Users can
> > > > > > > > > > > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > > > > > a PUT call with the updated
> > > > config
> > > > > > > which
> > > > > > > > > has
> > > > > > > > > > > the
> > > > > > > > > > > > > > > updated
> > > > > > > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > > > > number.
> > > > > > > > > > > > > > > > > > > > > > > > > The assisted mode could be
> > > handy
> > > > as
> > > > > > the
> > > > > > > > > user
> > > > > > > > > > > does
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > know the
> > > > > > > > > > > > > > > > > > > > > > > > > config but beyond this it
> > does
> > > > not
> > > > > > seem
> > > > > > > > to
> > > > > > > > > > > justify
> > > > > > > > > > > > > its
> > > > > > > > > > > > > > > > > > existence.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Regards
> > > > > > > > > > > > > > > > > > > > > > > > > Snehashis
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 22, 2022 at
> 10:50
> > > AM
> > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > <apan...@confluent.io.invalid>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Snehasis,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > This is a really useful
> > > feature
> > > > > and
> > > > > > > > > thanks
> > > > > > > > > > > for
> > > > > > > > > > > > > > > initiating
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > discussion.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I had the following
> > > questions -
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Can you elaborate on
> the
> > > > > > rejected
> > > > > > > > > > > > > alternatives ?
> > > > > > > > > > > > > > > > > Suppose
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > config is versioned and
> > has a
> > > > > > schema.
> > > > > > > > > Then
> > > > > > > > > > a
> > > > > > > > > > > > > single
> > > > > > > > > > > > > > > > > plugin
> > > > > > > > > > > > > > > > > > > > (whose
> > > > > > > > > > > > > > > > > > > > > > > > > > dependencies have not
> > > changed)
> > > > > can
> > > > > > > > handle
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > same connector class.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. Any plans to support
> > > > assisted
> > > > > > > > > migration
> > > > > > > > > > > e.g
> > > > > > > > > > > > > if a
> > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > > > invokes
> > > > > > > > > > > > > > > > > > > > > > > "POST
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > connector/config?migrate=latest",
> > > > > > the
> > > > > > > > > > latest
> > > > > > > > > > > > > version
> > > > > > > > > > > > > > > > > > > > __attempts__ to
> > > > > > > > > > > > > > > > > > > > > > > > > > transform the existing
> > config
> > > > to
> > > > > > the
> > > > > > > > > newer
> > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > require
> > > > > > > > > > > > > > > > > > > > > > > > > > adding a method like
> > "boolean
> > > > > > > > > > migrate(Version
> > > > > > > > > > > > > > > > > > fromVersion)" to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector interface.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Ashwin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 21, 2022 at
> > 2:27
> > > PM
> > > > > > > > > Snehashis <
> > > > > > > > > > > > > > > > > > > > snehashisp1...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to start a
> > > > discussion
> > > > > > > thread
> > > > > > > > > on
> > > > > > > > > > > > > KIP-891:
> > > > > > > > > > > > > > > > > Running
> > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > versions
> > > > > > > > > > > > > > > > > > > > > > > > > > > of a connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP aims to add the
> > > > ability
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > connect
> > > > > > > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > > > > > > versions of a
> connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+a+connector
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Please take a look and
> > let
> > > me
> > > > > > know
> > > > > > > > what
> > > > > > > > > > you
> > > > > > > > > > > > > think.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > > > > > > > > > > > > > Snehashis Pal
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
