Thanks all for the feedback!
Chris,
I agree that fixing the current endpoint helps a lot. Thanks for
raising these JIRAs and submitting a PR!
However thinking about the issue further, I decided to expand the
scope of the KIP to cover all user-visible plugins.
In practice, users want to know about all available plugins not only
connectors. This includes transformations, converters,
header_converters and predicates. As we also want to retrieve
configdef for these too, I think it makes sense to introduce a new
endpoint to do so. Alongside we obviously need a new endpoint for
listing all plugins.
Gunnar,
I took a look at exposing valid values via the API. I think the issue
is that Validators don't expose a way to retrieve valid values.
Changing validators will have an impact on all components so I'd
prefer to address this requirement in a separate KIP. I agree this
would be an interesting improvement and I'd happy to write a KIP for
it too.
I have updated the KIP accordingly. Let me know if you have further feedback.
Thanks,
Mickael
On Tue, Nov 16, 2021 at 9:31 PM Gunnar Morling
<gunnar.morl...@googlemail.com.invalid> wrote:
>
> Hi,
>
> I'm +1 for adding a GET endpoint for obtaining config definitions. It
> always felt odd to me that one has to issue a PUT for that purpose. If
> nothing else, it'd be better in terms of discoverability of the KC REST API.
>
> One additional feature request I'd have is to expose the valid enum
> constants for enum-typed options. That'll help to display the values in a
> drop-down or via radio buttons in a UI, give us tab completion in kcctl,
> etc.
>
> Best,
>
> --Gunnar
>
>
> Am Di., 16. Nov. 2021 um 16:31 Uhr schrieb Chris Egerton
> <chr...@confluent.io.invalid>:
>
> > Hi Viktor,
> >
> > It sounds like there are three major points here in favor of a new GET
> > endpoint for connector config defs.
> >
> > 1. You cannot issue a blank ("dummy") request for sink connectors because a
> > topic list/topic regex has to be supplied (otherwise the PUT endpoint
> > returns a 500 response)
> > 2. A dummy request still triggers custom validations by the connector,
> > which may be best to avoid if we know for sure that the config isn't worth
> > validating yet
> > 3. It's more ergonomic and intuitive to be able to issue a GET request
> > without having to give a dummy connector config
> >
> > With regards to 1, this is actually a bug in Connect (
> > https://issues.apache.org/jira/browse/KAFKA-13327) with a fix already
> > implemented and awaiting committer review (
> > https://github.com/apache/kafka/pull/11369). I think it'd be better to
> > focus on fixing this bug in general instead of implementing a new REST
> > endpoint in order to allow people to work around it.
> >
> > With regards to 2, this is technically possible but I'm unsure it'd be too
> > common out in the wild given that most validations that could be expensive
> > would involve things like connecting to a database, checking if a cloud
> > storage bucket exists, etc., none of which are possible without some
> > configuration properties from the user (db hostname, bucket name, etc.).
> >
> > With regards to 3, I do agree that it'd be easier for people designing UIs
> > to have a GET API to work against. I'm just not sure it's worth the
> > additional implementation, testing, and maintenance burden. If it were
> > possible to issue a PUT request without unexpected 500s for invalid
> > configs, would that suffice? AFAICT it'd basically be as simple as issuing
> > a PUT request with a dummy body consisting of nothing except the connector
> > class (which at this point we might even make unnecessary and just
> > automatically replace with the connector class from the URL) and then
> > filtering the response to just grab the "definition" field of each element
> > in the "configs" array in the response.
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Nov 16, 2021 at 9:52 AM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > I too think this would be a very useful feature. Some of our management
> > > applications would provide a wizard for creating connectors. In this
> > > scenario the user basically would fill out a sample configuration
> > generated
> > > by the UI which would send it back to Connect for validation and
> > eventually
> > > create a new connector. The first part of this workflow can be enhanced
> > if
> > > we had an API that can return the configuration definition of the given
> > > type of connector as the UI application would be able to generate a
> > sample
> > > for the user based on that (nicely drawn diagram:
> > > https://imgur.com/a/7S1Xwm5).
> > > The connector-plugins/{connectorType}/config/validate API essentially
> > works
> > > and returns the data that we need, however it is a HTTP PUT API that is a
> > > bit unintuitive for a fetch-like functionality and also functionally
> > > different as it validates the given (dummy) request. In case of sink
> > > connectors one would need to also provide a topic name.
> > >
> > > A suggestion for the KIP: I think it can be useful to return the config
> > > groups and the connector class' name similarly to the validate API just
> > in
> > > case any frontend needs them (and also the response would be more like
> > the
> > > validate API but simpler).
> > >
> > > Viktor
> > >
> > > On Fri, Aug 20, 2021 at 4:51 PM Ryanne Dolan <ryannedo...@gmail.com>
> > > wrote:
> > >
> > > > I think it'd be worth adding a GET version, fwiw. Could be the same
> > > handler
> > > > with just a different spelling maybe.
> > > >
> > > > On Fri, Aug 20, 2021, 7:44 AM Mickael Maison <mickael.mai...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > You're right, you can achieve the same functionality using the
> > > > > existing validate endpoint.
> > > > > In my mind it was only for validation once you have build a
> > > > > configuration but when used with an empty configuration, it basically
> > > > > serves the same purpose as the proposed new endpoint.
> > > > >
> > > > > I think it's a bit easier to use a GET endpoint but I don't think it
> > > > > really warrants a different endpoint.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Thu, Aug 19, 2021 at 2:56 PM Chris Egerton
> > > > > <chr...@confluent.io.invalid> wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I'm wondering about the use case here. The motivation section
> > states
> > > > that
> > > > > > "Connect does not provide a way to see what configurations a
> > > connector
> > > > > > requires. Instead users have to go look at the connector
> > > documentation
> > > > or
> > > > > > in the worst case, look directly at the connector source code.",
> > and
> > > > that
> > > > > > with this KIP, "users will be able to discover the required
> > > > > configurations
> > > > > > for connectors installed in a Connect cluster" and "tools will be
> > > able
> > > > to
> > > > > > generate wizards for configuring and starting connectors".
> > > > > >
> > > > > > Does the existing "PUT
> > > > > /connector-plugins/{connector-type}/config/validate"
> > > > > > endpoint not address these points? What will the newly-proposed
> > > > endpoint
> > > > > > allow users to do that they will not already be able to do with the
> > > > > > existing endpoint?
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > On Thu, Aug 19, 2021 at 9:20 AM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I've created KIP-769 to expose connector configuration
> > definitions
> > > in
> > > > > > > the Connect API
> > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
> > > > > > >
> > > > > > > Please take a look and let me know if you have any feedback.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > >
> > > >
> > >
> >
