Hi,
I have updated the KIP accordingly.
Thanks,
Mickael
On Fri, Jan 28, 2022 at 7:01 PM Chris Egerton
wrote:
>
> I think a simple approach here is fine. +1 for removing versioning for
> other plugins.
>
> On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison
> wrote:
>
> > Thanks Chris and Tom for t
I think a simple approach here is fine. +1 for removing versioning for
other plugins.
On Fri, Jan 28, 2022 at 11:17 AM Mickael Maison
wrote:
> Thanks Chris and Tom for the feedback.
>
> I'm having a hard time making a decision on this point!
>
> Re-reading your previous comments, I agree that we
Thanks Chris and Tom for the feedback.
I'm having a hard time making a decision on this point!
Re-reading your previous comments, I agree that we're lacking a
contract on the version field, and we can see value (supporting
multiple versions for example) if this was improved.
But I'm starting to w
To be honest I'm very sceptical that you can evolve (without breaking
compatibility) from a situation of having a large number of plugins which
have already implemented version() to return whatever they please, to a
situation where you can rely on it as the basis for some feature to support
multipl
Hi Mickael,
+1 for the warning at startup, it'd be great if we could get everyone to
start versioning all of their components. Although given the current state
of WARN-level logging in Connect (see
https://issues.apache.org/jira/browse/KAFKA-7509) there might be too much
noise for most people to n
Hi Chris,
Thanks for taking another look!
That's a good point which I should have mentioned in the KIP. I think
we can have a PossiblyVersioned interface that extends Versioned and
has the default implementation. I'd like to also print a warning at
startup if a plugin does not override version().
Hi Tom,
Thanks for the feedback.
You're right we don't have any contracts around the version field.
Like Chris I've only used the current field to find the exact source
code of connectors. Even if it's a limited use, I think that's enough
to justify adding it to all plugins and making them consis
Hi all,
With regards to Tom's question about possible uses of the version string,
one of the most useful applications I've found for it so far has been
debugging connector issues, especially based on captured logs. Using the
version string to identify a git tag containing the exact source code for
Hi Mickael,
Thanks for the KIP. Overall I think this looks good.
I have a small concern about the proliferation of Versioned. Currently this
interface has a really weak contract. There's no requirement for versions
to be comparable, or documentation about when versions should be changed by
author
Hi Arjun,
Thank you for taking a look at this KIP.
1.a Yes this is exactly what this KIP is proposing.
1.b I prefer returning all plugins together instead of having different paths.
2. While we seem to agree it would be nice to expose worker plugins, I
don't think we've reached a consensus on t
Thanks Randall, updated.
On Sun, Dec 5, 2021 at 6:05 AM Arjun Satish wrote:
>
> hey folks,
>
> Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
> (apologies if I missed some point in this thread).
>
> 1. it seems like we are changing the behavior of /connector-plugins b
hey folks,
Thanks a lot for the KIP and the discussions. Here are a couple of thoughts
(apologies if I missed some point in this thread).
1. it seems like we are changing the behavior of /connector-plugins by now
returning the list of transforms, converters as well? This would break
backward comp
Thanks, Mickael. Can we call out these as literals? For example:
> The response structure of the objects in the array remain unchanged, but this
> proposal does add "converter", "header_converter" and "transformation" to the
> allowed values for the "type" field, in addition to the existing "sin
Thanks, Chris and Mickael.
The use case does make sense, and I can see some value in it. We do
have precedent for some administrative endpoints, including KIP-495
[1] that adds the resource `admin/loggers`. Since the use case persona
is a Connect cluster administrator, perhaps we could move this
`
Hi Randall,
Sure. The types are sink, source, converter, header_converter,
transformation and predicate. I've updated the KIP.
Thanks,
Mickael
On Fri, Dec 3, 2021 at 10:14 PM Randall Hauch wrote:
>
> One more thing, inline below:
>
> Randall
>
> On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison
One more thing, inline below:
Randall
On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class o
Thanks for the quick response, Mickael. Responses inline below.
Randall
On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
>
Hi Mickael and Randall,
I think there's a decent use case for exposing worker plugins. If a cluster
administrator wants to enable a worker plugin on their cluster, it may make
things easier if they can verify ahead of time that that plugin is
available on every worker in the cluster (and possibly
Thanks Randall for the feedback!
1. Done
2. Yes that makes sense. I've updated the KIP
3.a I've updated to explicitly mention the plugin can be provided
using the fully qualified class or the alias.
3.b Only concrete classes will be returned. I made the mistake when I
crafted the examples by han
Thanks for the KIP, Mickael. And thanks to everyone else for the good
discussion so far.
Overall this is pretty good, but I do have a few questions/comments:
1. Can we include the HTTP method/verb in each of the endpoints in the
"Public Interfaces" section?
2. The sample output for `GET /connector
Hi Mickael,
Looks good to me! +1 non-binding when the vote thread opens :)
Cheers,
Chris
On Wed, Dec 1, 2021 at 8:53 AM Mickael Maison
wrote:
> Hi Chris,
>
> Thanks again for the feedback! I've updated the KIP based on our last
> discussions. I've decided to include the new endpoint for worke
Hi Chris,
Thanks again for the feedback! I've updated the KIP based on our last
discussions. I've decided to include the new endpoint for worker
plugins.
1. Yes I agree, it's best to gate the new behavior.
2. Yes, it was a remnant from the original proposal. I've now removed
the location field.
Hi Mickael,
I think that's a great idea! I especially like how we can establish the
expectation that any plugin type that appears in the response from the GET
/connector-plugins endpoint will have a corresponding GET
/connector-plugins//config endpoint, but (if we decide to add them in
the future)
Hi Chris,
Yes to keep compatibility we want a default implementation for
Converter.configs(), I've updated the KIP.
Regarding worker plugins, the use case you described seems valuable.
I'd prefer not mixing worker and connector plugins on the same
endpoint but I agree using /plugins and /worker-p
Hi Mickael,
I think one potential use case for exposing worker-level plugins is that it
may make it easier to determine whether a worker is set up correctly (the
current alternative requires looking through log files and can be a little
tedious), and might even make it possible to automatically id
Hi Chris,
1. If we want to expose worker plugins, I think we should do it via a
separate endpoint. But to be honest, I'm not even sure I see strong
use cases for exposing them as they are either enabled or not and
can't be changed at runtime. So I'd prefer to stick to "connector
level" plugins in
Hi Mickael,
I think the increase in scope here is great and the added value certainly
justifies the proposed changes. I have some thoughts but overall I like the
direction this is going in now.
1. The new /plugins endpoint is described as containing "all plugins that
are Connectors, Transformatio
Thanks all for the feedback!
Chris,
I agree that fixing the current endpoint helps a lot. Thanks for
raising these JIRAs and submitting a PR!
However thinking about the issue further, I decided to expand the
scope of the KIP to cover all user-visible plugins.
In practice, users want to know about
Hi,
I'm +1 for adding a GET endpoint for obtaining config definitions. It
always felt odd to me that one has to issue a PUT for that purpose. If
nothing else, it'd be better in terms of discoverability of the KC REST API.
One additional feature request I'd have is to expose the valid enum
constan
Hi Viktor,
It sounds like there are three major points here in favor of a new GET
endpoint for connector config defs.
1. You cannot issue a blank ("dummy") request for sink connectors because a
topic list/topic regex has to be supplied (otherwise the PUT endpoint
returns a 500 response)
2. A dumm
Hi Folks,
I too think this would be a very useful feature. Some of our management
applications would provide a wizard for creating connectors. In this
scenario the user basically would fill out a sample configuration generated
by the UI which would send it back to Connect for validation and eventu
I think it'd be worth adding a GET version, fwiw. Could be the same handler
with just a different spelling maybe.
On Fri, Aug 20, 2021, 7:44 AM Mickael Maison
wrote:
> Hi Chris,
>
> You're right, you can achieve the same functionality using the
> existing validate endpoint.
> In my mind it was o
Hi Chris,
You're right, you can achieve the same functionality using the
existing validate endpoint.
In my mind it was only for validation once you have build a
configuration but when used with an empty configuration, it basically
serves the same purpose as the proposed new endpoint.
I think it's
Hi Mickael,
I'm wondering about the use case here. The motivation section states that
"Connect does not provide a way to see what configurations a connector
requires. Instead users have to go look at the connector documentation or
in the worst case, look directly at the connector source code.", an
Hi,
I've created KIP-769 to expose connector configuration definitions in
the Connect API
https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+API+to+retrieve+connector+configuration+definitions
Please take a look and let me know if you have any feedback.
Thanks
35 matches
Mail list logo
