Hi Satish, I think this qualifies as a blocker. This API has been around for years now and, while we don't document it as not exposing duplicates*, it has come with that implicit contract since its inception. More importantly, it has also never exposed plugins that cannot be used on the worker. This change in behavior not only introduces duplicates*, it causes unreachable plugins to be displayed. With this in mind, it seems to qualify pretty clearly as a regression and we should not put out a release that includes it.
* - Really, these aren't duplicates; rather, they're multiple copies of the same plugin that come from different locations on the worker Best, Chris On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana <satish.dugg...@gmail.com> wrote: > Hi Greg, > Is this API documented that it does not return duplicate entries? > > Can we also get an opinion from PMC/Committers who have KafkaConnect > expertise on whether this issue is a release blocker? > > If we agree that it is not a release blocker then we can have a > release note clarifying this behaviour and add a reference to the JIRA > that follows up on the possible solutions. > > Thanks, > Satish. > > > On Tue, 19 Sept 2023 at 03:29, Greg Harris <greg.har...@aiven.io.invalid> > wrote: > > > > Hey Satish, > > > > After investigating further, I believe that this is a regression, but > > mostly a cosmetic one. > > I don't think there is significant risk of breaking clients with this > > change, but it would be confusing for users, so I'd still like to get > > the fix into the next RC. > > I've opened a PR here: https://github.com/apache/kafka/pull/14398 and > > I'll work to get it merged promptly. > > > > Thanks! > > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris <greg.har...@aiven.io> > wrote: > > > > > > Hi Satish, > > > > > > While validating 3.6.0-rc0, I noticed this regression as compared to > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473 > > > > > > Impact: The `connector-plugins` endpoint lists duplicates which may > > > cause confusion for users, or poor behavior in clients. > > > Using the other REST API endpoints appears unaffected. > > > I'll open a PR for this later today. > > > > > > Thanks, > > > Greg > > > > > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana > > > <satish.dugg...@gmail.com> wrote: > > > > > > > > Thanks Justine for the update. I saw in the morning that these > changes > > > > are pushed to trunk and 3.6. > > > > > > > > ~Satish. > > > > > > > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > Hi Satish, > > > > > We were able to merge > > > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday > > > > > and pick to 3.6. > > > > > > > > > > Hopefully nothing more from me on this release. > > > > > > > > > > Thanks, > > > > > Justine > > > > > > > > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana < > satish.dugg...@gmail.com> > > > > > wrote: > > > > > > > > > > > Thanks Luke for the update. > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > On Thu, 14 Sept 2023 at 07:29, Luke Chen <show...@gmail.com> > wrote: > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > Since this PR: > > > > > > > https://github.com/apache/kafka/pull/14366 only changes the > doc, I've > > > > > > > backported to 3.6 branch. FYI. > > > > > > > > > > > > > > Thanks. > > > > > > > Luke > > > > > > > > > > > > > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan > > > > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects > 3.6. > > > > > > > > PR should be finalized soon. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Justine > > > > > > > > > > > > > > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri < > fedeval...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Satish, this is a small documentation fix about ZK to > KRaft > > > > > > > > > migration, that we would like to backport to 3.5 and 3.6 > branches. > > > > > > Are > > > > > > > > > you ok with that? > > > > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/14366 > > > > > > > > > > > > > > > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana < > > > > > > satish.dugg...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Thanks David for the quick resolution. > > > > > > > > > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > > > > > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur > > > > > > > > > > <david.art...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > Satish, > > > > > > > > > > > > > > > > > > > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, > and 3.4) > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > David > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma < > m...@ismaeljuma.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Justine, > > > > > > > > > > > > > > > > > > > > > > > > Probably best to have the conversation in the JIRA > ticket vs > > > > > > the > > > > > > > > > release > > > > > > > > > > > > thread. Generally, we want to only include low risk > bug fixes > > > > > > that > > > > > > > > > are > > > > > > > > > > > > fully compatible in patch releases. > > > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan > > > > > > > > > > > > <jols...@confluent.io.invalid> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Satish. I understand. > > > > > > > > > > > > > Just curious, is this something that could be > added to > > > > > > 3.6.1? It > > > > > > > > > would be > > > > > > > > > > > > > nice to say that hanging transactions are fully > covered in a > > > > > > 3.6 > > > > > > > > > release. > > > > > > > > > > > > > I'm not as familiar with the rules around minor > releases, but > > > > > > > > > adding it > > > > > > > > > > > > > there would give more time to ensure stability. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Justine > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana < > > > > > > > > > satish.dugg...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > We can skip this change into 3.6 now as it is > not a > > > > > > blocker or > > > > > > > > > > > > > > regression and it involves changes to the API > > > > > > implementation. > > > > > > > > > Let us > > > > > > > > > > > > > > plan to add the gap in the release notes as you > mentioned. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Satish. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan > > > > > > > > > > > > > > <jols...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hey Satish, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We just discovered a gap in KIP-890 part 1. We > currently > > > > > > > > don't > > > > > > > > > verify > > > > > > > > > > > > > on > > > > > > > > > > > > > > > txn offset commits, so it is still possible to > have > > > > > > hanging > > > > > > > > > > > > > transactions > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > the consumer offsets partitions. > > > > > > > > > > > > > > > I've opened a jira to wire the verification in > that > > > > > > request. > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15449 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This also isn't a regression, but it would be > nice to > > > > > > have > > > > > > > > > part 1 > > > > > > > > > > > > fully > > > > > > > > > > > > > > > complete. I have opened a PR with the fix: > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/14370. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I understand if there are concerns about last > minute > > > > > > changes > > > > > > > > > to this > > > > > > > > > > > > > API > > > > > > > > > > > > > > > and we can hold off if that makes the most > sense. > > > > > > > > > > > > > > > If we take that route, I think we should still > keep > > > > > > > > > verification for > > > > > > > > > > > > > the > > > > > > > > > > > > > > > data partitions since it still provides full > protection > > > > > > there > > > > > > > > > and > > > > > > > > > > > > > > improves > > > > > > > > > > > > > > > the transactions experience. We will need to > call out > > > > > > the gap > > > > > > > > > in the > > > > > > > > > > > > > > > release notes for consumer offsets partitions > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > > > > > > > > Justine > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur > > > > > > > > > > > > > > > <david.art...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Another (small) ZK migration issue was > identified. > > > > > > This one > > > > > > > > > isn't a > > > > > > > > > > > > > > > > regression (it has existed since 3.4), but I > think it's > > > > > > > > > reasonable > > > > > > > > > > > > to > > > > > > > > > > > > > > > > include. It's a small configuration check > that could > > > > > > > > > potentially > > > > > > > > > > > > save > > > > > > > > > > > > > > end > > > > > > > > > > > > > > > > users from some headaches down the line. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15450 > > > > > > > > > > > > > > > > https://github.com/apache/kafka/pull/14367 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we can get this one committed to > trunk today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma < > > > > > > > > > m...@ismaeljuma.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That sounds great. I think we should aim > to only > > > > > > allow > > > > > > > > > blockers > > > > > > > > > > > > > > > > > (regressions, impactful security issues, > etc.) on > > > > > > the 3.6 > > > > > > > > > branch > > > > > > > > > > > > > > until > > > > > > > > > > > > > > > > > 3.6.0 is out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 9, 2023, 12:20 AM Satish > Duggana < > > > > > > > > > > > > > > satish.dugg...@gmail.com> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ismael, > > > > > > > > > > > > > > > > > > It looks like we will publish RC0 by > 14th Sep. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Satish. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 8 Sept 2023 at 19:23, Ismael > Juma < > > > > > > > > > m...@ismaeljuma.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Do you have a sense of when we'll > publish RC0? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 8, 2023 at 6:27 AM David > Arthur > > > > > > > > > > > > > > > > > > > <david.art...@confluent.io.invalid> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Quick update on my two blockers: > KAFKA-15435 is > > > > > > > > > merged to > > > > > > > > > > > > > > trunk and > > > > > > > > > > > > > > > > > > > > cherry-picked to 3.6. I have a PR > open for > > > > > > > > > KAFKA-15441 and > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > > > hopefully > > > > > > > > > > > > > > > > > > > > get it merged today. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 8, 2023 at 5:26 AM Ivan > Yurchenko < > > > > > > > > > > > > > i...@ivanyu.me> > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Satish and all, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I wonder if > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14993 > > > > > > > > > > > > > > > > > > should be > > > > > > > > > > > > > > > > > > > > > included in the 3.6 release plan. > I'm > > > > > > thinking > > > > > > > > > that when > > > > > > > > > > > > > > > > > > implemented, it > > > > > > > > > > > > > > > > > > > > > would be a small, but still a > change in the > > > > > > RSM > > > > > > > > > contract: > > > > > > > > > > > > > > throw > > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > > > > > > exception instead of returning an > empty > > > > > > > > > InputStream. > > > > > > > > > > > > Maybe > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > included right away to save the > migration > > > > > > later? > > > > > > > > > What do > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > Ivan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 8, 2023, at 02:52, > Satish Duggana > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Jose, > > > > > > > > > > > > > > > > > > > > > > Thanks for looking into this > issue and > > > > > > > > resolving > > > > > > > > > it > > > > > > > > > > > > with > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > quick > > > > > > > > > > > > > > > > > > fix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ~Satish. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 7 Sept 2023 at 21:40, > José Armando > > > > > > > > García > > > > > > > > > > > > Sancio > > > > > > > > > > > > > > > > > > > > > > <jsan...@confluent.io.invalid> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Satish, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 6, 2023 at 4:58 PM > Satish > > > > > > > > Duggana < > > > > > > > > > > > > > > > > > > > > > satish.dugg...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Greg, > > > > > > > > > > > > > > > > > > > > > > > > It seems > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-14273 > > > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > > > > > been > > > > > > > > > > > > > > > > > > > > > > > > there in 3.5.x too. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I also agree that it should be > a blocker > > > > > > for > > > > > > > > > 3.6.0. > > > > > > > > > > > > It > > > > > > > > > > > > > > should > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > > > > > > > > been a blocker for those > previous > > > > > > releases. I > > > > > > > > > didn't > > > > > > > > > > > > > fix > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > because, > > > > > > > > > > > > > > > > > > > > > > > unfortunately, I wasn't aware > of the > > > > > > issue > > > > > > > > and > > > > > > > > > jira. > > > > > > > > > > > > > > > > > > > > > > > I'll create a PR with a fix in > case the > > > > > > > > > original > > > > > > > > > > > > author > > > > > > > > > > > > > > > > doesn't > > > > > > > > > > > > > > > > > > > > > respond in time. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Satish, do you agree? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > -José > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > -David > > > > > > > > > > > > > > > > > > > > > > > >
