Re: [ANNOUNCE] New committer: Jeff Kim

[Josep Prat]

Congratulations Jeff!

Best,

On Mon, Sep 9, 2024 at 8:50 AM Chia-Ping Tsai  wrote:

> Congratulations, Jeff!  thanks for you to bring the great new group
> coordinator!
>
> Best,
> Chia-Ping
>
> David Jacot  於 2024年9月9日 週一 下午2:44寫道：
>
> > Hi all,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> Jeff
> > Kim.
> >
> > Jeff has been a Kafka contributor since May 2020. In addition to being
> > a regular contributor and reviewer, he has made significant
> > contributions to the next generation of the consumer rebalance
> > protocol (KIP-848) and to the new group coordinator. He authored
> > KIP-915 which improved how coordinators can be downgraded. He also
> > contributed multiple fixes/improvements to the fetch from follower
> > feature.
> >
> > Congratulations, Jeff!
> >
> > Thanks,
> > David (on behalf of the Apache Kafka PMC)
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [ANNOUNCE] New committer: Jeff Kim

[Kamal Chandraprakash]

Congrats, Jeff!

On Mon, Sep 9, 2024 at 12:20 PM Chia-Ping Tsai  wrote:

> Congratulations, Jeff!  thanks for you to bring the great new group
> coordinator!
>
> Best,
> Chia-Ping
>
> David Jacot  於 2024年9月9日 週一 下午2:44寫道：
>
> > Hi all,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> Jeff
> > Kim.
> >
> > Jeff has been a Kafka contributor since May 2020. In addition to being
> > a regular contributor and reviewer, he has made significant
> > contributions to the next generation of the consumer rebalance
> > protocol (KIP-848) and to the new group coordinator. He authored
> > KIP-915 which improved how coordinators can be downgraded. He also
> > contributed multiple fixes/improvements to the fetch from follower
> > feature.
> >
> > Congratulations, Jeff!
> >
> > Thanks,
> > David (on behalf of the Apache Kafka PMC)
> >
>

Re: New release branch 3.9

[Lucas Brutschy]

Hi Colin,

about KAFKA-17489, the bug Bruno mentioned. This needs to be fixed in
3.8 as well. The fix is probably small but tricky to get right - Bruno
has attempted to fix it, but soaking revealed that the fix is not
complete. Bruno is now out with Covid, so I will look into it. I'll
update this thread once I know more.

Cheers,
Lucas

On Mon, Sep 9, 2024 at 4:47 AM Colin McCabe  wrote:
>
> Hi Chia-Ping Tsai,
>
> Thanks for the bug report. Let’s follow up on this this week.
>
> Best,
> Colin
>
> On Fri, Sep 6, 2024, at 06:19, Chia-Ping Tsai wrote:
> > hi Colin
> >
> > I raise a issue (https://issues.apache.org/jira/browse/KAFKA-17492) as
> > 3.9 blocker, since the bug obstructs us from registering 3.9 broker to
> > 3.8 controller
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/08/09 01:46:35 Colin McCabe wrote:
> >> Hi all,
> >>
> >> I think it's time to transition the 3.9 branch to feature freeze. Please 
> >> note that this is 2 weeks past the original planned date. I apologize for 
> >> this; our planning wasn't perfect.
> >>
> >> A few updates on features that are in or out of Apache Kafka 3.9:
> >>
> >> - We have completed all the feature work for KIP-853: KRaft controller 
> >> membership changes. (Please note that there are still a few bug fixes 
> >> remaining, and the docs JIRA, KAFKA-17048, but those are all 
> >> post-feature-freeze tasks.)
> >>
> >> - After speaking with Calvin, I moved KIP-966 out of 3.9 and into 4.0. I 
> >> do feel bad about this but I just don't think we'll be able to get it into 
> >> this short release.
> >>
> >> There are a few other KIPs that probably need to be moved to 4.0, such as 
> >> KIP-1023 and KIP-1025. I will do a more thorough review tomorrow and reach 
> >> out to people if there are gray areas. If you have a feature that you have 
> >> questions about, please reach out.
> >>
> >> I would like to extend code freeze for Kafka 3.9 to August 29th. My 
> >> reasoning is that code freeze was 2 weeks later than expected, and also, I 
> >> will be out of the office most of next week.
> >>
> >> I have also created a PR to mark 3.9-IV0 as stable, so look for that 
> >> shortly in trunk and 3.9.
> >>
> >> As previously mentioned, please refrain from changing the Java version in 
> >> trunk (kafka 4.0) until we get an RC out. (However, please do continue 
> >> working on your 4.0 features and other refactors in trunk, even if they 
> >> don't apply to 3.9!)
> >>
> >> Thanks to everyone who has worked on this release, and thanks for your 
> >> patience. I do believe we will deliver on the promise of a 
> >> much-shorter-than-usual release cycle, even with the 2 week delay.
> >>
> >> best,
> >> Colin
> >>

Re: [ANNOUNCE] New committer: Jeff Kim

[Mickael Maison]

Congratulations Jeff!

On Mon, Sep 9, 2024 at 9:26 AM Kamal Chandraprakash
 wrote:
>
> Congrats, Jeff!
>
> On Mon, Sep 9, 2024 at 12:20 PM Chia-Ping Tsai  wrote:
>
> > Congratulations, Jeff!  thanks for you to bring the great new group
> > coordinator!
> >
> > Best,
> > Chia-Ping
> >
> > David Jacot  於 2024年9月9日 週一 下午2:44寫道：
> >
> > > Hi all,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Jeff
> > > Kim.
> > >
> > > Jeff has been a Kafka contributor since May 2020. In addition to being
> > > a regular contributor and reviewer, he has made significant
> > > contributions to the next generation of the consumer rebalance
> > > protocol (KIP-848) and to the new group coordinator. He authored
> > > KIP-915 which improved how coordinators can be downgraded. He also
> > > contributed multiple fixes/improvements to the fetch from follower
> > > feature.
> > >
> > > Congratulations, Jeff!
> > >
> > > Thanks,
> > > David (on behalf of the Apache Kafka PMC)
> > >
> >

[VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Kamal Chandraprakash]

Hi all,

I'd like to open voting for KIP-1058. This KIP improves the consumer
reading from remote storage when READ_COMMITTED isolation level is enabled.
PTAL.

KIP-1058


Thanks,
Kamal

[PR] [MINOR] Update Josep Role in committers.html [kafka-site]

[via GitHub]

jlprat opened a new pull request, #630:
URL: https://github.com/apache/kafka-site/pull/630

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [MINOR] Update Josep Role in committers.html [kafka-site]

[via GitHub]

jlprat commented on PR #630:
URL: https://github.com/apache/kafka-site/pull/630#issuecomment-2337607185

   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New committer: Jeff Kim

[Bill Bejeck]

Congrats Jeff!!

On Mon, Sep 9, 2024 at 7:50 AM Lianet M.  wrote:

> Congrats Jeff!!!
>
> On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton 
> wrote:
>
> > Congrats!
> >
> > On Mon, Sep 9, 2024, 06:36 Rajini Sivaram 
> wrote:
> >
> > > Congratulations, Jeff!
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Mon, Sep 9, 2024 at 10:49 AM Luke Chen  wrote:
> > >
> > > > Congrats, Jeff!
> > > >
> > > > On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
> > > >  wrote:
> > > >
> > > > > Congrats Jeff!
> > > > >
> > > > > On Mon, Sep 9, 2024, 11:02 Yash Mayya 
> wrote:
> > > > >
> > > > > > Congratulations Jeff!
> > > > > >
> > > > > > On Mon, 9 Sept, 2024, 12:13 David Jacot, 
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > committer,
> > > > > > Jeff
> > > > > > > Kim.
> > > > > > >
> > > > > > > Jeff has been a Kafka contributor since May 2020. In addition
> to
> > > > being
> > > > > > > a regular contributor and reviewer, he has made significant
> > > > > > > contributions to the next generation of the consumer rebalance
> > > > > > > protocol (KIP-848) and to the new group coordinator. He
> authored
> > > > > > > KIP-915 which improved how coordinators can be downgraded. He
> > also
> > > > > > > contributed multiple fixes/improvements to the fetch from
> > follower
> > > > > > > feature.
> > > > > > >
> > > > > > > Congratulations, Jeff!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > David (on behalf of the Apache Kafka PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Josep Prat

[Bruno Cadonna]

Congrats, Josep!

Best,
Bruno



On 9/7/24 12:15 AM, Viktor Somogyi-Vass wrote:

Congrats Kosep!

On Fri, Sep 6, 2024, 23:46 David Arthur  wrote:


Congrats, Josep! Very well deserved!

-David

On Fri, Sep 6, 2024 at 2:36 PM Kirk True 
wrote:


Congratulations Josep!

On Fri, Sep 6, 2024 at 11:06 AM Josep Prat 
wrote:


Thanks all!!

--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Sep 6, 2024, 19:45 Igor Soarez  wrote:


Congratulations Josep!

--
Igor








--
David Arthur

Re: [DISCUSS] KIP 1072 - Add @FunctionalInterface annotation to Kafka Streams SAM methods

[Ray McDermott]

It was on pause during the summer - vacations took priority.

I will check your feedback this week and give an update.

Ray


On Thursday, 22 August 2024 at 20:20, Matthias J. Sax  wrote:

> What is the status of this KIP?
> 
> On 7/30/24 7:36 PM, Matthias J. Sax wrote:
> 
> > Thanks a lot for the KIP
> > 
> > Ray
> > 
> > !
> > 
> > It seems to be a good improvement to make using KS with Clojure more
> > seamless.
> > 
> > However, I am not 100% sure if all listed interfaces make sense?
> > 
> > (100) GlobalKTable: it's basically a sibling to `KStream`, and `KTable`
> > interfaces, but users would never implemented it, and thus I think it
> > won't make much sense to add the annotation. Playing devils advocate, it
> > could even be "harmful" as it might provide a wrong signal to users that
> > this interface would be intended to be implemented by them.
> > 
> > (200) NamedOperation: this is some helper interface that user also won't
> > need to implement. I am less worried about being harmful (as I am for
> > GlobalKTable), but I also don't see much of an advantage.
> > 
> > (300) TransformerSupplier and ValueTransformerSupplier: both are going
> > to be deprecate with 4.0, which is only a side cleanup anyway. Both can
> > only be used with `KStream.transform()`, `.flatTransform()`,
> > `.transformValues()` and `.flatTransformValues()` and all four method
> > will be removed in 4.0 rending both interface practically useless. -- No
> > damage to include them, but also not useful.
> > 
> > (400) ProcessorSupplier and Processor: there is currently two interfaces
> > with each name, the old and already deprecated interfaces
> > `...processor.Processor[Supplier]` and the new
> > `...processor.api.Processor[Supplier]`. The KIP should be explicit and
> > say `api.Proceccor[Supplier]` as only the new interfaces have annotation
> > already, but not the old ones. - The old interfaces do also not need to
> > be updated IMHO, as will we remove both with 4.0, too.
> > 
> > (410) There is also two interfaces `...processor.ProcessorContext` and
> > `...processor.api.ProcessorContext`. Both are still in used, and thus
> > the KIP should mention both explicitly.
> > 
> > (500) I am not sure if the KIP need to explicitly list all interface it
> > does not update... It's a long list and it might be easier to read the
> > KIP if omitted?
> > 
> > (600) There is some other interface/abstract-classes which might benefit
> > from the annotation, too:
> > 
> > - org.apache.kafka.streams.errors:
> > 
> > DeserializationExceptionHandler (does not qualify now, but we could
> > include it in the KIP and file a follow up ticket for the future to add
> > it, when we remove the deprecated method? -- This would avoid the need
> > for another KIP in the future)
> > StreamsUncaughtExceptionHandler
> > 
> > - org.apache.kafka.streams.processor.api:
> > 
> > ContextualFixedKeyProcessor
> > ContextualProcessor
> > 
> > - org.apache.kafka.streams.processor:
> > 
> > CommitCallback
> > Punctuator
> > StateRestoreCallback
> > StreamPartitioner (there is already a open PR to remove the
> > deprecate method so it will qualify in 4.0 release)
> > TimestampExtractor
> > TopicNameExtractor
> > 
> > -Matthias
> > 
> > On 7/29/24 12:47 AM, Ray McDermott wrote:
> > 
> > > These annotations assist with clarifying the purpose of the interface,
> > > as well as assisting interop with non-Java JVM languages.
> > > 
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1072%3A+Add+@FunctionalInterface+annotation+to+Kafka+Streams+SAM+methods
> > > 
> > > Comments welcome
> > > 
> > > Thanks
> > > 
> > > Ray

Re: [VOTE] KIP-1074: Allow the replication of user internal topics

[Chris Egerton]

Hi Patrik,

Sorry for the delay, I've been on PTO. LGTM! +1 (binding)

Cheers,

Chris

On Tue, Sep 3, 2024 at 5:10 AM Viktor Somogyi-Vass
 wrote:

> Hi Patrik,
>
> Thanks for working on this. +1 from me (binding).
>
> Best,
> Viktor
>
> On Wed, Aug 28, 2024 at 1:34 PM Patrik Marton  >
> wrote:
>
> > Hi all,
> >
> > As the proposal is finalized, and there was no new feedback in the past
> > week, I would like to open voting for KIP-1074
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1074%3A+Allow+the+replication+of+user+internal+topics
> > >
> > .
> >
> > Please vote / let me know your thoughts.
> >
> > Thanks,
> > Patrik
> >
>

Re: [ANNOUNCE] New committer: Jeff Kim

[Satish Duggana]

Congratulations Jeff!

On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna  wrote:
>
> Congrats! Well deserved!
>
> Best,
> Bruno
>
>
>
> On 9/9/24 2:28 PM, Bill Bejeck wrote:
> > Congrats Jeff!!
> >
> > On Mon, Sep 9, 2024 at 7:50 AM Lianet M.  wrote:
> >
> >> Congrats Jeff!!!
> >>
> >> On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton 
> >> wrote:
> >>
> >>> Congrats!
> >>>
> >>> On Mon, Sep 9, 2024, 06:36 Rajini Sivaram 
> >> wrote:
> >>>
>  Congratulations, Jeff!
> 
>  Regards,
> 
>  Rajini
> 
>  On Mon, Sep 9, 2024 at 10:49 AM Luke Chen  wrote:
> 
> > Congrats, Jeff!
> >
> > On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Congrats Jeff!
> >>
> >> On Mon, Sep 9, 2024, 11:02 Yash Mayya 
> >> wrote:
> >>
> >>> Congratulations Jeff!
> >>>
> >>> On Mon, 9 Sept, 2024, 12:13 David Jacot, 
> >> wrote:
> >>>
>  Hi all,
> 
>  The PMC of Apache Kafka is pleased to announce a new Kafka
>  committer,
> >>> Jeff
>  Kim.
> 
>  Jeff has been a Kafka contributor since May 2020. In addition
> >> to
> > being
>  a regular contributor and reviewer, he has made significant
>  contributions to the next generation of the consumer rebalance
>  protocol (KIP-848) and to the new group coordinator. He
> >> authored
>  KIP-915 which improved how coordinators can be downgraded. He
> >>> also
>  contributed multiple fixes/improvements to the fetch from
> >>> follower
>  feature.
> 
>  Congratulations, Jeff!
> 
>  Thanks,
>  David (on behalf of the Apache Kafka PMC)
> 
> >>>
> >>
> >
> 
> >>>
> >>
> >

[jira] [Created] (KAFKA-17506) ZkMigrationFailoverTest is flaky

[David Arthur (Jira)]

David Arthur created KAFKA-17506:


 Summary: ZkMigrationFailoverTest is flaky
 Key: KAFKA-17506
 URL: https://issues.apache.org/jira/browse/KAFKA-17506
 Project: Kafka
  Issue Type: Test
Reporter: David Arthur






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Jeff Kim

[Matthias J. Sax]

Congrats!

On 9/9/24 12:34 PM, José Armando García Sancio wrote:

Congratulations Jeff!

On Mon, Sep 9, 2024 at 11:45 AM Justine Olshan
 wrote:


Congratulations Jeff!

On Mon, Sep 9, 2024 at 8:33 AM Satish Duggana 
wrote:


Congratulations Jeff!

On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna  wrote:


Congrats! Well deserved!

Best,
Bruno



On 9/9/24 2:28 PM, Bill Bejeck wrote:

Congrats Jeff!!

On Mon, Sep 9, 2024 at 7:50 AM Lianet M.  wrote:


Congrats Jeff!!!

On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton 


wrote:


Congrats!

On Mon, Sep 9, 2024, 06:36 Rajini Sivaram 

wrote:



Congratulations, Jeff!

Regards,

Rajini

On Mon, Sep 9, 2024 at 10:49 AM Luke Chen 

wrote:



Congrats, Jeff!

On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
 wrote:


Congrats Jeff!

On Mon, Sep 9, 2024, 11:02 Yash Mayya 

wrote:



Congratulations Jeff!

On Mon, 9 Sept, 2024, 12:13 David Jacot, 

wrote:



Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka

committer,

Jeff

Kim.

Jeff has been a Kafka contributor since May 2020. In addition

to

being

a regular contributor and reviewer, he has made significant
contributions to the next generation of the consumer rebalance
protocol (KIP-848) and to the new group coordinator. He

authored

KIP-915 which improved how coordinators can be downgraded. He

also

contributed multiple fixes/improvements to the fetch from

follower

feature.

Congratulations, Jeff!

Thanks,
David (on behalf of the Apache Kafka PMC)

Re: [ANNOUNCE] New committer: Jeff Kim

[David Arthur]

Nice! Congrats Jeff!

On Mon, Sep 9, 2024 at 9:25 PM Matthias J. Sax  wrote:

> Congrats!
>
> On 9/9/24 12:34 PM, José Armando García Sancio wrote:
> > Congratulations Jeff!
> >
> > On Mon, Sep 9, 2024 at 11:45 AM Justine Olshan
> >  wrote:
> >>
> >> Congratulations Jeff!
> >>
> >> On Mon, Sep 9, 2024 at 8:33 AM Satish Duggana  >
> >> wrote:
> >>
> >>> Congratulations Jeff!
> >>>
> >>> On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna 
> wrote:
> 
>  Congrats! Well deserved!
> 
>  Best,
>  Bruno
> 
> 
> 
>  On 9/9/24 2:28 PM, Bill Bejeck wrote:
> > Congrats Jeff!!
> >
> > On Mon, Sep 9, 2024 at 7:50 AM Lianet M.  wrote:
> >
> >> Congrats Jeff!!!
> >>
> >> On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton <
> fearthecel...@gmail.com
> 
> >> wrote:
> >>
> >>> Congrats!
> >>>
> >>> On Mon, Sep 9, 2024, 06:36 Rajini Sivaram  >
> >> wrote:
> >>>
>  Congratulations, Jeff!
> 
>  Regards,
> 
>  Rajini
> 
>  On Mon, Sep 9, 2024 at 10:49 AM Luke Chen 
> >>> wrote:
> 
> > Congrats, Jeff!
> >
> > On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Congrats Jeff!
> >>
> >> On Mon, Sep 9, 2024, 11:02 Yash Mayya 
> >> wrote:
> >>
> >>> Congratulations Jeff!
> >>>
> >>> On Mon, 9 Sept, 2024, 12:13 David Jacot, 
> >> wrote:
> >>>
>  Hi all,
> 
>  The PMC of Apache Kafka is pleased to announce a new Kafka
>  committer,
> >>> Jeff
>  Kim.
> 
>  Jeff has been a Kafka contributor since May 2020. In addition
> >> to
> > being
>  a regular contributor and reviewer, he has made significant
>  contributions to the next generation of the consumer rebalance
>  protocol (KIP-848) and to the new group coordinator. He
> >> authored
>  KIP-915 which improved how coordinators can be downgraded. He
> >>> also
>  contributed multiple fixes/improvements to the fetch from
> >>> follower
>  feature.
> 
>  Congratulations, Jeff!
> 
>  Thanks,
>  David (on behalf of the Apache Kafka PMC)
> 
> >>>
> >>
> >
> 
> >>>
> >>
> >
> >>>
> >
> >
> >
>


-- 
David Arthur

[jira] [Resolved] (KAFKA-17497) Add e2e for zk migration with old controller

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17497.

Resolution: Fixed

trunk: 
https://github.com/apache/kafka/commit/2dc3ee05579ddcf70a072fecd8db68ab9d1bbdb2

3.9: 
https://github.com/apache/kafka/commit/4b6437e6a523f72ef9d1572a77933057dbafafdd

> Add e2e for zk migration with old controller
> 
>
> Key: KAFKA-17497
> URL: https://issues.apache.org/jira/browse/KAFKA-17497
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Blocker
> Fix For: 3.9.0
>
>
> This is follow-up of KAFKA-17011 that zk migration will encounter min=0 issue 
> when the cluster consists of "new" broker and "old" controller.
> The e2e `zookeeper_migration_test` [0] is a good test for online migration, 
> but it does not test hybrid version. Hence, we can leverage it to increase 
> test coverage.
>  [0] 
> https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/zookeeper_migration_test.py#L91



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1088: Replace KafkaClientSupplier with KafkaClientInterceptor

[Sophie Blee-Goldman]

I agree with Alieh, there is definitely some inconsistency in what we do or
don't want to support/allow. However imo we should lean into the more
flexible approach of allowing an alternative consumer to be constructed and
returned, rather than attempting to enforce the interceptor to strictly
wrap the provided consumer. Mainly because the ability to bypass the "real"
KafkaConsumer and mock/stub methods is pretty essential to the test/mock
consumer use cases. Personally I feel this is a very advanced feature and
we should let people who know what they're doing implement any kind of
consumer for this, and if they misuse the API and break things then it's on
them. I feel we can document things clearly enough so that it will be
difficult to shoot yourself in the foot by accident (plus if they do end up
breaking things eg by constructing a new Consumer and just returning that
instead, it will at least break quickly and obviously. Figuring out that
the problem was in your interceptor might be challenging though. Whatever
we end up doing, if there's a way for someone to implement it incorrectly
then it would be good to figure out how we can detect this and warn the
user. I'm not quite sure how much of a safety net we can construct here but
we could try to cover some of the obvious misssteps, for example by
asserting that the returned Consumer is not an instance of KafkaConsumer
unless it's the same exact KafkaConsumer object passed into the
#getMainConsumer call. Not sure if there are other checks we could do?

I do suspect this problem just goes away if we pivot to the StreamsConsumer
approach we've discussed a bit before. The real issue with constructing and
returning a different KafkaConsumer instead of wrapping the one passed in,
is that Streams will be invoking some methods on the user's Consumer while
invoking other methods on the original KafkaConsumer. Another reason that
splitting up the consumer into a public interface and a hidden internal one
seems to complicate things for no clear reason. If every consumer method
call is being made directly on the Consumer returned from #getMainConsumer,
it doesn't really matter if they constructed a different one. In fact we
could probably just skip having Streams construct its own consumer and
instead let the user construct it or provide an alternative implementation
instead. As I understand it, the only reason we want to create the
underlying consumer within Streams is to access an internal constructor
that puts it into "Streams" mode. But that could be solved in a different
way, for example if/when Streams invokes a Streams-specific method like
#setProcessId this automatically puts the consumer into Streams mode. Not
sure if this works since I don't have the full picture with details on this
internal constructor, just feels like that doesn't have to be the only
possible way to do things (not necessarily better ways, just different)

On a somewhat related note, I had a question about the deprecation plan. As
I understand it, the stated goal of KIP-1071 is to eventually deprecate and
then remove  support for the current rebalancing protocol, but the timeline
is undefined and nothing is being deprecated right now. If this KIP is
going to immediately deprecate the old KafkaClientSupplier, then we need to
support implementing an interceptor while configured to use the current
rebalancing protocol. Is this reading of the KIP correct? That is, can we
assume that the KafkaConsumer passed into #getMainConsumer will only be
constructed via the internal constructor/put into "Streams group mode" if
the app is configured to use the new protocol, and otherwise it will just
pass in a regular KafkaConsumer? Just want to make sure my understanding is
correct here

On Fri, Sep 6, 2024 at 8:29 AM Alieh Saeedi 
wrote:

> Sorry for the typo:
> contracts = contradicts*
>
> On Fri, Sep 6, 2024 at 5:20 PM Alieh Saeedi  wrote:
>
> > Thanks Matthias for the KIP.
> >
> > A quick question regarding the sentence `It would be invalid to create a
> > new client instance `: What would happen if the implemented class creates
> > a new instance, or, in other words, how do we prevent it? Considering
> that
> > `Config` is going to be passed in as well (the 2nd point raised by
> Sophie),
> > Also, the new Consumer object (`mockConsumer`) you created in your
> example
> > here 
> > confused me since it contracts with the above sentence.
> >
> > Thanks,
> > Alieh
> >
> > On Fri, Sep 6, 2024 at 5:08 AM Sophie Blee-Goldman <
> sop...@responsive.dev>
> > wrote:
> >
> >> 1. Fair enough. In the end it doesn't matter that much to us, so I just
> >> figured from basic principles if this is a config then it should go in
> the
> >> StreamsConfig. Also happy to hear what others think
> >>
> >> 2. We need the client-specific configs (for example to extract the
> client
> >> ids for monitoring, resource management, etc). However, I checked our
> >