Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Hi all,

I'm working on enhancing the KTable foreign key join API to allow
extracting foreign keys from both key and value. However, I've encountered
a Java type erasure issue that affects our API design choices.

Current Situation:
- We want to allow foreign key extraction from both key and value
- We'd like to maintain backward compatibility with existing value-only
extractors
- Initial attempt was to use method overloading

The Issue:
Due to Java type erasure, we can't have both:
- Function foreignKeyExtractor
- Function, KO> foreignKeyExtractor
in method overloads as they become identical at runtime.

Options:
1. Breaking Change Approach
   - Replace value-only extractor with key-value extractor
   - Clean API but breaks backward compatibility

2. Compatibility Approach
   - Keep both capabilities through different method names (e.g.,
join/joinWithKey)
   - Maintains compatibility but less elegant API

Questions for Discussion:
1. Should we prioritize API elegance or backward compatibility?
2. If we choose compatibility, which naming convention should we use for
the new methods?
3. Should we consider deprecating the value-only extractor in a future
release?

Looking forward to your thoughts.

Best regards,
Peter

On Thu, Oct 31, 2024 at 1:08 PM Chu Cheng Li  wrote:

> Hi Everyone,
> I would like to start a discussion on KIP-1104:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1103%3A+Additional+metrics+for+cooperative+consumption
> <https://bitli.pro/2h2om_f8e74a0c>
>
> This KIP allow foreign key extraction from both key and value in KTable
> Joins, before this KIP user can only extract foreign from record's value,
> this KIP provides more flexibility on it.
>
> Regards,
> Peter Lee
>
>
>

[DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Hi Everyone,
I would like to start a discussion on KIP-1104:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1103%3A+Additional+metrics+for+cooperative+consumption


This KIP allow foreign key extraction from both key and value in KTable
Joins, before this KIP user can only extract foreign from record's value,
this KIP provides more flexibility on it.

Regards,
Peter Lee

Re: [VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

[Chu Cheng Li]

+1

On Thu, Nov 7, 2024 at 12:39 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi all,
>
> I'd like to call for a vote on KIP-1105
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
>
> Thanks,
> Kamal
>

[VOTE] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Hi all,

I'd like to call for a vote on KIP-1104
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1104%3A+Allow+Foreign+Key+Extraction+from+Both+Key+and+Value+in+KTable+Joins


Thanks,
Peter

Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Thank you, Lucas and Matthias, for your feedback and thoughts.

I'll start the voting session for this KIP shortly.

Best regards,
Peter

On Wed, Nov 6, 2024 at 6:35 PM Lucas Brutschy
 wrote:

> Hey,
>
> this makes sense to me! Thank you for the KIP.
>
> I think the change is straightforward, and I think we can just vote
> this through. The only thing we could discuss is whether we want to
> keep the old overloads. I thought about it, but I think I agree that
> it's nicer to simplify the API a bit eventually, since the new
> overload is strictly more powerful than the old one. So no complaints
> at all, just writing down my thoughts.
>
> Cheers,
> Lucas
>
> On Tue, Nov 5, 2024 at 3:01 AM Matthias J. Sax  wrote:
> >
> > Seems you did link to the wrong KIP. Should be
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1104%3A+Allow+Foreign+Key+Extraction+from+Both+Key+and+Value+in+KTable+Joins
> <https://bitli.pro/2hrKC_5e65c78f>
> >
> > :)
> >
> > Overall the KIP does make sense and I don't have any comments. Seems to
> > be quite clear.
> >
> >
> > -Matthias
> >
> > On 10/30/24 10:08 PM, Chu Cheng Li wrote:
> > > Hi Everyone,
> > > I would like to start a discussion on KIP-1104:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1103%3A+Additional+metrics+for+cooperative+consumption
> <https://bitli.pro/2hrKD_200035b1>
> > > <https://bitli.pro/2h2om_f8e74a0c <https://bitli.pro/2hrKE_d7d53872>>
> > >
> > > This KIP allow foreign key extraction from both key and value in KTable
> > > Joins, before this KIP user can only extract foreign from record's
> value,
> > > this KIP provides more flexibility on it.
> > >
> > > Regards,
> > > Peter Lee
> > >
>

Re: [DISCUSS] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Thanks guys !
BiFunction seems promising!

KIP updated!
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1104%3A+Allow+Foreign+Key+Extraction+from+Both+Key+and+Value+in+KTable+Joins
<https://bitli.pro/2i7Ja_1c6b6a68>

On Wed, Nov 13, 2024 at 8:30 AM Matthias J. Sax  wrote:

> I can just second was Lucas and Bill said already.
>
> 1. We cannot break compatibility
> 2. BiFunction sounds like a good alternative
> 3. I would personally deprecate the existing method, but don't feel
> strong about it.
>
>
> -Matthias
>
>
> On 11/12/24 8:33 AM, Bill Bejeck wrote:
> > Hi Peter,
> >
> > It's important that we don't break compatibility.
> > We faced a similar situation in KIP-149
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner
> >
> > when we
> > provided access to the key in mapping and joining. I think it's worth
> > exploring Lucas's suggestion of using a `BiFunction`.
> > But if for some reason it won't work (I don't see why it wouldn't) we'll
> > need to maintain compatibly and go with the renaming the new method.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Nov 12, 2024 at 8:06 AM Lucas Brutschy
> >  wrote:
> >
> >> Hi,
> >>
> >> 1. I don't think we can/should break backwards compatibility.
> >> 2. Have you considered using `BiFunction foreignKeyExtractor`
> >> ? This should work without renaming the method.
> >> 3. I don't see the benefit of deprecating it. I agree, we wouldn't add
> >> both overloads normally, but the value-only overload is useful /
> >> correct as it is, I wouldn't break users code just to "clean" things
> >> very slighly.
> >>
> >> Cheers,
> >> Lucas
> >>
> >> On Mon, Nov 11, 2024 at 6:43 PM Chu Cheng Li 
> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I'm working on enhancing the KTable foreign key join API to allow
> >>> extracting foreign keys from both key and value. However, I've
> >> encountered
> >>> a Java type erasure issue that affects our API design choices.
> >>>
> >>> Current Situation:
> >>> - We want to allow foreign key extraction from both key and value
> >>> - We'd like to maintain backward compatibility with existing value-only
> >>> extractors
> >>> - Initial attempt was to use method overloading
> >>>
> >>> The Issue:
> >>> Due to Java type erasure, we can't have both:
> >>> - Function foreignKeyExtractor
> >>> - Function, KO> foreignKeyExtractor
> >>> in method overloads as they become identical at runtime.
> >>>
> >>> Options:
> >>> 1. Breaking Change Approach
> >>> - Replace value-only extractor with key-value extractor
> >>> - Clean API but breaks backward compatibility
> >>>
> >>> 2. Compatibility Approach
> >>> - Keep both capabilities through different method names (e.g.,
> >>> join/joinWithKey)
> >>> - Maintains compatibility but less elegant API
> >>>
> >>> Questions for Discussion:
> >>> 1. Should we prioritize API elegance or backward compatibility?
> >>> 2. If we choose compatibility, which naming convention should we use
> for
> >>> the new methods?
> >>> 3. Should we consider deprecating the value-only extractor in a future
> >>> release?
> >>>
> >>> Looking forward to your thoughts.
> >>>
> >>> Best regards,
> >>> Peter
> >>>
> >>> On Thu, Oct 31, 2024 at 1:08 PM Chu Cheng Li 
> >> wrote:
> >>>
> >>>> Hi Everyone,
> >>>> I would like to start a discussion on KIP-1104:
> >>>>
> >>>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1103%3A+Additional+metrics+for+cooperative+consumption
> >>>> <https://bitli.pro/2h2om_f8e74a0c>
> >>>>
> >>>> This KIP allow foreign key extraction from both key and value in
> KTable
> >>>> Joins, before this KIP user can only extract foreign from record's
> >> value,
> >>>> this KIP provides more flexibility on it.
> >>>>
> >>>> Regards,
> >>>> Peter Lee
> >>>>
> >>>>
> >>>>
> >>
> >
>

Re: [VOTE] KIP-1104: Allow Foreign Key Extraction from Both Key and Value in KTable Joins

[Chu Cheng Li]

Thanks everyone for the votes and your time reviewing this KIP.

The vote passed with 3 binding votes and 0 non-binding vote.

Best,
Peter

On Sat, Nov 9, 2024 at 2:10 AM Bill Bejeck  wrote:

> Hi,
>
> Thanks for the KIP, this will be a useful addition.
>
> +1(binding)
>
> Thanks,
> Bill
>
>
> On Thu, Nov 7, 2024 at 3:47 PM Matthias J. Sax  wrote:
>
> > +1 (binding)
> >
> > On 11/7/24 3:51 AM, Lucas Brutschy wrote:
> > > Hi,
> > >
> > > thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Cheers,
> > > Lucas
> > >
> > > On Thu, Nov 7, 2024 at 7:31 AM Chu Cheng Li 
> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> I'd like to call for a vote on KIP-1104
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1104%3A+Allow+Foreign+Key+Extraction+from+Both+Key+and+Value+in+KTable+Joins
> > >> <https://bitli.pro/2hrUf_957ee740>
> > >>
> > >> Thanks,
> > >> Peter
> >
>