[DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams
Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
Hi Both, Thanks. I added remarks to account for this. https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks In short, let's add a note in the Java docs? The exact wording of the note can be scrutinized in the pull request? What do you think? On Sun, 6 Aug 2023 at 19:41, Guozhang Wang wrote: > I'm just thinking we can try to encourage users to migrate from XX to > XXWithKey in the docs, giving this as one good example that the latter > can help you distinguish different scenarios whereas the former > cannot. > > On Fri, Aug 4, 2023 at 6:32 PM Matthias J. Sax wrote: > > > > Guozhang, > > > > thanks for pointing out ValueJoinerWithKey. In the end, it's just a > > documentation change, ie, point out that the passed in key could be > > `null` and similar? > > > > -Matthias > > > > > > On 8/2/23 3:20 PM, Guozhang Wang wrote: > > > Thanks Florin for the writeup, > > > > > > One quick thing I'd like to bring up is that in KIP-149 > > > ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner > ) > > > we introduced ValueJoinerWithKey which is aimed to enhance > > > ValueJoiner. It would have a benefit for this KIP such that > > > implementers can distinguish "null-key" v.s. "not-null-key but > > > null-value" scenarios. > > > > > > Hence I'd suggest we also include the semantic changes with > > > ValueJoinerWithKey, which can help distinguish these two scenarios, > > > and also document that if users apply ValueJoiner only, they may not > > > have this benefit, and hence we suggest users to use the former. > > > > > > > > > Guozhang > > > > > > On Mon, Jul 31, 2023 at 12:11 PM Florin Akermann > > > wrote: > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams >
Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
Hi Lucas, Thanks. I added the point about the upgrade guide as well. Florin On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy wrote: > Hi Florin, > > thanks for the KIP! This looks good to me. I agree that the precise > Java doc wording doesn't have to be discussed as part of the KIP. > > I would also suggest to include an update to > https://kafka.apache.org/documentation/streams/upgrade-guide > > Cheers, > Lucas > > On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann > wrote: > > > > Hi Both, > > > > Thanks. > > I added remarks to account for this. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks > > > > In short, let's add a note in the Java docs? The exact wording of the > note > > can be scrutinized in the pull request? > > > > What do you think? > > > > > > On Sun, 6 Aug 2023 at 19:41, Guozhang Wang > > wrote: > > > > > I'm just thinking we can try to encourage users to migrate from XX to > > > XXWithKey in the docs, giving this as one good example that the latter > > > can help you distinguish different scenarios whereas the former > > > cannot. > > > > > > On Fri, Aug 4, 2023 at 6:32 PM Matthias J. Sax > wrote: > > > > > > > > Guozhang, > > > > > > > > thanks for pointing out ValueJoinerWithKey. In the end, it's just a > > > > documentation change, ie, point out that the passed in key could be > > > > `null` and similar? > > > > > > > > -Matthias > > > > > > > > > > > > On 8/2/23 3:20 PM, Guozhang Wang wrote: > > > > > Thanks Florin for the writeup, > > > > > > > > > > One quick thing I'd like to bring up is that in KIP-149 > > > > > ( > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner > > > ) > > > > > we introduced ValueJoinerWithKey which is aimed to enhance > > > > > ValueJoiner. It would have a benefit for this KIP such that > > > > > implementers can distinguish "null-key" v.s. "not-null-key but > > > > > null-value" scenarios. > > > > > > > > > > Hence I'd suggest we also include the semantic changes with > > > > > ValueJoinerWithKey, which can help distinguish these two scenarios, > > > > > and also document that if users apply ValueJoiner only, they may > not > > > > > have this benefit, and hence we suggest users to use the former. > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > On Mon, Jul 31, 2023 at 12:11 PM Florin Akermann > > > > > wrote: > > > > >> > > > > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams > > > >
Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
Hi All, I added a remark about the repartition of null-key records. https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords Can we relax this constraint for any kind of repartitioning or should it only be relaxed in the context of left stream-table and left/outer stream-stream joins? Florin On Mon, 7 Aug 2023 at 13:23, Florin Akermann wrote: > Hi Lucas, > > Thanks. I added the point about the upgrade guide as well. > > Florin > > On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy > wrote: > >> Hi Florin, >> >> thanks for the KIP! This looks good to me. I agree that the precise >> Java doc wording doesn't have to be discussed as part of the KIP. >> >> I would also suggest to include an update to >> https://kafka.apache.org/documentation/streams/upgrade-guide >> >> Cheers, >> Lucas >> >> On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann >> wrote: >> > >> > Hi Both, >> > >> > Thanks. >> > I added remarks to account for this. >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks >> > >> > In short, let's add a note in the Java docs? The exact wording of the >> note >> > can be scrutinized in the pull request? >> > >> > What do you think? >> > >> > >> > On Sun, 6 Aug 2023 at 19:41, Guozhang Wang >> > wrote: >> > >> > > I'm just thinking we can try to encourage users to migrate from XX to >> > > XXWithKey in the docs, giving this as one good example that the latter >> > > can help you distinguish different scenarios whereas the former >> > > cannot. >> > > >> > > On Fri, Aug 4, 2023 at 6:32 PM Matthias J. Sax >> wrote: >> > > > >> > > > Guozhang, >> > > > >> > > > thanks for pointing out ValueJoinerWithKey. In the end, it's just a >> > > > documentation change, ie, point out that the passed in key could be >> > > > `null` and similar? >> > > > >> > > > -Matthias >> > > > >> > > > >> > > > On 8/2/23 3:20 PM, Guozhang Wang wrote: >> > > > > Thanks Florin for the writeup, >> > > > > >> > > > > One quick thing I'd like to bring up is that in KIP-149 >> > > > > ( >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner >> > > ) >> > > > > we introduced ValueJoinerWithKey which is aimed to enhance >> > > > > ValueJoiner. It would have a benefit for this KIP such that >> > > > > implementers can distinguish "null-key" v.s. "not-null-key but >> > > > > null-value" scenarios. >> > > > > >> > > > > Hence I'd suggest we also include the semantic changes with >> > > > > ValueJoinerWithKey, which can help distinguish these two >> scenarios, >> > > > > and also document that if users apply ValueJoiner only, they may >> not >> > > > > have this benefit, and hence we suggest users to use the former. >> > > > > >> > > > > >> > > > > Guozhang >> > > > > >> > > > > On Mon, Jul 31, 2023 at 12:11 PM Florin Akermann >> > > > > wrote: >> > > > >> >> > > > >> >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams >> > > >> >
Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
Thank you for the feedback. > Not sure if this is the right phrasing? You are right. I adjusted the phrasing accordingly. Given your description of the current behavior, do I understand correctly that the current documentation for the left join KStream-GlobalKtable is out of date? https://github.com/apache/kafka/blob/9318b591d7a57b9db1e7519986d78f0402cd5b5e/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java#L2948C7-L2948C7 > I would remove this from the KIP I agree,Removed. Plus, relevant doc links added. > I think the way it's phrased is good. [...] We can cover details on the PR. Great. Yes, in general, I hope everybody agrees that we shouldn't add more details to this KIP On Thu, 10 Aug 2023 at 00:16, Matthias J. Sax wrote: > Thanks for the KIP. > > > left join KStream-GlobalTable: no longer drop left records with null-key > and call KeyValueMapper with 'null' for left key. The case where > KeyValueMapper returns null is already handled in the current > implementation. > > Not sure if this is the right phrasing? In the end, even now, the stream > input record key can be null (cf > https://issues.apache.org/jira/browse/KAFKA-10277) -- a stream record is > only dropped if the `KeyValueMapper` returns `null` (note that the > key-extractor has no default implemenation but is a required argument) > -- this KIP would relax this case for left-join. > > > > In the pull request all relevant Javadocs will be updated with the > information on how to keep the old behavior for a given operator / method. > > I would remove this from the KIP -- I am also not sure if we should put > it into the JavaDoc? -- I agree that it should go into the upgrade docs > as well as "join section" in the docs: > > https://kafka.apache.org/35/documentation/streams/developer-guide/dsl-api.html#joining > > We also have > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics > that we should update. > > > > I added a remark about the repartition of null-key records. > > I think the way it's phrase is good. In the end, it's an optimization to > drop records upstream (instead of piping them through the topic and drop > the downstream), and thus we don't have to cover it in the KIP in > detail. In general, for aggregations we can still apply the > optimization, however, we need to be careful as we could also have two > downstream operators with a shared repartition topic: for this case, we > can only drop upstream if all downstream operator would drop null-key > records anyway. We can cover details on the PR. > > > > -Matthias > > > > On 8/9/23 5:39 AM, Florin Akermann wrote: > > Hi All, > > > > I added a remark about the repartition of null-key records. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords > > > > Can we relax this constraint for any kind of repartitioning or should it > > only be relaxed in the context of left stream-table and left/outer > > stream-stream joins? > > > > Florin > > > > On Mon, 7 Aug 2023 at 13:23, Florin Akermann > > wrote: > > > >> Hi Lucas, > >> > >> Thanks. I added the point about the upgrade guide as well. > >> > >> Florin > >> > >> On Mon, 7 Aug 2023 at 11:06, Lucas Brutschy .invalid> > >> wrote: > >> > >>> Hi Florin, > >>> > >>> thanks for the KIP! This looks good to me. I agree that the precise > >>> Java doc wording doesn't have to be discussed as part of the KIP. > >>> > >>> I would also suggest to include an update to > >>> https://kafka.apache.org/documentation/streams/upgrade-guide > >>> > >>> Cheers, > >>> Lucas > >>> > >>> On Mon, Aug 7, 2023 at 10:51 AM Florin Akermann > >>> wrote: > >>>> > >>>> Hi Both, > >>>> > >>>> Thanks. > >>>> I added remarks to account for this. > >>>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Remarks > >>>> > >>>> In short, let's add a note in the Java docs? The exact wording of the > >>> note > >>>> can be scrutinized in the pull request? > >>>> > >>>> What do you think? > >>>> > >>>> > >>&g
Re: [DISCUSS] KIP-962 Relax non-null key requirement in Kafka Streams
Thanks. Here is the PR for this catch: https://github.com/apache/kafka/pull/14187 On Thu, 10 Aug 2023 at 19:04, Matthias J. Sax wrote: > Good catch about the JavaDocs. > > Seems we missed to update them when we did K10277. Would you like to do > a PR to fix them right away for upcoming 3.6 release? > > If there is no more other comments, I think you can start a VOTE thread. > > > -Matthias > > On 8/10/23 4:22 AM, Florin Akermann wrote: > > Thank you for the feedback. > > > >> Not sure if this is the right phrasing? > > > > You are right. I adjusted the phrasing accordingly. > > Given your description of the current behavior, do I understand correctly > > that the current documentation for the left join KStream-GlobalKtable is > > out of date? > > > https://github.com/apache/kafka/blob/9318b591d7a57b9db1e7519986d78f0402cd5b5e/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java#L2948C7-L2948C7 > > > >> I would remove this from the KIP > > > > I agree,Removed. > > Plus, relevant doc links added. > > > >> I think the way it's phrased is good. [...] We can cover details on the > > PR. > > > > Great. Yes, in general, I hope everybody agrees that we shouldn't add > more > > details to this KIP > > > > > > On Thu, 10 Aug 2023 at 00:16, Matthias J. Sax wrote: > > > >> Thanks for the KIP. > >> > >>> left join KStream-GlobalTable: no longer drop left records with > null-key > >> and call KeyValueMapper with 'null' for left key. The case where > >> KeyValueMapper returns null is already handled in the current > >> implementation. > >> > >> Not sure if this is the right phrasing? In the end, even now, the stream > >> input record key can be null (cf > >> https://issues.apache.org/jira/browse/KAFKA-10277) -- a stream record > is > >> only dropped if the `KeyValueMapper` returns `null` (note that the > >> key-extractor has no default implemenation but is a required argument) > >> -- this KIP would relax this case for left-join. > >> > >> > >>> In the pull request all relevant Javadocs will be updated with the > >> information on how to keep the old behavior for a given operator / > method. > >> > >> I would remove this from the KIP -- I am also not sure if we should put > >> it into the JavaDoc? -- I agree that it should go into the upgrade docs > >> as well as "join section" in the docs: > >> > >> > https://kafka.apache.org/35/documentation/streams/developer-guide/dsl-api.html#joining > >> > >> We also have > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics > >> that we should update. > >> > >> > >>> I added a remark about the repartition of null-key records. > >> > >> I think the way it's phrase is good. In the end, it's an optimization to > >> drop records upstream (instead of piping them through the topic and drop > >> the downstream), and thus we don't have to cover it in the KIP in > >> detail. In general, for aggregations we can still apply the > >> optimization, however, we need to be careful as we could also have two > >> downstream operators with a shared repartition topic: for this case, we > >> can only drop upstream if all downstream operator would drop null-key > >> records anyway. We can cover details on the PR. > >> > >> > >> > >> -Matthias > >> > >> > >> > >> On 8/9/23 5:39 AM, Florin Akermann wrote: > >>> Hi All, > >>> > >>> I added a remark about the repartition of null-key records. > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams#KIP962:RelaxnonnullkeyrequirementinKafkaStreams-Repartitionofnull-keyrecords > >>> > >>> Can we relax this constraint for any kind of repartitioning or should > it > >>> only be relaxed in the context of left stream-table and left/outer > >>> stream-stream joins? > >>> > >>> Florin > >>> > >>> On Mon, 7 Aug 2023 at 13:23, Florin Akermann < > florin.akerm...@gmail.com> > >>> wrote: > >>> > >>>> Hi Lucas, > >>>> > >>>> Thanks. I added the point about the upgrade guide as well. > >>>> > >>>> Flo
[VOTE] KIP-962 Relax non-null key requirement in Kafka Streams
https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams
Re: [VOTE] KIP-962 Relax non-null key requirement in Kafka Streams
Thanks all. The 72h window is through. The vote on KIP-962 passes with: 2 binding +1 1 non-binding +1 no vetoes Florin On Fri, 11 Aug 2023 at 15:43, Bill Bejeck wrote: > +1(binding) > > On Fri, Aug 11, 2023 at 7:33 AM Lucas Brutschy > wrote: > > > +1 (non-binding) > > > > On Fri, Aug 11, 2023 at 1:08 AM Matthias J. Sax > wrote: > > > > > > +1 (binding) > > > > > > On 8/10/23 12:31 PM, Florin Akermann wrote: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams > > > > > > >
KIP-962: Review of second Jira/PR before 3.7 release
Hi, As part of KIP-962 https://github.com/apache/kafka/pull/14174 got merged. I think it would be good to tackle the related jira in the same release (3.7) https://issues.apache.org/jira/browse/KAFKA-14748 https://github.com/apache/kafka/pull/14107 Thanks, Florin
Re: [DISCUSS] KIP-837 Allow MultiCasting a Result Record.
Hi Sagar, Thanks for the KIP. I am wondering about the following. What about other classes than the org.apache.kafka.streams.processor.internals.RecordCollectorImpl. Would they still call .partition(...) and just crash? Or is it a given that they never receive a reference to a Partitioner of type MultiCastStreamPartitioner? Florin On Sat, 28 May 2022 at 05:44, Sagar wrote: > Hi All, > > I’m thinking to move this KIP to vote section if there aren’t any > objections. > > Plz let me know if I that’s ok. > > Thanks! > Sagar. > > On Tue, 24 May 2022 at 10:32 PM, Sagar wrote: > > > Hi All, > > > > Bumping this discussion thread again to see if there are any > > comments/concerns on this. > > > > Thanks! > > Sagar. > > > > On Wed, May 18, 2022 at 11:44 PM Sagar > wrote: > > > >> Hi All, > >> > >> I would like to open a discussion thread on > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883356 > >> . > >> > >> Thanks! > >> Sagar. > >> > > >
Contributer List
Hi, I would like to become a contributor. My Jira account username is 'aki' Thank you in advance. Florin
[DISCUSS] KIP-798 Add possibility to write kafka headers in Kafka Console Producer
https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer
Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer
Hi John, Thanks for the vote and feedback. The thought occurred to me too. Do I understand it correctly: the current version of the kafka-console-producer cannot be used for anything other than UTF-8 keys and values? (There is no other implementation of MessageReader other than the ConsoleProducer$LineMessageReader) In other words, currently users seem to only apply it with utf-8 strings for keys and values? This is why I figured I would not deviate from this assumption solely for the headers. I will happily raise another KIP / Jira if there is a need to specify other formats / serializers for headers, keys and/or values. Thanks, Florin On Sat, 20 Nov 2021 at 19:34, John Roesler wrote: > Hi Florin, > > Thanks for the KIP! > > I think the assumption that header values are UTF-8 strings might not hold > up in the long run, but it seems like we can easily add a property later to > specify the format. It seems like this scope is probably a handy addition > on its own. > > I’m +1 (binding) > > Thanks, > John > > > On Fri, Nov 19, 2021, at 15:06, flo wrote: > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer > > >
Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer
Hi Luke and Tom @Tom: Thanks for the vote. @Luke: Thanks for the feedback. I have updated the KIP accordingly with regards to your comments on the remaining case (false,false) and the motivation. Regarding the "not only UTF-8": As far as I understand John it is fine to limit the scope for this change to UTF-8 only as it is a handy addition on its own. Other formats can be relatively easily supported by adding more properties in later KIPs. In my reply to John (email from 21 Nov 2021, 11:29 UTC) I also added an explanation why I limited the scope to UTF-8 only. Thanks, Florin On Mon, 22 Nov 2021 at 10:32, Tom Bentley wrote: > Hi Florin, > > Thanks for the KIP! > > +1 (binding), > > Kind regards, > > Tom > > On Mon, Nov 22, 2021 at 6:51 AM Luke Chen wrote: > > > Hi Florin, > > Thanks for the KIP. > > > > This KIP makes sense to me. Just a comment that the motivation section is > > not clearly explain why this KIP is important. > > I think John already mentioned a good motivation, which is to support > "not > > only UTF-8". > > You should put that into the KIP, and of course if you have other > thoughts, > > please also add them into KIP. > > > > Also, in the "public interface" section, there are 3 "Default parsing > > pattern", I think you should add 1 remaining case (false, false) to make > it > > complete. > > > > Otherwise, look good to me. > > > > Thank you. > > Luke > > > > > > On Sun, Nov 21, 2021 at 7:37 PM Florin Akermann < > florin.akerm...@gmail.com > > > > > wrote: > > > > > Hi John, > > > > > > Thanks for the vote and feedback. > > > > > > The thought occurred to me too. > > > > > > Do I understand it correctly: the current version of the > > > kafka-console-producer cannot be used for anything other than UTF-8 > keys > > > and values? > > > (There is no other implementation of MessageReader other than the > > > ConsoleProducer$LineMessageReader) > > > In other words, currently users seem to only apply it with utf-8 > strings > > > for keys and values? > > > This is why I figured I would not deviate from this assumption solely > for > > > the headers. > > > > > > I will happily raise another KIP / Jira if there is a need to specify > > other > > > formats / serializers for headers, keys and/or values. > > > > > > Thanks, > > > Florin > > > > > > > > > On Sat, 20 Nov 2021 at 19:34, John Roesler > wrote: > > > > > > > Hi Florin, > > > > > > > > Thanks for the KIP! > > > > > > > > I think the assumption that header values are UTF-8 strings might not > > > hold > > > > up in the long run, but it seems like we can easily add a property > > later > > > to > > > > specify the format. It seems like this scope is probably a handy > > addition > > > > on its own. > > > > > > > > I’m +1 (binding) > > > > > > > > Thanks, > > > > John > > > > > > > > > > > > On Fri, Nov 19, 2021, at 15:06, flo wrote: > > > > > < > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer > > > > > > > > > > > > > > >
Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer
Hi Bill and David, Thank you both for the vote. @David: KIP is updated. Florin On Mon, 22 Nov 2021 at 18:28, David Jacot wrote: > Hi Florin, > > Thanks for the KIP. I am +1 (binding). > > There is a small typo in the Proposed Changes section: > `parse.header` should be `parse.headers`. > > Best, > David > > On Mon, Nov 22, 2021 at 6:20 PM Bill Bejeck wrote: > > > > Hi Florin, > > > > Thanks for the KIP, this seems like a very useful addition. > > > > +1(binding). > > > > -Bill > > > > On Mon, Nov 22, 2021 at 12:00 PM Florin Akermann < > florin.akerm...@gmail.com> > > wrote: > > > > > Hi Luke and Tom > > > > > > @Tom: Thanks for the vote. > > > > > > @Luke: Thanks for the feedback. > > > > > > I have updated the KIP accordingly with regards to your comments on the > > > remaining case (false,false) and the motivation. > > > > > > Regarding the "not only UTF-8": As far as I understand John it is fine > to > > > limit the scope for this change to UTF-8 only as it is a handy > addition on > > > its own. Other formats can be relatively easily supported by adding > more > > > properties in later KIPs. In my reply to John (email from 21 Nov 2021, > > > 11:29 UTC) I also added an explanation why I limited the scope to UTF-8 > > > only. > > > > > > Thanks, > > > Florin > > > > > > > > > > > > On Mon, 22 Nov 2021 at 10:32, Tom Bentley wrote: > > > > > > > Hi Florin, > > > > > > > > Thanks for the KIP! > > > > > > > > +1 (binding), > > > > > > > > Kind regards, > > > > > > > > Tom > > > > > > > > On Mon, Nov 22, 2021 at 6:51 AM Luke Chen wrote: > > > > > > > > > Hi Florin, > > > > > Thanks for the KIP. > > > > > > > > > > This KIP makes sense to me. Just a comment that the motivation > section > > > is > > > > > not clearly explain why this KIP is important. > > > > > I think John already mentioned a good motivation, which is to > support > > > > "not > > > > > only UTF-8". > > > > > You should put that into the KIP, and of course if you have other > > > > thoughts, > > > > > please also add them into KIP. > > > > > > > > > > Also, in the "public interface" section, there are 3 "Default > parsing > > > > > pattern", I think you should add 1 remaining case (false, false) to > > > make > > > > it > > > > > complete. > > > > > > > > > > Otherwise, look good to me. > > > > > > > > > > Thank you. > > > > > Luke > > > > > > > > > > > > > > > On Sun, Nov 21, 2021 at 7:37 PM Florin Akermann < > > > > florin.akerm...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Hi John, > > > > > > > > > > > > Thanks for the vote and feedback. > > > > > > > > > > > > The thought occurred to me too. > > > > > > > > > > > > Do I understand it correctly: the current version of the > > > > > > kafka-console-producer cannot be used for anything other than > UTF-8 > > > > keys > > > > > > and values? > > > > > > (There is no other implementation of MessageReader other than the > > > > > > ConsoleProducer$LineMessageReader) > > > > > > In other words, currently users seem to only apply it with utf-8 > > > > strings > > > > > > for keys and values? > > > > > > This is why I figured I would not deviate from this assumption > solely > > > > for > > > > > > the headers. > > > > > > > > > > > > I will happily raise another KIP / Jira if there is a need to > specify > > > > > other > > > > > > formats / serializers for headers, keys and/or values. > > > > > > > > > > > > Thanks, > > > > > > Florin > > > > > > > > > > > > > > > > > > On Sat, 20 Nov 2021 at 19:34, John Roesler > > > > wrote: > > > > > > > > > > > > > Hi Florin, > > > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > > > I think the assumption that header values are UTF-8 strings > might > > > not > > > > > > hold > > > > > > > up in the long run, but it seems like we can easily add a > property > > > > > later > > > > > > to > > > > > > > specify the format. It seems like this scope is probably a > handy > > > > > addition > > > > > > > on its own. > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > Thanks, > > > > > > > John > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 19, 2021, at 15:06, flo wrote: > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer
Thanks all. The 72h window is through. @Comitters can this vote be concluded? The vote on KIP-798 would pass with: 4 binding +1 1 non-binding +1 no vetoes Thanks, Florin On Tue, 23 Nov 2021 at 06:59, Luke Chen wrote: > Hi Florin, > Thanks for the update! > > +1 (non-binding) > > Thank you. > Luke > > On Tue, Nov 23, 2021 at 2:00 AM Florin Akermann > > wrote: > > > Hi Bill and David, > > > > Thank you both for the vote. > > @David: KIP is updated. > > > > Florin > > > > On Mon, 22 Nov 2021 at 18:28, David Jacot > > wrote: > > > > > Hi Florin, > > > > > > Thanks for the KIP. I am +1 (binding). > > > > > > There is a small typo in the Proposed Changes section: > > > `parse.header` should be `parse.headers`. > > > > > > Best, > > > David > > > > > > On Mon, Nov 22, 2021 at 6:20 PM Bill Bejeck wrote: > > > > > > > > Hi Florin, > > > > > > > > Thanks for the KIP, this seems like a very useful addition. > > > > > > > > +1(binding). > > > > > > > > -Bill > > > > > > > > On Mon, Nov 22, 2021 at 12:00 PM Florin Akermann < > > > florin.akerm...@gmail.com> > > > > wrote: > > > > > > > > > Hi Luke and Tom > > > > > > > > > > @Tom: Thanks for the vote. > > > > > > > > > > @Luke: Thanks for the feedback. > > > > > > > > > > I have updated the KIP accordingly with regards to your comments on > > the > > > > > remaining case (false,false) and the motivation. > > > > > > > > > > Regarding the "not only UTF-8": As far as I understand John it is > > fine > > > to > > > > > limit the scope for this change to UTF-8 only as it is a handy > > > addition on > > > > > its own. Other formats can be relatively easily supported by adding > > > more > > > > > properties in later KIPs. In my reply to John (email from 21 Nov > > 2021, > > > > > 11:29 UTC) I also added an explanation why I limited the scope to > > UTF-8 > > > > > only. > > > > > > > > > > Thanks, > > > > > Florin > > > > > > > > > > > > > > > > > > > > On Mon, 22 Nov 2021 at 10:32, Tom Bentley > > wrote: > > > > > > > > > > > Hi Florin, > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > +1 (binding), > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > Tom > > > > > > > > > > > > On Mon, Nov 22, 2021 at 6:51 AM Luke Chen > > wrote: > > > > > > > > > > > > > Hi Florin, > > > > > > > Thanks for the KIP. > > > > > > > > > > > > > > This KIP makes sense to me. Just a comment that the motivation > > > section > > > > > is > > > > > > > not clearly explain why this KIP is important. > > > > > > > I think John already mentioned a good motivation, which is to > > > support > > > > > > "not > > > > > > > only UTF-8". > > > > > > > You should put that into the KIP, and of course if you have > other > > > > > > thoughts, > > > > > > > please also add them into KIP. > > > > > > > > > > > > > > Also, in the "public interface" section, there are 3 "Default > > > parsing > > > > > > > pattern", I think you should add 1 remaining case (false, > false) > > to > > > > > make > > > > > > it > > > > > > > complete. > > > > > > > > > > > > > > Otherwise, look good to me. > > > > > > > > > > > > > > Thank you. > > > > > > > Luke > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 21, 2021 at 7:37 PM Florin Akermann < > > > > > > florin.akerm...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > >
[jira] [Resolved] (KAFKA-14748) Relax non-null FK left-join requirement
[ https://issues.apache.org/jira/browse/KAFKA-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florin Akermann resolved KAFKA-14748. - Resolution: Fixed > Relax non-null FK left-join requirement > --- > > Key: KAFKA-14748 > URL: https://issues.apache.org/jira/browse/KAFKA-14748 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax > Assignee: Florin Akermann >Priority: Major > Labels: kip > > Kafka Streams enforces a strict non-null-key policy in the DSL across all > key-dependent operations (like aggregations and joins). > This also applies to FK-joins, in particular to the ForeignKeyExtractor. If > it returns `null`, it's treated as invalid. For left-joins, it might make > sense to still accept a `null`, and add the left-hand record with an empty > right-hand-side to the result. > KIP-962: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12317) Relax non-null key requirement for left/outer KStream joins
[
https://issues.apache.org/jira/browse/KAFKA-12317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Florin Akermann resolved KAFKA-12317.
-
Resolution: Fixed
> Relax non-null key requirement for left/outer KStream joins
> ---
>
> Key: KAFKA-12317
> URL: https://issues.apache.org/jira/browse/KAFKA-12317
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Matthias J. Sax
> Assignee: Florin Akermann
>Priority: Major
> Labels: kip
>
> Currently, for a stream-streams and stream-table/globalTable join
> KafkaStreams drops all stream records with a `null`{-}key (`null`-join-key
> for stream-globalTable), because for a `null`{-}(join)key the join is
> undefined: ie, we don't have an attribute the do the table lookup (we
> consider the stream-record as malformed). Note, that we define the semantics
> of _left/outer_ join as: keep the stream record if no matching join record
> was found.
> We could relax the definition of _left_ stream-table/globalTable and
> _left/outer_ stream-stream join though, and not drop `null`-(join)key stream
> records, and call the ValueJoiner with a `null` "other-side" value instead:
> if the stream record key (or join-key) is `null`, we could treat is as
> "failed lookup" instead of treating the stream record as corrupted.
> If we make this change, users that want to keep the current behavior, can add
> a `filter()` before the join to drop `null`-(join)key records from the stream
> explicitly.
> Note that this change also requires to change the behavior if we insert a
> repartition topic before the join: currently, we drop `null`-key record
> before writing into the repartition topic (as we know they would be dropped
> later anyway). We need to relax this behavior for a left stream-table and
> left/outer stream-stream join. User need to be aware (ie, we might need to
> put this into the docs and JavaDocs), that records with `null`-key would be
> partitioned randomly.
> KIP-962:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-962%3A+Relax+non-null+key+requirement+in+Kafka+Streams]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16123) KStreamKStreamJoinProcessor forwards null records for left/outer joins unconditionally of the join window.
Florin Akermann created KAFKA-16123: --- Summary: KStreamKStreamJoinProcessor forwards null records for left/outer joins unconditionally of the join window. Key: KAFKA-16123 URL: https://issues.apache.org/jira/browse/KAFKA-16123 Project: Kafka Issue Type: Bug Reporter: Florin Akermann Assignee: Florin Akermann As part of KIP-962 the non-null key requirements have been relaxed for left and outer joins. However, the implementation forwards null records for left/outer joins unconditionally of the join window. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14049) Relax Non Null Requirement for KStreamGlobalKTable Left Join
[ https://issues.apache.org/jira/browse/KAFKA-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florin Akermann resolved KAFKA-14049. - Resolution: Fixed > Relax Non Null Requirement for KStreamGlobalKTable Left Join > > > Key: KAFKA-14049 > URL: https://issues.apache.org/jira/browse/KAFKA-14049 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Saumya Gupta > Assignee: Florin Akermann >Priority: Major > > Null Values in the Stream for a Left Join would indicate a Tombstone Message > that needs to propagated if not actually joined with the GlobalKTable > message, hence these messages should not be ignored . -- This message was sent by Atlassian Jira (v8.20.10#820010)
