RE: Re: [DISCUSS] KIP-960: Support interactive queries (IQv2) for versioned state stores

Hey Alieh, Thanks for the KIP! It looks like the KIP proposes three different types of interactive queries for versioned stores, though they are grouped together into two classes: VersionedKeyQuery adds supports for single-key, single-timestamp lookups, and also for single-key, multi-timestam

Re: [DISCUSS] KIP-960: Support interactive queries (IQv2) for versioned state stores

; KIPs for range queries. There is no need to do everything with a single > >> KIP. > >> > >> About the original v-store KIP and `long` vs `Instance` -- I don't think > >> we forget it. If the store is use inside a `Processor` using `long` is > &

RE: Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi Alieh, Thanks for the KIP! 1. As mentioned in the discussion thread for KIP-960, I think it'd be good to add a new method to the VersionedKeyValueStore interface for supporting single-key, multi-timestamp lookups, as part of implementing these new interactive queries so that the IQ implemen

[DISCUSS] KIP-889 Versioned State Stores

Hi everyone, I have a proposal for introducing versioned state stores in Kafka Streams. Versioned state stores are similar to key-value stores except they can store multiple record versions for a single key. This KIP focuses on interfaces only in order to limit the scope of the KIP. https://cwiki

Re: [DISCUSS] KIP-889 Versioned State Stores

hand, > > out-of-order writes and reads which access older segments may slow down > > if there are too many segments." -- Wondering if JavaDocs should make > > any statements about expected performance? Seems to be an implementation > > detail? > > > >

Re: [DISCUSS] KIP-889 Versioned State Stores

g to the semantics that you want > to > > support like `getAsOf` or something like that? I am not sure if we do > that > > in our codebase though. Maybe the experts can chime in. > > > > 3) hmm I would have named it `validUpto` But again not very pic

Re: [DISCUSS] KIP-889 Versioned State Stores

t. I > would prefer to keep the behavior consistent though and not change it > just for the new versioned-store, but change it globally in one shot > independent of this KIP. > > > -Matthias > > > > On 12/1/22 10:15 AM, Sagar wrote: > > Thanks Victoria, > > > &

Re: [DISCUSS] KIP-889 Versioned State Stores

at seems a small side benefit. > (Btw: your code snippet in the KIP shows what `VersionedRecord` would > have a non-null requirement for the value, but I think it would need to > allow null as value?) > > > -Matthias > > On 12/7/22 5:23 PM, Victoria Xia wrote: > &

[VOTE] KIP-889 Versioned State Stores

Hi all, I'd like to start a vote on KIP-889 for introducing versioned key-value state stores to Kafka Streams: https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores The discussion thread has been open for a few weeks now and has converged among the current participan

Re: [DISCUSS] KIP-889 Versioned State Stores

on. SGTM. > > On 12/13/22 3:06 PM, Victoria Xia wrote: > > Hi Matthias, > > > > Thanks for chiming in! Barring objections from anyone on this thread, I > > will start the vote for this KIP on Thursday. That should be enough time > to > > incorporate any lingeri

Re: [DISCUSS] KIP-889 Versioned State Stores

> The public constructor would do exactly the same, wouldn't it? > > I would make the constructor public and remove the static method make(). > > Best, > Bruno > > On 15.12.22 20:58, Victoria Xia wrote: > > Thanks again for the great discussion, Sagar, Bruno, and

Re: [VOTE] KIP-889 Versioned State Stores

20:03, Matthias J. Sax wrote: > > > +1 (binding) > > > > > > On 12/15/22 1:27 PM, John Roesler wrote: > > >> Thanks for the thorough KIP, Victoria! > > >> > > >> I'm +1 (binding) > > >> > > >> -John > &g

Re: [VOTE] KIP-889 Versioned State Stores

cuss, Victoria On Wed, Dec 21, 2022 at 8:22 AM Victoria Xia wrote: > Hi everyone, > > We have 3 binding and 1 non-binding vote in favor of this KIP (and no > objections) so KIP-889 is now accepted. > > Thanks for voting, and for your excellent comments in the KIP discussion > thr

[DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

Hi everyone, I have a proposal for updating Kafka Streams's stream-table join and table-table join semantics for the new versioned key-value state stores introduced in KIP-889 . Would love to hear your thoughts an

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

ks for the KIP, Victoria! > > > > I had some questions/concerns, but you addressed them in the Rejected > Alternatives section. Thanks for the thorough proposal! > > > > -John > > > > On Thu, Mar 9, 2023, at 18:59, Victoria Xia wrote: > > > Hi everyone, &

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

-order records, we just > "drop" them anyway, so no change for left-joins either I believe. > > > -Matthias > > > > On 3/15/23 2:00 PM, Guozhang Wang wrote: > > Sounds good to me. Thanks! > > > > On Wed, Mar 15, 2023 at 12:07 PM Victoria Xia >

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

vote at the end of the day. Happy to continue discussing even after the vote has started. Thanks for all the great suggestions so far! Best, Victoria On Thu, Mar 16, 2023 at 10:38 AM Victoria Xia wrote: > Thanks for your comments, Matthias! > > > For stream-table joins, I thin

[VOTE] KIP-914 Join Processor Semantics for Versioned Stores

Hi all, I'd like to start a vote on KIP-914 for updating the Kafka Streams join processors to use proper timestamp-based semantics in applications with versioned stores: https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+Join+Processor+Semantics+for+Versioned+Stores To avoid compatibili

Re: [DISCUSS] KIP-914 Join Processor Semantics for Versioned Stores

The voting thread for this KIP is open. Thanks, Guozhang and Matthias, for already having voted! On Fri, Mar 17, 2023 at 1:18 PM Guozhang Wang wrote: > Thanks Matthias / Victoria, both bullet points make sense to me. > > On Thu, Mar 16, 2023 at 10:39 AM Victoria Xia > wrote: > &

Re: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

ing) > > Thanks Victoria! > > Best, > Bruno > > On 20.03.23 17:13, Matthias J. Sax wrote: > > +1 (binding) > > > > On 3/20/23 9:05 AM, Guozhang Wang wrote: > >> +1, thank you Victoria! > >> > >> On Sat, Mar 18, 2023 at 8:27 AM Victoria Xia >

Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

along with the rest of KIP-914 in the 3.5 release. Apologies for not noticing these additional semantics implications earlier, Victoria -- Forwarded message - From: Victoria Xia Date: Wed, Mar 22, 2023 at 10:08 AM Subject: Re: [VOTE] KIP-914 Join Processor Semantics for Version

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

timestamp extractor, can I not > > > get it the same way from the old record that is passed down to the > > > aggregation after KIP-904? > > > > > > Thanks, > > > Lucas > > > > > > On Tue, Apr 11, 2023 at 5:35 AM Matthias J. Sax >

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

3.5 as the release deadline is fast approaching. Thanks, Victoria On Tue, Apr 11, 2023 at 2:56 PM Victoria Xia wrote: > Thanks for your comments and suggestions, Matthias, Lucas, and Guozhang! > > I was just in the process of responding when I saw Guozhang's message. I > came up

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

Cool KIP, Walker! Thanks for sharing this proposal. A few clarifications: 1. Is the order that records exit the buffer in necessarily the same as the order that records enter the buffer in, or no? Based on the description in the KIP, it sounds like the answer is no, i.e., records will exit the bu

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

er > > is really just a "linear buffer", we could easily preserve offset-order. > > But I also see a benefit of re-ordering and emitting out-of-order data > > right away when read (instead of blocking them behind in-order records > > that are not ready yet). -- It

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

t; potentially required an application re-set, or following the upgrade > >>>>>> path for versioned state stores, and also changing the program to > >>>>>> explicitly materialize using a versioned store. Also note, that we > >> might > >&

Re: [VOTE] KIP-923: Add A Grace Period to Stream Table Join

Hi Walker, Thanks for the KIP! Left a clarification question on the discussion thread just now but it's about an implementation detail, so I don't think it changes anything in this vote thread. +1 (non-binding) Cheers, Victoria On Mon, Jun 5, 2023 at 10:23 AM Bill Bejeck wrote: > Hi Walker, >

Re: [DISCUSS] KIP-923: Add A Grace Period to Stream Table Join

ecords with the same key and same timestamp. > > > > > > Just an idea! Maybe Walker comes up with something better. > > > > > > Best, > > > Bruno > > > > > > On 05.06.23 20:38, Victoria Xia wrote: > > >> Hi Walker, > > &

[DISCUSS] KIP-775: Custom partitioners in foreign key joins

Hi, I've opened a small KIP for adding Kafka Streams support for foreign key joins on tables with custom partitioners: https://cwiki.apache.org/confluence/display/KAFKA/KIP-775%3A+Custom+partitioners+in+foreign+key+joins Feedback appreciated. Thanks! - Victoria

Re: [DISCUSS] KIP-775: Custom partitioners in foreign key joins

, we would deprecate the existing methods > using `Named` and replace them with the new methods. Net benefit is, > that we don't get more overloads (after we removed the deprecated ones). > > Not sure how we want to call the new object. Maybe `TableJoined` in > alignment to `Stre

Re: [DISCUSS] KIP-775: Custom partitioners in foreign key joins

or stream-stream > >> joins). This new object could also implement `NamedOperation` and thus > >> replace `Named`. To this end, we would deprecate the existing methods > >> using `Named` and replace them with the new methods. Net benefit is, > >> that we don't ge

Re: [DISCUSS] KIP-775: Custom partitioners in foreign key joins

#x27;s currently only one situation in which we have > >>>>> to _inform_ streams about the partitioning of a KTable or > >>>>> KStream: > >>>>> 3. when we issue a key-based query via IQ, we need to know > >>>>> the partitioner, so the IQ interface allows

[VOTE] KIP-775: Custom partitioners in foreign key joins

Hi all, I'd like to start a vote for KIP-775 for adding Kafka Streams support for foreign key joins on tables with custom partitioners: https://cwiki.apache.org/confluence/display/KAFKA/KIP-775%3A+Custom+partitioners+in+foreign+key+joins Thanks, Victoria

Re: [VOTE] KIP-775: Custom partitioners in foreign key joins

(non-binding) > > > > > > Glad to see this in here :) > > > > > > On Tue, Sep 28, 2021 at 5:11 PM Bill Bejeck wrote: > > > > > >> +1 (binding) > > >> > > >> On Tue, Sep 28, 2021 at 12:59 PM Matthias J. Sax > >

[jira] [Created] (KAFKA-10008) Symbol not found when running Kafka Streams with RocksDB dependency on MacOS 10.13.6

Victoria Xia created KAFKA-10008: Summary: Symbol not found when running Kafka Streams with RocksDB dependency on MacOS 10.13.6 Key: KAFKA-10008 URL: https://issues.apache.org/jira/browse/KAFKA-10008

[jira] [Created] (KAFKA-14491) Introduce Versioned Key-Value Stores to Kafka Streams

Victoria Xia created KAFKA-14491: Summary: Introduce Versioned Key-Value Stores to Kafka Streams Key: KAFKA-14491 URL: https://issues.apache.org/jira/browse/KAFKA-14491 Project: Kafka Issue

[jira] [Created] (KAFKA-14723) Do not write expired store records to changelog

Victoria Xia created KAFKA-14723: Summary: Do not write expired store records to changelog Key: KAFKA-14723 URL: https://issues.apache.org/jira/browse/KAFKA-14723 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14834) Improved stream-table and table-table join semantics for versioned stores

Victoria Xia created KAFKA-14834: Summary: Improved stream-table and table-table join semantics for versioned stores Key: KAFKA-14834 URL: https://issues.apache.org/jira/browse/KAFKA-14834 Project

[jira] [Created] (KAFKA-14864) Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy

Victoria Xia created KAFKA-14864: Summary: Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy Key: KAFKA-14864 URL: https://issues.apache.org/jira/browse/KAFKA-14864 Project

[jira] [Created] (KAFKA-14949) Add Streams upgrade tests from AK 3.4

Victoria Xia created KAFKA-14949: Summary: Add Streams upgrade tests from AK 3.4 Key: KAFKA-14949 URL: https://issues.apache.org/jira/browse/KAFKA-14949 Project: Kafka Issue Type: Task

[jira] [Created] (KAFKA-12366) Performance regression in stream-table joins on trunk

Victoria Xia created KAFKA-12366: Summary: Performance regression in stream-table joins on trunk Key: KAFKA-12366 URL: https://issues.apache.org/jira/browse/KAFKA-12366 Project: Kafka Issue