Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Actually, scratch that. On reflection I think I prefer Bruno's original idea to specify it in the configuration. Cheers, Nick On Sat, 16 Nov 2024 at 17:59, Nick Telford wrote: > Hey everyone, > > With respect to Bruno's proposal, could instances cache their topology > e

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Hey everyone, With respect to Bruno's proposal, could instances cache their topology epoch on disk, and then upgrades/downgrades would simply involve deleting the cached epoch before starting the instance? My thinking is that this might be potentially simpler for users than modifying configuratio

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

operations point of view, just letting the app being blocked > > with a informational log entry to quickly ping-down the zombie clients > > may actually be acceptable. All in all, it makes the code simpler > > programmingly by not trying to abstract away issue scenario a) from > &

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

> more that locking is required to prevent concurrent access. In > particular, I was expecting that the lock will avoid two threads > opening the same RocksDB in KIP-1035. Wouldn't this cause problems? > > Cheers, > Lucas > > On Fri, Aug 16, 2024 at 11:34 AM Nick Telford &

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

le typically use in > their production workloads. > > NT4. We will actually only report the offsets if we manage to acquire > the lock. I tried to make this more precise. I suppose also with > KIP-1035, we'd require the lock to read the offset? > > Cheers, > Lucas > >

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

Hi everyone, Looks really promising, and I can see this resolving several issues I've noticed. I particularly like the choice to use a String for Subtopology ID, as it will (eventually) lead to a better solution to KIP-816. I noticed a few typos in the KIP that I thought I'd mention: NT1. In sev

Re: [VOTE] KIP-1035: StateStore managed changelog offsets

) > > > > > > Best, > > > Bruno > > > > > > On 6/13/24 2:31 AM, Matthias J. Sax wrote: > > > > Thanks Nick. > > > > > > > > +1 (binding) > > > > > > > > > > > > Looking forward t

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Hi everyone, On a semantic note, would it perhaps make more sense to rename them " uncaught." instead? These handlers are essentially the "last resort" exception handlers, because Exceptions can be caught *within* a component, e.g. a Deserializer can catch and handle an exception without the confi

[VOTE] KIP-1035: StateStore managed changelog offsets

Hi everyone, I'd like to call a vote on KIP-1035[1]. Regards, Nick 1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

tructure POV is seems best to add to all segments, but it seem > sufficient to keep the information up-to-date only in the latest > segments (what would imply that we need to copy the information from the > current latest segment to a newly created segment explicitly) and only > _read_ t

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

) -> numCommittedEntryStored() flushedEntryRemoved(K) -> committedEntryRemoved(K) flushedEntryStored(K) -> committedEntryStored(K) The old methods will obviously be marked as @Deprecated. Any objections before I add this to the KIP? Regards, Nick On Wed, 29 May 2024 at 11:20, Nick Telford wr

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

e this covers all the outstanding changes that were requested. Please let me know if I've missed anything or you think further changes are needed. Regards, Nick On Wed, 29 May 2024 at 09:28, Nick Telford wrote: > Hi everyone, > > Sorry I haven't got around to updating the KIP yet.

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

> > release, but would also like to move forward with with one. > > > > Should we start a VOTE? > > > > For merging PRs we need to wait after code freeze, and 3.8 branch was > > but. But we could start reviewing PRs before this already. > > > > > >

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

ss by its own. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Couple of question / comment: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>&

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

p Prat > Open Source Engineering Director, aivenjosep.p...@aiven.io | > +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Thu, May 16, 2024,

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

t; > > > On 5/14/24 9:19 AM, Lucas Brutschy wrote: > > > Hi Nick! > > > > > > Thanks for the KIP. > > > > > > +1 (binding) > > > > > > On Tue, May 14, 2024 at 5:16 PM Nick Telford > > wrote: > > >> &g

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

/current/streams/monitoring.html#state-store-metrics On Thu, 16 May 2024 at 12:15, Nick Telford wrote: > Good point! I've updated it to "Improved StateStore Iterator metrics for > detecting leaks" - let me know if you have a better suggestion. > > This should affect th

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

e: > One quick thing -- can you update the title of this KIP to reflect the > decision to implement these metrics for all state stores implementations > rather than just RocksDB? > > > On Tue, May 14, 2024 at 1:36 PM Nick Telford > wrote: > > > Woops! Thanks for the cat

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Josep, Would it be possible to sneak KIP-989 into 3.8? Just as with 1028, it's currently being voted on and has already received the requisite votes. The only thing holding it back is the 72 hour voting window. Vote thread here: https://lists.apache.org/thread/nhr65h4784z49jbsyt5nx8ys81q90k6s

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

etter, I > > > > would propose `iterator-max-age-ms`. I should be sufficient to call > out > > > > (as it's kinda "obvious" anyway) that the metric applies to open > > > > iterator only. > > > > > > > > And yes, I was hoping that the code inside M

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

>>> > >>>>>>> 102: It's unclear to me, how `.position` information is added. > >>>>>>> The KIP only says: "position offsets will be stored in RocksDB, > >>>>>>> in the same column family as the changelog offsets". Do you >

[VOTE] KIP-989: RocksDB Iterator Metrics

Hi everyone, I'd like to call a vote on the Kafka Streams KIP-989: RocksDB Iterator Metrics: https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics All of the points in the discussion thread have now been addressed. Regards, Nick

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Walker, Feel free to ask away, either on the mailing list of the Confluent Community Slack, where I hang out :-) The implementation is *mostly* complete, although it needs some polishing. It's worth noting that KIP-1035 is a hard prerequisite for this. Regards, Nick

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

gt; } > > And of course add the TaskId parameter to each of the actual > state store constructors returned here. > > Does that make sense? It's entirely possible I'm missing something > important here, but I think this would be a pretty small addition that

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

t takes in a taskId parameter? We can have it default to > invoking the old one for compatibility reasons and it should be completely > safe to tack on. > > Would also prefer the same for a ProcessorSupplier, but that's definitely > outside the scope of this KIP > > On

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

gt; possible). > > 5. I think we should be clear that this KIP only covers the DLQ record > produced. > Everything related to replay messages or recovery plan should be > considered out-of-scope as it is use-case and error specific. > > Let me know if that's not clear, t

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

essed. I'd like to see a section that considers these consequences, and perhaps make those risks clear to users. For the record, this is exactly what sunk KIP-990, which was an alternative approach to error handling that introduced the same issues. Cheers, Nick On Fri, 12 Apr 2024 at 1

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Damien, Thanks for the KIP! Dead-letter queues are something that I think a lot of users would like. I think there are a few points with this KIP that concern me: 1. It looks like you can only define a single, global DLQ for the entire Kafka Streams application? What about applications that w

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

, but it looks like we'll have to live with it. Unless you have any better ideas? Regards, Nick On Wed, 10 Apr 2024 at 14:12, Nick Telford wrote: > Hi Bruno, > > Immediately after I sent my response, I looked at the codebase and came to > the same conclusion. If it's possibl

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

encapsulation for the unused task directories? > > > Best, > Bruno > > > > On 4/10/24 11:31 AM, Nick Telford wrote: > > Hi Bruno, > > > > Thanks for the review! > > > > 1, 4, 5. > > Done > > > > 3. > > You'r

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

plementation detail. I would remove it from the KIP, but still > state that the legacy checkpointing behavior will be supported when the > state store does not manage the checkpoints. > > > 5. > Regarding the metrics, could you please add the tags, and the recording > level

[DISCUSS] KIP-1035: StateStore managed changelog offsets

Hi everyone, Based on some offline discussion, I've split out the "Atomic Checkpointing" section from KIP-892: Transactional Semantics for StateStores, into its own KIP KIP-1035: StateStore managed changelog offsets https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

terator-age-seconds" should be > > "oldest-open-iterator-age-ms". Milliseconds is obviously a better > > granularity for such a metric. > > > > Still accepting suggestions for a better name. > > > > On Thu, 28 Mar 2024 at 13:41, Nick Telford &g

Re: [DISCUSS] KIP-816: Topology changes without local state reset

treams. Let me know what you think, Nick On Tue, 15 Feb 2022 at 16:23, Nick Telford wrote: > In the KIP, for Option A I suggested a new path of: > > /state/dir/stores// > > I made the mistake of thinking that the rocksdb/ segment goes *after* the > store name in the current schem

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

Quick addendum: My suggested metric "oldest-open-iterator-age-seconds" should be "oldest-open-iterator-age-ms". Milliseconds is obviously a better granularity for such a metric. Still accepting suggestions for a better name. On Thu, 28 Mar 2024 at 13:41, Nick Telford w

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

;INFO" level for this > metric, > > > since unlike the other metrics which are "Measurables", the current > > > timestamp won't need to be retrieved on each recording > > > > > > 5. Can you list the tags that would be associated with each of

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

offset from outside while the application is running. The offset in > > >> question is cached inside the consumer and the consumer would not go > > >> back to Kafka to re-read the offset (only when a partitions is > > >> re-assigned to a new consumer, the consum

Re: Kafka trunk test & build stability

Addendum: I've opened a PR with what I believe are the changes necessary to enable Remote Build Caching, if you choose to go that route: https://github.com/apache/kafka/pull/15109 On Tue, 2 Jan 2024 at 14:31, Nick Telford wrote: > Hi everyone, > > Regarding building a &qu

Re: Kafka trunk test & build stability

Hi everyone, Regarding building a "dependency graph"... Gradle already has this information, albeit fairly coarse-grained. You might be able to get some considerable improvement by configuring the Gradle Remote Build Cache. It looks like it's currently disabled explicitly: https://github.com/apach

Re: Apache Kafka 3.7.0 Release

Hi Stan, I'd like to propose including KIP-892 in the 3.7 release. The KIP has been accepted and I'm just working on rebasing the implementation against trunk before I open a PR. Regards, Nick On Tue, 21 Nov 2023 at 11:27, Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > Hi Stan > > C

Re: [VOTE] KIP-892: Transactional StateStores

n-binding). > >>> > >>> Thank you, Nick, for making all of the changes (especially around the > >>> `default.state.isolation.level` config). > >>> > >>> Colt McNealy > >>> > >>> *Founder, LittleHorse.dev* > >

[VOTE] KIP-892: Transactional StateStores

Hi everyone, I'd like to call a vote for KIP-892: Transactional StateStores[1], which makes Kafka Streams StateStores transactional under EOS. Regards, Nick 1: https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

he #resume method so that it always skips over the bad record. This > will probably be the easiest to implement anyways, as it is effectively the > same as the CONTINUE option internally, but gives the user time to > decide if they really do want to CONTINUE or not > > Not sure if

Re: [DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

wnstream/dependent tasks? Should be able to add this > information to the subscription metadata and send to all clients via a > rebalance. There might be better options I'm not seeing. Or we could just > decide to trust the users not to shoot themselves in the

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

ficantly. So we may even want to detect if the > > iterators are opened by one-time / rare queries against the state > > store. > > > > But, as I said, that would be an addition and not a change of the > > current contents of the KIP, so I'd support the KIP movi

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

This KIP is already improving the > situation a lot by not wiping the state store. > > Cheers, > Lucas > > On Tue, Oct 17, 2023 at 3:51 PM Nick Telford > wrote: > > > > Hi Lucas, > > > > Yeah, this is pretty much the direction I'm thinking of goin

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

; behavior (we don't necessarily have to be, because it doesn't violate > ALOS guarantees as far as I can see), we could make > ALOS/READ_COMMITTED more similar to ALOS/READ_UNCOMITTED by flushing > the WriteBatch on error (obviously, only if we have a chance to do > that). >

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

closed iterator or the > number of iterators. But it could still be good to identify those > leaks early. One option would be to add `iterator-duration-max` and > take open iterators into account when computing the metric. > > Cheers, > Lucas > > > On Thu, Oct 5, 2023 at 3:5

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

n that we can > potentially deprecate than "shut the door, clean for everyone". More > specifically, allowing the processing mode / IQ read mode to be > decoupled, and if we found that there's no such cases as I speculated > above or people started complaining a lot, we

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

re is a guideline in Kafka not to use the get prefix for getters (at > least in the public API). Thus, could you please rename > > getCommittedOffset(TopicPartition partition) -> > committedOffsetFor(TopicPartition partition) > > You can also propose an alternative to c

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

always including a > feature > > flag in large structural > > changes like this. No matter how much I trust someone or myself to > > implement a feature, you just > > never know what kind of bugs might slip in, especially with the very > first > > iteration

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

goes well > you can do a quick KIP to enable it by default as soon as the > isolation.level config has been > completed. But feel free to just pick whichever option is easiest or > quickest for you to implement) > > Hope this helps move the discussion forward, > Sophie > >

[DISCUSS] KIP-990: Capability to SUSPEND Tasks on DeserializationException

Hi everyone, This is a Streams KIP to add a new DeserializationHandlerResponse, "SUSPEND", that suspends the failing Task but continues to process other Tasks normally. https://cwiki.apache.org/confluence/display/KAFKA/KIP-990%3A+Capability+to+SUSPEND+Tasks+on+DeserializationException I'm not ye

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

erator have its own "createdAt()" or equivalent field, or do we > need to keep track of the Iterator's start time upon creation? > > Cheers, > Colt McNealy > > *Founder, LittleHorse.dev* > > > On Wed, Oct 4, 2023 at 9:07 AM Nick Telford > wrote: > > &g

[DISCUSS] KIP-989: RocksDB Iterator Metrics

Hi everyone, KIP-989 is a small Kafka Streams KIP to add a few new metrics around the creation and use of RocksDB Iterators, to aid users in identifying "Iterator leaks" that could cause applications to leak native memory. Let me know what you think! https://cwiki.apache.org/confluence/display/K

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

t 2023 at 09:30, Bruno Cadonna wrote: > Why do we need to add READ_COMMITTED to InMemoryKeyValueStore? I think > it is perfectly valid to say InMemoryKeyValueStore do not support > READ_COMMITTED for now, since READ_UNCOMMITTED is the de-facto default > at the moment. > > Best, &g

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Oh! One other concern I haven't mentioned: if we make IsolationLevel a query-time constraint, then we need to add support for READ_COMMITTED to InMemoryKeyValueStore too, which will require some changes to the implementation. On Mon, 18 Sept 2023 at 17:24, Nick Telford wrote: > Hi

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

because processing is deterministic. Additionally, IQ being able to read > > uncommitted records is crucial to enable "read your own writes" on our > API: > > Due to the deterministic processing, we send an "ack" to the client who > > makes the request as soo

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

IQ Iterators in the same way that RocksDB WriteBatches do. -- Nick On Wed, 13 Sept 2023 at 16:58, Nick Telford wrote: > Hi Bruno, > > I've updated the KIP based on our conversation. The only things I've not > yet done are: > > 1. Using transactions under ALOS and EOS.

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

nage to get > the internals right. Regarding state stores that do not support > READ_COMMITTED, they should throw an error stating that they do not > support READ_COMMITTED. No need to adapt all state stores immediately. > > 3b. > I am in favor of using transactions also for ALOS. &g

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

tionLevels, even if the StateStore has many layers of wrappers (as is the case at the point IQv1 deals with the store). Would this be acceptable, or do you have another approach in mind? Regards, Nick On Wed, 13 Sept 2023 at 10:57, Nick Telford wrote: > Hi Bruno, > > Thanks for gettin

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

patibility, Deprecation, and Migration Plan". > > > 6. > Describing upgrading and downgrading in the KIP is a good idea. > Regarding downgrading, I think you could also catch the exception and do > what is needed to downgrade, e.g., drop the column family. See here for > an example: >

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

e has a > detailed breakdown of the testing. > > Anyways, I'm super excited about this KIP and if a bit more future testing > goes well, we plan to ship our product with a build of KIP-892, Speedb OSS, > and potentially a few other minor tweaks that we are thinking about

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

old > ones. You can find examples of deprecated metrics here: > https://kafka.apache.org/documentation/#selector_monitoring > > > 5. > Why does the KIP mention position files? I do not think they are related > to transactions or flushes. > > > 6. > I think we will

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

already grown quite large! On Fri, 21 Jul 2023 at 21:33, Nick Telford wrote: > Hi everyone, > > I've updated the KIP ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores) > with the latest changes; mostly bringing back &quo

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

e either of these, so my primary test environment doesn't test these areas. I'm going on Parental Leave starting next week for a few weeks, so will not have time to move this forward until late August. That said, your feedback is welcome and appreciated, I just won't be able to respo

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

tore is idempotent, so eos should not violated. > This solution needs at least one new config to specify when a checkpoint > should be written. > > > > A small correction to your previous e-mail that does not change anything > you said: Under alos the default commit interval is

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

ommit I'm also open to suggestions. Regards, Nick On Thu, 22 Jun 2023 at 14:06, Nick Telford wrote: > Hi Bruno! > > 3. > By "less predictable for users", I meant in terms of understanding the > performance profile under various circumstances. The more complex the &

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

that you proposed. > > 8. > A high-level description (without implementation details) of how Kafka > Streams will manage the commit of changelog transactions, state store > transactions and checkpointing would be great. Would be great if you > could also add some sentences about t

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Writing and ingesting SST files is not a > RocksDB internal, but rather a supported usage pattern on public APIs. > Regardless, I think your overall preference is fine with me, especially > if we can internalize this change within the store implementation itself. > > Thanks, > -Jo

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Bruno, 1. Isn't this exactly the same issue that WriteBatchWithIndex transactions have, whereby exceeding (or likely to exceed) configured memory needs to trigger an early commit? 2. This is one of my big concerns. Ultimately, any approach based on cracking open RocksDB internals and using it

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

ge across RocksDB memtables, which points to another benefit of > disabling memtables and managing the write buffer ourselves (simplified > memory configuration). > > Thanks, > -John > > On 6/20/23 16:05, Nick Telford wrote: > > Potentially we could just go the memorable with R

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

probably not be > performant, but benchmarking may prove different. Or maybe we can have > some reserved cache space on top of the user-configured cache, which we > would have reclaimed from the memtable space. Or some other, more > creative solution. > > Thanks, > -John > &g

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

ble the cache, which would still be > ok, I think. We wouldn't ingest the SST files on every record, but just > append to them and only ingest them on commit, when we're already > waiting for acks and a RocksDB commit. > > Thanks, > -John > > On 6/20/23 14:09,

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi John, I think you're referring to the "record cache" that's provided by the ThreadCache class? 1-3. I was hoping to (eventually) remove the "flush-on-commit" behaviour from RocksDbStore, so that RocksDB can choose when to flush memtables, enabling users to tailor RocksDB performance to their w

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

file ingestions > > I suspect this'll have a number of benefits: > * writes to RocksDB will always be atomic > * we don't fragment memory between the RecordCache and the memtables > * RecordCache gives far higher performance than memtable for reads and > writes > *

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

disk. You would only need to change the doc of the config, if you agree > with me. > > 8. > Section "Transaction Management" about the wrappers is rather a > implementation detail that should not be in the KIP. > > 9. > Could you add a section that describes ho

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi everyone, Quick update: I've added a new section to the KIP: "Offsets for Consumer Rebalances", that outlines my solution to the problem that StreamsPartitionAssignor needs to read StateStore offsets even if they're not currently open. Regards, Nick On Wed, 3 May 2023 at

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

during the last commit. Basically, Streams > would overwrite the records written to the state store between the last > two commits with the same records read from the changelogs. While I > understand that this is wasteful, it is -- at the same time -- > acceptable and most imp

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

. The downside is that we wouldn't be able to remove the explicit RocksDB flush on-commit, which likely hurts performance. If anyone has any thoughts or ideas on this subject, I would appreciate it! Regards, Nick On Wed, 19 Apr 2023 at 15:05, Nick Telford wrote: > Hi Colt, > &g

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

osed solution works equivalently and I don't believe > there is much overhead to an additional column family in RocksDB. Perhaps > it may even perform better than making separate writes to the checkpoint > file. > > Colt McNealy > *Founder, LittleHorse.io* > > >

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

eplay a few more records (at a cost of > <100ms). Am I missing something there? > > Other than that, everything makes sense to me. > > Cheers, > Colt McNealy > *Founder, LittleHorse.io* > > > On Tue, Apr 18, 2023 at 3:59 AM Nick Telford > wrote: > > > Hi everyo

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

a bunch of interface changes relating to Atomic Checkpointing, which is the final piece of the puzzle to making everything robust. Let me know what you think! Regards, Nick On Tue, 3 Jan 2023 at 11:33, Nick Telford wrote: > Hi Lucas, > > Thanks for looking over my KIP. > > A)

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

ound, but I > do not understand this, it would be great if you could clarify what > you mean here. > D) Could you please clarify why IQ has to call newTransaction(), when > it's read-only. > > And one last thing not strictly related to your KIP: if there is an > easy way for

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

that it would be considered a major change, I like your approach > the best. > > Wishing you a speedy recovery and happy coding! > > Thanks, > Colt McNealy > *Founder, LittleHorse.io* > > > On Tue, Dec 6, 2022 at 10:30 AM Nick Telford > wrote: > > > Hi Co

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

d > definitely want to enable queries on dirty reads; otherwise users would > have to wait 30 seconds (default) to see an update. > > Thank you for doing this fantastic work! > Colt McNealy > *Founder, LittleHorse.io* > > > On Wed, Nov 30, 2022 at 10:44 AM Nick Telford >

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

rns around concurrency, especially in the presence of Iterators. I'm thinking of wrapping WriteBatchWithIndex with a reference-counting copy-on-write implementation (that only makes a copy if there's an active iterator), but I'm open to suggestions. Regards, Nick On Mon, 28 Nov 20

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

> store values when copying from the uncommitted to committed store, but I > wasn't able to figure that out when I scanned the PR. > > Colt McNealy > *Founder, LittleHorse.io* > > > On Mon, Nov 28, 2022 at 7:56 AM Nick Telford > wrote: > > > Hi everyone, &

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

dy be explicitly configuring this for their purposes. I think a further optimization for ALOS to only commit the StateStores when exceeding this limit is reasonable, to preserve the user's desired commit.interval.ms as much as possible. On Mon, 28 Nov 2022 at 15:55, Nick Telford wrote: >

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

end on the processing.mode; under ALOS it would allow more frequently committing stores, whereas under EOS it couldn't. Any better ideas? On Wed, 23 Nov 2022 at 16:25, Nick Telford wrote: > Hi Alex, > > Thanks for the feedback. > > I've updated the discussion of OOM issues

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

is not > necessary to solve the recovery-under-EOS problem. On the other hand, once > WriteBatchWithIndex is in, it will be much easier to add > state-store-managed checkpointing. > > If you share the current implementation, I am happy to help you address the > OOMe and config

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

method replaces flush, > updateChangelogOffsets, and checkpoint. It seems to me that the point about > atomicity and Position also suggests that it replaces the Position > callbacks. However, the proposal only deprecates `flush`. Should we be > deprecating other methods as well? > >

[DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi everyone, As I mentioned in the discussion thread for KIP-844, I've been working on an alternative approach to achieving better transactional semantics for Kafka Streams StateStores. I've published this separately as KIP-892: Transactional Semantics for StateStores

Re: [DISCUSS] KIP-844: Transactional State Stores

w in January. > > Best, > Alex > > On Fri, Nov 11, 2022 at 4:24 PM Nick Telford > wrote: > > > Hi everyone, > > > > Sorry to dredge this up again. I've had a chance to start doing some > > testing with the WIP Pull Request, and it appears as thou

Streams PR review request

Hi everyone, I found a small performance improvement in Kafka Streams state stores, would someone be able to review/merge it please? https://github.com/apache/kafka/pull/12842 (I'm not sure if this is the correct forum for requesting a review/merge. If it isn't, please let me know). Regards, Ni

Re: Streams: clarification needed, checkpoint vs. position files

;t think > > it would sense for Streams to try and consolidate these or replace one > with > > another. > > > > Hope this answers your question, and I'll ping John to make sure I'm not > > misleading > > you regarding the usage/intention of P

Re: [DISCUSS] KIP-844: Transactional State Stores

option c. Existing state is considered to be committed > and there will be an additional RocksDB for uncommitted writes. > > I am out of office until October 24. I will update KIP and make sure that > we have an upgrade test for that after coming back from vacation. > > Best, > Alex &

Streams: clarification needed, checkpoint vs. position files

Hi everyone, I'm trying to understand how StateStores work internally for some changes that I plan to propose, and I'd like some clarification around checkpoint files and position files. It appears as though position files are relatively new, and were created as part of the IQv2 initiative, as a

Re: [VOTE] KIP-869: Improve Streams State Restoration Visibility

Can't wait! +1 (non-binding) On Wed, 12 Oct 2022, 18:02 Guozhang Wang, wrote: > Hello all, > > I'd like to start a vote for the following KIP, aiming to improve Kafka > Stream's restoration visibility via new metrics and callback methods: > > > https://cwiki.apache.org/confluence/display/KAFKA/K

Re: [DISCUSS] KIP-869: Improve Streams State Restoration Visibility

Hi Guozhang, What you propose sounds good to me. Having the more detailed Task-level metrics at DEBUG makes sense. Regards, Nick

  1   2   >