Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

Hi all, Thanks all for voting! Currently we have +3 binding (from Divij Vaidya, Luke Chen and ziming deng) and +2 non-binding(from Kirk True and Kamal Chandraprakash). This vote thread has been open over 72 hours and has sufficient votes, so I’ll close the voting at this time. KIP-872 has PASSE

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

Hi John, Thank you for your reminder! Best, ShunKang John Roesler 于2023年6月20日 周二22:11写道： > Hi Divij and ShunKang, > > I pulled open this thread to see if you needed my vote, but FYI, Divij is > a committer now, so he can re-cast his vote as binding. > > Thanks, > -John > > On 2023/06/20 13:37:0

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1940

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #24

See

[GitHub] [kafka-site] mjsax opened a new pull request, #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

mjsax opened a new pull request, #528: URL: https://github.com/apache/kafka-site/pull/528 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mai

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #145

See Changes: -- [...truncated 525857 lines...] [2023-06-20T23:26:33.505Z] /home/jenkins/workspace/Kafka_kafka_3.4@2/streams/src/main/java/org/apache/kafka/streams/kstream/KSt

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #178

See Changes: -- [...truncated 256696 lines...] [2023-06-20T23:01:13.548Z] [2023-06-20T23:01:13.548Z] org.apache.kafka.streams.integration.StandbyTaskCreationIntegrationTest

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1939

See

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Here's what I'm thinking: based on Bruno's earlier feedback, I'm going to try to simplify my original design down such that it needs no/minimal changes to the public interface. If that succeeds, then it should also be possible to transparently implement the "no memtables" solution as a performance

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Oh, that's a good point. On the topic of a behavioral switch for disabled caches, the typical use case for disabling the cache is to cause each individual update to propagate down the topology, so another thought might be to just go ahead and add the memory we would have used for the memtables

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

+1 (again) - binding :) Please update the PR ShunKang and tag me for review when you are ready. -- Divij Vaidya On Tue, Jun 20, 2023 at 4:11 PM John Roesler wrote: > Hi Divij and ShunKang, > > I pulled open this thread to see if you needed my vote, but FYI, Divij is > a committer now, so he

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Potentially we could just go the memorable with Rocks WriteBatches route if the cache is disabled? On Tue, 20 Jun 2023, 22:00 John Roesler, wrote: > Touché! > > Ok, I agree that figuring out the case of a disabled cache would be > non-trivial. Ingesting single-record SST files will probably not

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Touché! Ok, I agree that figuring out the case of a disabled cache would be non-trivial. Ingesting single-record SST files will probably not be performant, but benchmarking may prove different. Or maybe we can have some reserved cache space on top of the user-configured cache, which we would

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

> Note that users can disable the cache, which would still be ok, I think. We wouldn't ingest the SST files on every record, but just append to them and only ingest them on commit, when we're already waiting for acks and a RocksDB commit. In this case, how would uncommitted records be read by join

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Ah, sorry Nick, I just meant the regular heap based cache that we maintain in Streams. I see that it's not called "RecordCache" (my mistake). The actual cache is ThreadCache: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/ThreadCache.

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi John, I think you're referring to the "record cache" that's provided by the ThreadCache class? 1-3. I was hoping to (eventually) remove the "flush-on-commit" behaviour from RocksDbStore, so that RocksDB can choose when to flush memtables, enabling users to tailor RocksDB performance to their w

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi John, By "RecordCache", do you mean the RocksDB "WriteBatch"? I can't find any class called "RecordCache"... Cheers, Nick On Tue, 20 Jun 2023 at 19:42, John Roesler wrote: > Hi Nick, > > Thanks for picking this up again! > > I did have one new thought over the intervening months, which I'd

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1938

See

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Nick, Thanks for picking this up again! I did have one new thought over the intervening months, which I'd like your take on. What if, instead of using the RocksDB atomic write primitive at all, we instead just: 1. disable memtables entirely 2. directly write the RecordCache into SST file

[jira] [Created] (KAFKA-15109) ISR not expanding on ZK brokers during migration

David Arthur created KAFKA-15109: Summary: ISR not expanding on ZK brokers during migration Key: KAFKA-15109 URL: https://issues.apache.org/jira/browse/KAFKA-15109 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Bruno, Thanks for reviewing the KIP. It's been a long road, I started working on this more than a year ago, and most of the time in the last 6 months has been spent on the "Atomic Checkpointing" stuff that's been benched, so some of the reasoning behind some of my decisions have been lost, but

Re: First Contribution

Hi Steven, Thanks for your contribution! This is definitively a process we need to automate. I left a few comments in your PRs. Thanks, Mickael On Tue, Jun 13, 2023 at 8:11 PM Lovish Madhu wrote: > > Looks good, but I am also a newbie. > > However I feel like variables such as n = 10 as some me

Re: [DISCUSS] KIP-714: Client metrics and observability

Hi Andrew, > On Jun 13, 2023, at 8:06 AM, Andrew Schofield > wrote: > > Hi, > I would like to start a new discussion thread on KIP-714: Client metrics and > observability. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability > > I have edited th

[GitHub] [kafka-site] mimaison commented on pull request #521: KAFKA-14995: Automate asf.yaml collaborators refresh

mimaison commented on PR #521: URL: https://github.com/apache/kafka-site/pull/521#issuecomment-1599054544 Let's sort out https://github.com/apache/kafka/pull/13842 first but this will need to be updated to include our new committer. -- This is an automated message from the Apache Git Serv

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

Hey all, I'd like to revive this discussion. I've created https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness last November and it seems to be that there is a nice overlap between the two and would be good to merge. Should we revive KIP-660 and merge the two KIP

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1937

See Changes: -- [...truncated 484806 lines...] [2023-06-20T14:57:29.188Z] 230 actionable tasks: 124 executed, 106 up-to-date [2023-06-20T14:57:29.188Z] [2023-06-20T14:57:2

[jira] [Resolved] (KAFKA-15087) Move InterBrokerSendThread to server-commons module

[ https://issues.apache.org/jira/browse/KAFKA-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15087. - Fix Version/s: 3.6.0 Reviewer: David Jacot Resolution: Fixed > Move InterBrokerS

Request for review for my PR

Hi there, Can I ask for a quick review for my PR below ? I've addressed the comments a few days ago and I am eager to hear from you guys. https://github.com/apache/kafka/pull/13773 Thanks a lot Owen

[GitHub] [kafka-site] zzccctv opened a new pull request, #527: Update powered-by.html

zzccctv opened a new pull request, #527: URL: https://github.com/apache/kafka-site/pull/527 Added Kafkaide -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

Hi Divij and ShunKang, I pulled open this thread to see if you needed my vote, but FYI, Divij is a committer now, so he can re-cast his vote as binding. Thanks, -John On 2023/06/20 13:37:04 ShunKang Lin wrote: > Hi all, > > Bump this thread again and see if we could get a few more votes. > Cur

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

Hi all, Bump this thread again and see if we could get a few more votes. Currently we have +3 non-binding (from Divij Vaidya, Kirk True and Kamal Chandraprakash) and +2 binding (from Luke Chen and ziming deng). Hoping we can get this approved, reviewed, and merged in time for 3.6.0. Best, ShunKa

Build streaming graph processing on kafka

Kafka is a widely used streaming storage system. Many streaming processing system have build on kafka. e.g Flink & Spark Streaming.Recently We have build a streaming graph processing engine on kafka which can read data from kafkaand do incremental graph computing, in the end writing the results

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Nick, Thanks for the updates! I really appreciate that you simplified the KIP by removing some aspects. As I have already told you, I think the removed aspects are also good ideas and we can discuss them on follow-up KIPs. Regarding the current KIP, I have the following feedback. 1. Is t

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

Thanks Николай, We’d like to open a vote next week. Hopefully getting some more feedback before then. Edo On Wed, 7 Jun 2023 at 19:15, Николай Ижиков wrote: > Hello. > > As author of one of related KIPs I’m +1 for this change. > Long waited feature. > > > 7 июня 2023 г., в 19:02, Edoardo Comar

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

Thanks Colin! You're right. I started this KIP only thinking on the latest incremental API, and haven't thought much on the legacy one. After taking a another look at both APIs, I can see some inconsistencies on how the policies are applied in each case. I have added a section "Current workflow" [

Re: [VOTE] KIP-937: Improve Message Timestamp Validation

+1 (non-binding). Thanks, Andrew > On 19 Jun 2023, at 11:42, Divij Vaidya wrote: > > This KIP solves a real operational pain point for the administrator of > Kafka cluster. > > +1 (binding) > > -- > Divij Vaidya > > > > On Sun, Jun 18, 2023 at 5:09 AM Kirk True wrote: > >> +1 (non-binding) >> >