[GitHub] [kafka-site] divijvaidya commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

divijvaidya commented on code in PR #528: URL: https://github.com/apache/kafka-site/pull/528#discussion_r1236535926 ## blog.html: ## @@ -31,6 +31,7 @@ We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and impro

[jira] [Resolved] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances

[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass resolved KAFKA-15059. - Resolution: Fixed [~ChrisEgerton] since the PR is merged, I resolve this ticket.

[jira] [Created] (KAFKA-15110) Wrong version may be run, which will cause it to fail to run when there are multiple version jars under core/build/libs

hudeqi created KAFKA-15110: -- Summary: Wrong version may be run, which will cause it to fail to run when there are multiple version jars under core/build/libs Key: KAFKA-15110 URL: https://issues.apache.org/jira/browse/KA

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

I have an implementation of a layered Bloom filter in [1] (note the layered branch). This should handle the layering Bloom filter and allow for layers that 1. Do not become over populated and thus yield too many false positives. 2. Expire and are removed automatically. The layered Bloom fi

[GitHub] [kafka-site] mimaison commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

mimaison commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1600601309 I agree with Divij that we need to be careful about the wording. Deprecated typically means "do not use for anything new" and "migrate to the alternative as soon as possible". For both

[GitHub] [kafka-site] showuon commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

showuon commented on code in PR #528: URL: https://github.com/apache/kafka-site/pull/528#discussion_r1236779033 ## blog.html: ## @@ -31,6 +31,7 @@ We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improveme

Re: [DISCUSS] KIP-936: Throttle number of active PIDs

I think that the either using a Stable bloom filter or a Layered bloom filter constructed as follows: - Each layer is configured for the maximum number of principal-PID pairs expected in a single minute. - Expect 60 layers (one for each minute) - If the layer becomes fully populated a

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1941

See Changes: -- [...truncated 580123 lines...] [2023-06-21T11:07:24.556Z] Gradle Test Run :streams:integrationTest > Gradle Test Executor 181 > StoreQueryIntegrationTest >

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi John, Hi Nick, Thanks for the interesting ideas! Here my comments. 1. It is not clear to me what happens if the cache exceeds its configured size between two commits? Currently, the cache evicts its entries and writes dirty entries to the state store. Should the cache write those dirty en

[GitHub] [kafka-site] mumrah merged pull request #525: Add ZK to KRaft migration docs

mumrah merged PR #525: URL: https://github.com/apache/kafka-site/pull/525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.o

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

Hi Chris, Firstly, thanks for sharing your detailed thoughts on this thorny issue! Point taken on Kafka Connect being a brownfield project and I guess we might just need to trade off elegant / "clean" interfaces for fixing this gap in functionality. Also, thanks for calling out all the existing cr

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1942

See

[GitHub] [kafka-site] ijuma commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on code in PR #528: URL: https://github.com/apache/kafka-site/pull/528#discussion_r1237224150 ## blog.html: ## @@ -31,6 +31,7 @@ We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improvement

[GitHub] [kafka-site] ijuma commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on code in PR #528: URL: https://github.com/apache/kafka-site/pull/528#discussion_r1237224150 ## blog.html: ## @@ -31,6 +31,7 @@ We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improvement

[GitHub] [kafka-site] ijuma commented on a diff in pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on code in PR #528: URL: https://github.com/apache/kafka-site/pull/528#discussion_r1237224150 ## blog.html: ## @@ -31,6 +31,7 @@ We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improvement

[GitHub] [kafka-site] ijuma commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601095246 > Deprecated typically means "do not use for anything new" and "migrate to the alternative as soon as possible" @mimaison Right and that's true here too. For something like kafka, m

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

Hi Edo/Adrian! Thanks for the KIP. I have some questions, and apologies that the may fall under the “stupid” column because I’m not that familiar with this area :) 1. Does record.validation.policy.class.name support multiple classes, or just one? I’m probably not wrapping my head around it, bu

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Hi Bruno, 1. Isn't this exactly the same issue that WriteBatchWithIndex transactions have, whereby exceeding (or likely to exceed) configured memory needs to trigger an early commit? 2. This is one of my big concerns. Ultimately, any approach based on cracking open RocksDB internals and using it

Re: [DISCUSS] KIP-941: Range queries to accept null lower and upper bounds

Hi Lucia, Thanks for the KIP! The KIP wasn’t in the email and I didn’t see it on the main KIP directory. Here it is: https://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds Can the KIP be added to the main KIP page (https://cwiki.apache

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

Hi All, I have created this PR: https://github.com/apache/kafka/pull/13899 which implements the approach outlined in the latest version of the KIP. I thought I could use this to validate the approach based on my understanding while the KIP itself gets reviewed. I can always change the implementati

Re: [DISCUSS] KIP-941: Range queries to accept null lower and upper bounds

Hi Lucia, One question: 1. Since the proposed implementation change for withRange() method uses Optional.ofNullable() (which only catches nulls and not blank/whitespace strings), wouldn’t users still need to have code like that in the example? 2. Why don't users create RangeQuery objects that

[GitHub] [kafka-site] mimaison commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

mimaison commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601241236 > Agreed, but it's better than waiting until 3.6. Also, as I said in the other comment, the video does mention this. Finally, we will reinforce it in subsequent releases too - there's n

Re: [DISCUSS] KIP-941: Range queries to accept null lower and upper bounds

Hi all, Thanks for the KIP, Lucia! This is a nice change. To Kirk's question (1), the example is a bit misleading. The typical case that would ease user pain is specifically using "null" to indicate an open-ended range, especially since null is not a valid key. I could additionally see an em

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Thanks Nick, That sounds good to me. I can't let (2) slide, though.. Writing and ingesting SST files is not a RocksDB internal, but rather a supported usage pattern on public APIs. Regardless, I think your overall preference is fine with me, especially if we can internalize this change within

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

Sorry John, I didn't mean to mis-characterize it like that. I was mostly referring to disabling memtables. AFAIK the SstFileWriter API is primarily designed for bulk ingest, e.g. for bootstrapping a database from a backup, rather than during normal operation of an online database. That said, I was

[GitHub] [kafka-site] divijvaidya commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

divijvaidya commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601303527 IMO, we should have a living document "Zk deprecation roadmap" which is updated with every release. Every release blog post will have a section pointing to this document. This solves

[GitHub] [kafka-site] ijuma commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601308506 > The video is not from Apache so I'm not sure we should use that as authority. That's fair - I thought these release videos had been moved to the Apache account, but it was a diffe

[GitHub] [kafka-site] ijuma commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601312969 @divijvaidya The KIP linked from this PR is the living document right now. I think we're overthinking this, but if anyone here wants to volunteer to write an additional blog post - please

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

No worries, I should have included a ";)" to let you know it was mostly tongue-in-cheek. Thanks, -John On 6/21/23 12:34, Nick Telford wrote: Sorry John, I didn't mean to mis-characterize it like that. I was mostly referring to disabling memtables. AFAIK the SstFileWriter API is primarily desig

Re: [DISCUSS] KIP-941: Range queries to accept null lower and upper bounds

Hi John/Lucia, Thanks for the feedback! Of course I only noticed the private-ness of the RangeQuery constructor moments after sending my email ¯\_(ツ)_/¯ Just to be clear, I’m happy with the proposed change as it conforms to Postel’s Law ;) Apologies that it was worded tersely. Thanks, Kirk

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1943

See

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hi Justine, I have initiated the voting process for this KIP here: https://lists.apache.org/thread/y3yfnphsmrgwfdhx3xfhjtwdb7p1dn0v We have already received two binding votes, and I am seeking a third vote for the adoption of the KIP. As you have previously reviewed this KIP, would you be will

[GitHub] [kafka-site] divijvaidya commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

divijvaidya commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601385284 > I still think our users are better off if we update the announcement and docs now vs a few months later. Can someone explain why the latter would be better? For me, I don't

[GitHub] [kafka-site] mimaison commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

mimaison commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601606196 To clarify I'm not against updating the blog post. I hope my replies are not perceived as confrontational. I think we (and that includes me) messed up the external communication d

[jira] [Created] (KAFKA-15111) Correction kafka examples

Dmitry created KAFKA-15111: -- Summary: Correction kafka examples Key: KAFKA-15111 URL: https://issues.apache.org/jira/browse/KAFKA-15111 Project: Kafka Issue Type: Task Reporter: Dmitry

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

Hey Mehari, I was just getting annoyed looking at logs of kafka clusters with timestamp issues. Let me take final look at the KIP. Thanks, Justine On Wed, Jun 21, 2023 at 11:13 AM Beyene, Mehari wrote: > Hi Justine, > > I have initiated the voting process for this KIP here: > https://lists.apac

Re: [VOTE] KIP-937: Improve Message Timestamp Validation

Hey Mehari, Thanks for the KIP. One minor change is now that we have the before and after configs does this error make sense? > we will return error code 32 (INVALID_TIMESTAMP) with the error message "Timestamp of the message with offset [record offset] is ahead of the broker's current time." I

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1944

See Changes: -- [...truncated 566711 lines...] [2023-06-21T21:27:01.978Z] [2023-06-21T21:27:01.978Z] > Task :streams:javadoc [2023-06-21T21:27:01.978Z] > Task :streams:jav

[GitHub] [kafka-site] ijuma commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601730235 @divijvaidya @mimaison I agree with what you both said. Also, I take this whole debate as constructive conversation to do the right thing for our users. :) > We can keep updating KI

[GitHub] [kafka-site] ijuma commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

ijuma commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601732256 > what is the EOL security support for last minor version of 3.x (we could point to existing policy if no change) > what is the EOL bug support for last minor version of 3.x (we could po

Re: [VOTE] KIP-937: Improve Message Timestamp Validation

Thank you, Justin. That makes sense. I have updated the KIP to remove the concept of ahead/behind. Instead, we will use the existing error message that utilizes the acceptable range for the timestamps. Thanks, Mehari

[jira] [Created] (KAFKA-15112) Allow ignoring advertised listeners

Marcin Wisnicki created KAFKA-15112: --- Summary: Allow ignoring advertised listeners Key: KAFKA-15112 URL: https://issues.apache.org/jira/browse/KAFKA-15112 Project: Kafka Issue Type: Wish

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1945

See Changes: -- [...truncated 581913 lines...] [2023-06-22T00:56:57.390Z] [2023-06-22T00:56:57.390Z] > Task :streams:javadoc [2023-06-22T00:56:58.491Z] > Task :streams:jav

Re: [DISCUSS] KIP-940: Broker extension point for validating record contents at produce time

Hi Eduardo, Adrian. Thanks for the KIP. I agree that allowing custom validations on the broker-side addresses a real gap as you clearly stated on the motivation. Some initial thoughts from my side: 1. Similar to Kirk's first point, I'm also concerned on how would the plugin developers / operator

[GitHub] [kafka-site] mjsax commented on pull request #528: MINOR: Add statmenet about ZK deprecation to 3.5 release blog post

mjsax commented on PR #528: URL: https://github.com/apache/kafka-site/pull/528#issuecomment-1601987494 Haha. This "escalated" quickly 🤣 I agree with everything discussed. I only don't understand > Kraft production readiness is an independent topic than Zk deprecation. To me,