date:20231009

Re: Kafka Connect - Customize REST request headers

2023-10-09 Thread Yeikel Santana

Thank you for the explanation, Chris.

In case it helps, what I'm looking for is similar to KIP 577[1]. My specific 
example involves a hard-coded key/value pair that needs to be used for 
pod-to-pod as I can connect to any worker without that specific header, but 
workers cannot communicate among themselves without it.

To clarify, my environment is behind Istio[2], where Egress Traffic can be 
created using the following format: `..svc.cluster.local`.  For example, a request among workers should be:

curl -H "Host: ..svc.cluster.local" workerIP:PORT

Regarding temporary solutions, I've explored options like utilizing a proxy but 
I am running within containers that can complicate it further, along with the 
possibilities of patching, recompiling, or replacing the connect-runtime jar 
temporarily. I think that something like this might work but I need to test it 
: 

private static void addHeadersToRequest(HttpHeaders headers, Request req) {

 req.header("Host","..svc.cluster.local");

 if (headers != null) {

   String credentialAuthorization = 
headers.getHeaderString(HttpHeaders.AUTHORIZATION);

    if (credentialAuthorization != null) {

    req.header(HttpHeaders.AUTHORIZATION, credentialAuthorization);

    }

    }

    }

This is of course risky and it would be significantly more convenient if this 
functionality is integrated into Kafka Connect itself

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/KIP+577%3A+Allow+HTTP+Response+Headers+to+be+Configured+for+Kafka+Connect

[2] https://istio.io/

 On Sat, 07 Oct 2023 02:05:14 -0400 Chris Egerton  
wrote ---

Hi Yeikel, 

Neat question! And thanks for the link to the RestClient code; very helpful. 

I don't believe there's a way to configure Kafka Connect to add these 
headers to forwarded requests right now. You may be able to do some kind of 
out-of-band proxy magic to intercept forwarded requests and insert the 
proper headers there. 

I don't see a reason for Kafka Connect to only forward authorization 
headers, even after examining the PR [1] and corresponding Jira ticket [2] 
that altered the RestClient class to begin including authorization headers 
in forwarded REST requests. We may be able to tweak the RestClient to 
include all headers instead of just the authorization header. I know that 
this doesn't help your immediate situation, but if other committers and 
contributors agree that the change would be beneficial, we may be able to 
include it in the next release (which may be 3.7.0, or a patch release for 
3.4, 3.5, or 3.6). Alternatively, we may have to gate such a change behind 
a feature flag (either a coarse-grained boolean that enables/disables 
forwarding of all non-authorization headers, or more fine-grained logic 
such as include/exclude lists or even regexes), which would require a KIP 
and may take longer to release. 

I've CC'd the dev list to gather their perspective on this potential 
change, and to solicit their input on possible workarounds that may be 
useful to you sooner than the next release takes place. 

[1] - https://github.com/apache/kafka/pull/6791 
[2] - https://issues.apache.org/jira/browse/KAFKA-8404 

Cheers, 

Chris 

On Fri, Oct 6, 2023 at 10:14 PM Yeikel Santana  wrote: 

> Hello everyone, 
> 
> I'm currently running Kafka Connect behind a firewall that mandates the 
> inclusion of a specific header. This situation becomes particularly 
> challenging when forwarding requests among multiple workers, as it appears 
> that only the Authorization header is included in the request. 
> 
> I'm wondering if there's a way to customize the headers of Kafka Connect 
> before they are forwarded between workers. From my observations, it seems 
> that this capability may not be available[1], and only the response headers 
> can be customized. 
> 
> I'd appreciate any realistic alternatives or suggestions you may have in 
> mind. 
> 
> Thanks! 
> 
> 
> 
> 
> 
> 
> [1] 
> https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestClient.java#L191-L198

[jira] [Created] (KAFKA-15566) Klaky tests in FetchRequestTest.scala in KRaft mode

2023-10-09 Thread Deng Ziming (Jira)

Deng Ziming created KAFKA-15566:
---

 Summary: Klaky tests in FetchRequestTest.scala in KRaft mode
 Key: KAFKA-15566
 URL: https://issues.apache.org/jira/browse/KAFKA-15566
 Project: Kafka
  Issue Type: Improvement
Reporter: Deng Ziming


|[https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/#showFailuresLink]
[Build / JDK 11 and Scala 2.13 /

kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testFetchWithPartitionsWithIdError(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testFetchWithPartitionsWithIdError_String__quorum_kraft_2/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidation(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidation_String__quorum_kraft_2/]
[Build / JDK 11 and Scala 2.13 / 
kafka.server.FetchRequestTest.testLastFetchedEpochValidationV12(String).quorum=kraft|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14295/4/testReport/junit/kafka.server/FetchRequestTest/Build___JDK_11_and_Scala_2_13___testLastFetchedEpochValidationV12_String__quorum_kraft_2/]|
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



mimaison commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1350023124


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact
+your read committed consumers and prevent compacted 
logs from being compacted. KIP-890 helps
+address hanging transactions by verifying partition 
additions. Part 2 of KIP-890 will optimize
+verification, which currently adds an extra hop.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797:
+Accept duplicate listener on port for IPv4/IPv6: 
Until now, Kafka has not supported
+duplicate listeners on the same port. This works when 
using only a single IP stack, but presents
+an issue if you are working with both IPv4 and IPv6. 
With KIP-797, brokers can be configured
+with listeners that have the same port on different IP 
stacks. This update does not affect
+advertised listeners, which already have this feature.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863:
+Reduce CompletedFetch#parseRecord() memory copy: 
reduces memory allocation an

Re: [DISCUSS] KIP-714: Client metrics and observability

2023-10-09 Thread Andrew Schofield

Hi Matthias,
Good point. Makes sense to me.

Is this something that can also be included in the proposed Kafka Streams 
follow-on KIP, or would you prefer that I add it to KIP-714?
I have a slight preference for the former to put all of the KS enhancements 
into a separate KIP.

Thanks,
Andrew

> On 7 Oct 2023, at 02:12, Matthias J. Sax  wrote:
>
> Thanks Andrew. SGTM.
>
> One point you did not address is the idea to add a method to `KafkaStreams` 
> similar to the proposed `clientInstanceId()` that will be added to 
> consumer/producer/admin clients.
>
> Without addressing this, Kafka Streams users won't have a way to get the 
> assigned `instanceId` of the internally created clients, and thus it would be 
> very difficult for them to know which metrics that the broker receives belong 
> to a Kafka Streams app. It seems they would only find the `instanceIds` in 
> the log4j output if they enable client logging?
>
> Of course, because there is multiple clients inside Kafka Streams, the return 
> type cannot be an single "String", but must be some some complex data 
> structure -- we could either add a new class, or return a Map 
> using a client key that maps to the `instanceId`.
>
> For example we could use the following key:
>
>   [Global]StreamThread[-][-restore][consumer|producer]
>
> (Of course, only the valid combination.)
>
> Or maybe even better, we might want to return a `Future` because collection 
> all the `instanceId` might be a blocking all on each client? I have already a 
> few idea how it could be implemented but I don't think it must be discussed 
> on the KIP, as it's an implementation detail.
>
> Thoughts?
>
>
> -Matthias
>
> On 10/6/23 4:21 AM, Andrew Schofield wrote:
>> Hi Matthias,
>> Thanks for your comments. I agree that a follow-up KIP for Kafka Streams 
>> makes sense. This KIP currently has made a bit
>> of an effort to embrace KS, but it’s not enough by a long way.
>> I have removed `application.id `. This should be 
>> done properly in the follow-up KIP. I don’t believe there’s a downside to
>> removing it from this KIP.
>> I have reworded the statement about temporarily. In practice, the 
>> implementation of this KIP that’s going on while the voting
>> progresses happens to use delta temporality, but that’s an implementation 
>> detail. Supporting clients must support both
>> temporalities.
>> I thought about exposing the client instance ID as a metric, but non-numeric 
>> metrics are not usual practice and tools
>> do not universally support them. I don’t think the KIP is improved by adding 
>> one now.
>> I have also added constants for the various Config classes for 
>> ENABLE_METRICS_PUSH_CONFIG, including to
>> StreamsConfig. It’s best to be explicit about this.
>> Thanks,
>> Andrew
>>> On 2 Oct 2023, at 23:47, Matthias J. Sax  wrote:
>>>
>>> Hi,
>>>
>>> I did not pay attention to this KIP in the past; seems it was on-hold for a 
>>> while.
>>>
>>> Overall it sounds very useful, and I think we should extend this with a 
>>> follow up KIP for Kafka Streams. What is unclear to me at this point is the 
>>> statement:
>>>
 Kafka Streams applications have an application.id configured and this 
 identifier should be included as the application_id metrics label.
>>>
>>> The `application.id` is currently only used as the (main) consumer's 
>>> `group.id` (and is part of an auto-generated `client.id` if the user does 
>>> not set one).
>>>
>>> This comment related to:
>>>
 The following labels should be added by the client as appropriate before 
 metrics are pushed.
>>>
>>> Given that Kafka Streams uses the consumer/producer/admin client as "black 
>>> boxes", a client does at this point not know that it's part of a Kafka 
>>> Streams application, and thus, it won't be able to attach any such label to 
>>> the metrics it sends. (Also producer and admin don't even know the value of 
>>> `application.id` -- only the (main) consumer, indirectly via `group.id`, 
>>> but also restore and global consumer don't know it, because they don't have 
>>> `group.id` set).
>>>
>>> While I am totally in favor of the proposal, I am wondering how we intent 
>>> to implement it in clean way? Or would we do ok to have some internal 
>>> client APIs that KS can use to "register" itself with the client?
>>>
>>>
>>>
 While clients must support both temporalities, the broker will initially 
 only send GetTelemetrySubscriptionsResponse.DeltaTemporality=True
>>>
>>> Not sure if I can follow. How make the decision about DELTA or CUMULATIVE 
>>> metrics? Should the broker side plugin not decide what metrics it what to 
>>> receive in which form? So what does "initially" mean -- the broker won't 
>>> ship with a default plugin implementation?
>>>
>>>
>>>
 The following method is added to the Producer, Consumer, and Admin client 
 interfaces:
>>>
>>> Should we add anything to Kafka Streams to expose the underlying clients' 
>>> assigned client-instance-

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Federico Valeri

Hi Chris,

+1 (non binding)

Thanks
Fede

On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya  wrote:
>
> Hi Chris,
>
> Thanks for the KIP!
> +1 (binding)
>
> Yash
>
> On Fri, Oct 6, 2023 at 9:54 PM Greg Harris 
> wrote:
>
> > Hey Chris,
> >
> > Thanks for the KIP!
> > I think that preserving the ephemeral nature of the logging change is
> > the right choice here, and using the config topic for intra-cluster
> > broadcast is better than REST forwarding.
> >
> > +1 (binding)
> >
> > Thanks,
> > Greg
> >
> > On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-976, which augments the existing
> > dynamic
> > > logger adjustment REST API for Kafka Connect to apply changes
> > cluster-wide
> > > instead on a per-worker basis.
> > >
> > > The KIP:
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect
> > >
> > > The discussion thread:
> > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5
> > >
> > > Cheers,
> > >
> > > Chris
> >

Re:CodeStyle Configuration

2023-10-09 Thread Taras Ledkov

Hi, Arpit.

Great question. I cannot see any reason why community style and inspections 
settings not pushed to `.idea/` folder at the repository..  

-- 
With best regards
Taras Ledkov

Re:[DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-10-09 Thread Taras Ledkov

Hi, Kafka team.

1. Ping to review KIP.
2. I dare say that the low activity in the discussion of KIP-967 means that 
KIP-967 is ready for voting?

-- 
With best regards
Taras Ledkov

[jira] [Resolved] (KAFKA-15521) Refactor build.gradle to align gradle swagger plugin with swagger dependencies

2023-10-09 Thread Mickael Maison (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15521.

Fix Version/s: 3.7.0
   Resolution: Fixed

> Refactor build.gradle to align gradle swagger plugin with swagger dependencies
> --
>
> Key: KAFKA-15521
> URL: https://issues.apache.org/jira/browse/KAFKA-15521
> Project: Kafka
>  Issue Type: Improvement
>  Components: build
>Reporter: Mickael Maison
>Assignee: Atul Sharma
>Priority: Major
> Fix For: 3.7.0
>
>
> We use both the Swagger Gradle plugin 
> "io.swagger.core.v3.swagger-gradle-plugin" and 2 Swagger dependencies 
> swaggerAnnotations and swaggerJaxrs2. The version for the Gradle plugin is in 
> build.gradle while the version for the dependency is in 
> gradle/dependencies.gradle.
> When we upgrade the version of one or the other it sometimes cause build 
> breakages, for example https://github.com/apache/kafka/pull/13387 and 
> https://github.com/apache/kafka/pull/14464
> We should try to have the version defined in a single place to avoid breaking 
> the build again.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Mickael Maison

Hi Chris,

+1 (binding)

Thanks for the KIP!

On Mon, Oct 9, 2023 at 11:16 AM Federico Valeri  wrote:
>
> Hi Chris,
>
> +1 (non binding)
>
> Thanks
> Fede
>
> On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya  wrote:
> >
> > Hi Chris,
> >
> > Thanks for the KIP!
> > +1 (binding)
> >
> > Yash
> >
> > On Fri, Oct 6, 2023 at 9:54 PM Greg Harris 
> > wrote:
> >
> > > Hey Chris,
> > >
> > > Thanks for the KIP!
> > > I think that preserving the ephemeral nature of the logging change is
> > > the right choice here, and using the config topic for intra-cluster
> > > broadcast is better than REST forwarding.
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton 
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-976, which augments the existing
> > > dynamic
> > > > logger adjustment REST API for Kafka Connect to apply changes
> > > cluster-wide
> > > > instead on a per-worker basis.
> > > >
> > > > The KIP:
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect
> > > >
> > > > The discussion thread:
> > > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2266

2023-10-09 Thread Apache Jenkins Server

See

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-09 Thread Chris Egerton

Thanks for the KIP, Yash!

+1 (binding)

On Mon, Oct 9, 2023, 01:12 Yash Mayya  wrote:

> Hi all,
>
> I'd like to start a vote on KIP-980 which proposes allowing the creation of
> connectors in a stopped (or paused) state.
>
> KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
>
> Discussion Thread -
> https://lists.apache.org/thread/om803vl191ysf711qm7czv94285rtt5d
>
> Thanks,
> Yash
>

Re: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread hudeqi

Hi Chris,

+1 (non-binding)

Finally, there is no need to use external intrusion tools to change the log 
level of kafka connect online! Thanks for the KIP!

best,
hudeqi

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2267

2023-10-09 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-15567) ReplicaFetcherThreadBenchmark is not working

2023-10-09 Thread Haruki Okada (Jira)

Haruki Okada created KAFKA-15567:


 Summary: ReplicaFetcherThreadBenchmark is not working
 Key: KAFKA-15567
 URL: https://issues.apache.org/jira/browse/KAFKA-15567
 Project: Kafka
  Issue Type: Improvement
Reporter: Haruki Okada
Assignee: Haruki Okada


* ReplicaFetcherThreadBenchmark is not working as of current trunk 
(https://github.com/apache/kafka/tree/c223a9c3761f796468ccfdae9e177e764ab6a965)

 
{code:java}
% jmh-benchmarks/jmh.sh ReplicaFetcherThreadBenchmark
(snip)
java.lang.NullPointerException
    at kafka.server.metadata.ZkMetadataCache.(ZkMetadataCache.scala:89)
    at kafka.server.MetadataCache.zkMetadataCache(MetadataCache.scala:120)
    at 
org.apache.kafka.jmh.fetcher.ReplicaFetcherThreadBenchmark.setup(ReplicaFetcherThreadBenchmark.java:220)
    at 
org.apache.kafka.jmh.fetcher.jmh_generated.ReplicaFetcherThreadBenchmark_testFetcher_jmhTest._jmh_tryInit_f_replicafetcherthreadbenchmark0_G(ReplicaFetcherThreadBenchmark_testFetcher_jmhTest.java:448)
    at 
org.apache.kafka.jmh.fetcher.jmh_generated.ReplicaFetcherThreadBenchmark_testFetcher_jmhTest.testFetcher_AverageTime(ReplicaFetcherThreadBenchmark_testFetcher_jmhTest.java:164)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
    at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:527)
    at 
org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:504)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829) {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15568) Use incrementalAlterConfigs to update the dynamic config of broker in ConfigCommand tool

2023-10-09 Thread Aman Singh (Jira)

Aman Singh created KAFKA-15568:
--

 Summary: Use incrementalAlterConfigs to update the dynamic config 
of broker in ConfigCommand tool
 Key: KAFKA-15568
 URL: https://issues.apache.org/jira/browse/KAFKA-15568
 Project: Kafka
  Issue Type: Improvement
Reporter: Aman Singh






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] Added 3.6 documentation [kafka-site]

2023-10-09 Thread via GitHub



C0urante commented on PR #541:
URL: https://github.com/apache/kafka-site/pull/541#issuecomment-1753214137

   @satishd the version in the Javadocs here is 3.6.0-SNAPSHOT. Do you plan to 
address this before announcing the 3.6.0 release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE]KIP-966: Eligible Leader Replicas

2023-10-09 Thread Calvin Liu

Hi Colin,
Thanks for the feedback. I have updated the KIP but with the following
changes.
--The request is still grouped by topics. For each topic, the caller can
specify either partition IDs or a range of partitions to query. If it is a
range query, the request should specify the "first partition id", then the
partition larger or equal to the id will be returned.
--The response is still grouped by topics. When quota limit reached:
 If it is a range request, the "next partition id" will be specified
within the topic when a partial of partitions can be returned. When the
whole topic can't be returned, the topic will have the
error REQUEST_LIMIT_REACHED.
 If it is a partition-specific topic, all the partition can't be
returned will have error REQUEST_LIMIT_REACHED
Note, that the request can have partition-specific and range-request topic
mixed.

On Fri, Oct 6, 2023 at 4:30 PM Colin McCabe  wrote:

> Hi Calvin,
>
> Thanks for the KIP. I think the config discussion was good and I have no
> more comments there.
>
> I have one last thing I think we should fix up:
>
> I think we should improve DescribeTopicRequest. The current mechanism of
> "you can only list 20 topics" doesn't do a very good job of limiting the
> results. After all, if those topics only have 1 partition each, this means
> a pretty small RPC. If they have 10,000 partitions each, then it's a very
> large RPC.
>
> I think a better mechanism would be:
> 1. Have the request be a list of (topic_name, partition_id) pairs plus a
> (first_topic_name, first_partition_id) pair.
> (for the initial request, first_topic_name="" and first_partition_id=-1,
> of course)
> (if partition_id = -1 then we should list all partitions for the topic)
>
> 2. When returning results, sort everything alphabetically and return the
> first 1000, plus a (next_topic, next_partition_id) pair. (if there is
> nothing more to return, next_topic = null.)
>
> With those changes I would be +1
>
> best,
> Colin
>
>
> If the response wasn't long enough, the caller can set
> On Wed, Oct 4, 2023, at 17:44, Jun Rao wrote:
> > Hi, Calvin,
> >
> > Thanks for the KIP. +1 from me too.
> >
> > Jun
> >
> > On Wed, Sep 20, 2023 at 5:28 PM Justine Olshan
> 
> > wrote:
> >
> >> Thanks Calvin.
> >> I think this will be very helpful going forward to minimize data loss.
> >>
> >> +1 from me (binding)
> >>
> >> Justine
> >>
> >> On Wed, Sep 20, 2023 at 3:42 PM Calvin Liu 
> >> wrote:
> >>
> >> > Hi all,
> >> > I'd like to call for a vote on KIP-966 which includes a series of
> >> > enhancements to the current ISR model.
> >> >
> >> >- Introduce the new HWM advancement requirement which enables the
> >> system
> >> >to have more potentially data-safe replicas.
> >> >- Introduce Eligible Leader Replicas(ELR) to represent the above
> >> >data-safe replicas.
> >> >- Introduce Unclean Recovery process which will deterministically
> >> choose
> >> >the best replica during an unclean leader election.
> >> >
> >> >
> >> > KIP:
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> >> >
> >> > Discussion thread:
> >> > https://lists.apache.org/thread/gpbpx9kpd7c62dm962h6kww0ghgznb38
> >> >
> >>
>

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

2023-10-09 Thread Greg Harris

Hey all,

I think it may be helpful to discuss the KIP draft more informally on
the ASF slack #kafka channel.
I've invited those of you who responded above, and for anyone else who
wants an invite please just let me know on this thread.

Thanks!
Greg

On Mon, Oct 2, 2023 at 1:39 PM Greg Harris  wrote:
>
> Hi everyone,
>
> Following up on this thread, I've opened a draft KIP for everyone to
> contribute to:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication
> https://lists.apache.org/thread/4563fqdcwlqwjqlm7mbj8wftx1rpo0yd
>
> Please consider yourselves owners of this feature, and make edits to
> the KIP directly when you feel it is appropriate.
>
> Thanks!
> Greg Harris
>
> On Mon, May 22, 2023 at 4:59 AM Andrew Otto  wrote:
> >
> > Hello!
> >
> > The Wikimedia Foundation would be very interested in this feature.  It
> > would make deployment of multi DC/region streaming apps much simpler.  We'd
> > like to investigate more the feasibility of using a Kafka stretch cluster
> > to solve
> > this problem, but something built into Kafka itself that still allows for
> > distinct Kafka clusters would be really great!
> >
> > -Andrew Otto
> >
> > On Sat, May 20, 2023 at 7:42 AM hudeqi <16120...@bjtu.edu.cn> wrote:
> >
> > > Hi, I am also very excited to see this discussion, because I also
> > > implemented the "federation model" based on the kafka-0.10.2.1 version
> > > before in company and got benefits from going online. It solves the 
> > > problem
> > > of seamlessly migrating the bytesIn/bytesOut of a topic to another kafka
> > > cluster without the client being aware. Therefore, it includes the
> > > cross-cluster synchronous topic data mentioned in this discussion, and
> > > ensures the consistency of the offset. It also provides the global
> > > coordinator service to ensure the consistency of the group offset, and the
> > > metadata service to ensure that the client is visible of the real traffic
> > > cluster of the topic. So I am very excited to see that the community has
> > > finally discussed this feature, and I especially hope to participate in 
> > > the
> > > development and reviewer of this feature as a co-author.
> > >
> > >
> > > > -原始邮件-
> > > > 发件人: "Chia-Ping Tsai" 
> > > > 发送时间: 2023-05-20 18:57:31 (星期六)
> > > > 收件人: dev@kafka.apache.org
> > > > 抄送: t...@conduktor.io.invalid
> > > > 主题: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call
> > > for Interest & Co-Authors
> > > >
> > >

[jira] [Resolved] (KAFKA-15278) Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC

2023-10-09 Thread Philip Nee (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Nee resolved KAFKA-15278.

Resolution: Fixed

> Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC
> 
>
> Key: KAFKA-15278
> URL: https://issues.apache.org/jira/browse/KAFKA-15278
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Philip Nee
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848, 
> kip-848-client-support, kip-848-e2e, kip-848-preview
>
> The necessary Java code that represents the {{ConsumerGroupHeartbeatRequest}} 
> and {{ConsumerGroupHeartbeatResponse}} are already present in the codebase. 
> It is assumed that the scaffolding for the other two will come along in time.
>  * Implement {{ConsumerGroupRequestManager}}
>  * Ensure that {{DefaultBackgroundThread}} correctly calculates I/O timeouts 
> so that the heartbeat occurs within the {{group.consumer.session.timeout.ms}} 
> interval regardless of other {{RequestManager}} instance activity
>  * Ensure error is handled correctly
>  * Ensure MembershipStateManager is updated on both successful and failures 
> cases, and the state machine is transioned to the correct state.
> This task is part of the work to implement support for the new KIP-848 
> consumer group protocol.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-10-09 Thread Colin McCabe

On Fri, Oct 6, 2023, at 18:30, Igor Soarez wrote:
> Hi Colin,
>
>> I would call #2 LOST. It was assigned in the past, but we don't know where.
>> I see that you called this OFFLINE). This is not really normal...
>> it should happen only when we're migrating from ZK mode to KRaft mode,
>> or going from an older KRaft release with multiple directories to a
>> post-JBOD release.
>
> What you refer to as #2 LOST is actually what I named SELECTED,
> as in: a directory has already been _selected_ sometime before,
> we just don't know which one yet.
>
> In the mean time this change has already been merged, but let me know
> if you feel strongly about the naming here as I'm happy to rename
> SELECTED_DIR to LOST_DIR in a new PR.
> https://github.com/apache/kafka/pull/14291
>
>> As for the third state -- I'm not sure why SELECTED_DIR needs to exist.
>
> The third state (actually it is ordered second) - OFFLINE_DIR - conveys
> that a replica is assigned to an unspecified offline directory.
>
> This can be used by the broker in the following way:
>
>   * When catching up with metadata, seeing that one of it's partitions
>   is mapped to SELECTED_DIR, and it cannot find that partition in
>   any of the online log directories, and at least one log dir is offline,
>   then the broker sends AssignReplicasToDirs to converge the assignment
>   to OFFLINE_DIR
>
>   * If a log directory failure happens during an intra-broker (across dir)
>   replica movement, after sending AssignReplicasToDirs with the new UUID,
>   and before the future replica catches up again. (there's a section
>   in the KIP about this).
>
> We could just use a random UUID, as if a replica is assigned to a dir
> that is not in the broker's registration online dirs set then it is
> considered offline by controllers and metadata cache, but using a
> reserved UUID feels cleaner.
>

Hi Igor,

Thanks. I remember the third case now. Basically "unassigned" can transition 
either to the actual assigned directory, or to a special reserved directory ID 
that indicates that the broker can't find it. Maybe this is the one we should 
be calling "lost" :)

How do you feel about the following names for special directory IDs?

MIGRATING : during ZK migration or during migration from an older KRaft 
metadata version, ALL directory IDs get set to this initially. The expectation 
is that the replica exists somewhere on the given broker, but due to migration 
we don't know where ... yet.

UNASSIGNED : the replica was just created. Due to the fact that we're in JBOD 
mode and we want the broker itself to choose the directory, we set the 
directory ID to this. (If the broker only has a single active directory, the 
controller just sets that directory ID initially, and skips UNASSIGNED.)

LOST : a replica was in MIGRATING state, but the broker can't find it anywhere. 
The broker then sets it to LOST.

MIGRATING and LOST only get used for migrations; UNASSIGNED Is the common one. 
For simplicity we can set MIGRATING to the all-zeros UUID. Since UUID fields in 
records will get this by default.

>> I think we need a general mechanism for checking that replicas are
>> in the directories we expect and sending an RPC to the controller
>> if they are not. A mechanism like this will automatically get rid
>> of the LOST replicas just as part of normal operation -- nothing
>> special required.
>
> Thanks for pointing this out, I forgot to put in the notes in my
> previous email that we discussed this too.
>
> The KIP proposes this is done when catching up with metadata,
> but you also suggested we extend the stray replica detection
> mechanism to also check for these inconsistencies. I think
> this is a good idea, and we'll look into that as well.
>

Yes, I think we are on the same page here.

What I was proposing was that there is some piece of code that handles 
reconciling the controller's view of where replicas are with the broker's view. 
Then we could have the broker wait until that code is done with its work before 
unfencing. It's not too different from how we wait for metadata to be caught up 
before requesting unfencing.

Probably the messy thing is handling the interaction between this code and 
ReplicaManager. Maybe it would work best if the interaction was sort of 
one-way: if ReplicaManager sees a discrepancy, it asks this new manager code to 
correct it. After all, we don't want to have ReplicaManager responsible for 
sending RPCs to the controller. It already has enough to do! And there are 
issues like handling retries and so on.

With regard to the failure detection "gap" during hybrid mode: the kraft 
controller sends a full LeaderAndIsrRequest to the brokers that are in hybrid 
mode, right? And there is a per-partition response as well. Right now, we don't 
pay attention to the error codes sent back in the response. But we could. Any 
replica with an error could be transitioned from MIGRATING -> LOST, right? That 
would close the failure detection gap.

best,
Colin

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-09 Thread Knowles Atchison Jr

This is super useful for pipeline setup!

+1 (non binding)

On Mon, Oct 9, 2023, 7:57 AM Chris Egerton  wrote:

> Thanks for the KIP, Yash!
>
> +1 (binding)
>
> On Mon, Oct 9, 2023, 01:12 Yash Mayya  wrote:
>
> > Hi all,
> >
> > I'd like to start a vote on KIP-980 which proposes allowing the creation
> of
> > connectors in a stopped (or paused) state.
> >
> > KIP -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
> >
> > Discussion Thread -
> > https://lists.apache.org/thread/om803vl191ysf711qm7czv94285rtt5d
> >
> > Thanks,
> > Yash
> >
>

Apache Kafka 3.7.0 Release

2023-10-09 Thread Stanislav Kozlovski

Hey all!

I would like to volunteer to be the release manager driving the next
release - Apache Kafka *3.7.0*.

If there are no objections, I will start and share a release plan soon
enough!

Cheers,
Stanislav

Re: Apache Kafka 3.7.0 Release

2023-10-09 Thread Ismael Juma

Thanks for volunteering Stanislav!

Ismael

On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
 wrote:

> Hey all!
>
> I would like to volunteer to be the release manager driving the next
> release - Apache Kafka *3.7.0*.
>
> If there are no objections, I will start and share a release plan soon
> enough!
>
> Cheers,
> Stanislav
>

Re: Apache Kafka 3.7.0 Release

2023-10-09 Thread Bill Bejeck

+1

Thanks, Stanislav!

-Bill

On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma  wrote:

> Thanks for volunteering Stanislav!
>
> Ismael
>
> On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
>  wrote:
>
> > Hey all!
> >
> > I would like to volunteer to be the release manager driving the next
> > release - Apache Kafka *3.7.0*.
> >
> > If there are no objections, I will start and share a release plan soon
> > enough!
> >
> > Cheers,
> > Stanislav
> >
>

Re: Apache Kafka 3.7.0 Release

2023-10-09 Thread Chris Egerton

+1, thanks Stanislav!

On Mon, Oct 9, 2023, 14:02 Bill Bejeck  wrote:

> +1
>
> Thanks, Stanislav!
>
> -Bill
>
> On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma  wrote:
>
> > Thanks for volunteering Stanislav!
> >
> > Ismael
> >
> > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
> >  wrote:
> >
> > > Hey all!
> > >
> > > I would like to volunteer to be the release manager driving the next
> > > release - Apache Kafka *3.7.0*.
> > >
> > > If there are no objections, I will start and share a release plan soon
> > > enough!
> > >
> > > Cheers,
> > > Stanislav
> > >
> >
>

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-09 Thread Greg Harris

Thanks Yash for the well written KIP!

And thank you for finally adding JSON support to the standalone mode
that isn't file extension sensitive. That will be very useful.

+1 (binding)

On Mon, Oct 9, 2023 at 10:45 AM Knowles Atchison Jr
 wrote:
>
> This is super useful for pipeline setup!
>
> +1 (non binding)
>
> On Mon, Oct 9, 2023, 7:57 AM Chris Egerton  wrote:
>
> > Thanks for the KIP, Yash!
> >
> > +1 (binding)
> >
> > On Mon, Oct 9, 2023, 01:12 Yash Mayya  wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a vote on KIP-980 which proposes allowing the creation
> > of
> > > connectors in a stopped (or paused) state.
> > >
> > > KIP -
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
> > >
> > > Discussion Thread -
> > > https://lists.apache.org/thread/om803vl191ysf711qm7czv94285rtt5d
> > >
> > > Thanks,
> > > Yash
> > >
> >

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Hector Geraldino (BLOOMBERG/ 919 3RD A)

Good stuff, +1 (non-binding) from me as well

De: dev@kafka.apache.org A: 10/09/23 05:16:06 UTC-4:00A:  dev@kafka.apache.org
Subject: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka 
Connect

Hi Chris,

+1 (non binding)

Thanks
Fede

On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya  wrote:
>
> Hi Chris,
>
> Thanks for the KIP!
> +1 (binding)
>
> Yash
>
> On Fri, Oct 6, 2023 at 9:54 PM Greg Harris 
> wrote:
>
> > Hey Chris,
> >
> > Thanks for the KIP!
> > I think that preserving the ephemeral nature of the logging change is
> > the right choice here, and using the config topic for intra-cluster
> > broadcast is better than REST forwarding.
> >
> > +1 (binding)
> >
> > Thanks,
> > Greg
> >
> > On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-976, which augments the existing
> > dynamic
> > > logger adjustment REST API for Kafka Connect to apply changes
> > cluster-wide
> > > instead on a per-worker basis.
> > >
> > > The KIP:
> > >
> > 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+
log+adjustment+for+Kafka+Connect
> > >
> > > The discussion thread:
> > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5
> > >
> > > Cheers,
> > >
> > > Chris
> >

Re: Apache Kafka 3.7.0 Release

2023-10-09 Thread Josep Prat

Thanks Stanislav!

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Oct 9, 2023, 20:05 Chris Egerton  wrote:

> +1, thanks Stanislav!
>
> On Mon, Oct 9, 2023, 14:02 Bill Bejeck  wrote:
>
> > +1
> >
> > Thanks, Stanislav!
> >
> > -Bill
> >
> > On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma  wrote:
> >
> > > Thanks for volunteering Stanislav!
> > >
> > > Ismael
> > >
> > > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
> > >  wrote:
> > >
> > > > Hey all!
> > > >
> > > > I would like to volunteer to be the release manager driving the next
> > > > release - Apache Kafka *3.7.0*.
> > > >
> > > > If there are no objections, I will start and share a release plan
> soon
> > > > enough!
> > > >
> > > > Cheers,
> > > > Stanislav
> > > >
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2268

2023-10-09 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-15569) Update test and add test cases in IQv2StoreIntegrationTest

2023-10-09 Thread Hanyu Zheng (Jira)

Hanyu Zheng created KAFKA-15569:
---

 Summary: Update test and add test cases in IQv2StoreIntegrationTest
 Key: KAFKA-15569
 URL: https://issues.apache.org/jira/browse/KAFKA-15569
 Project: Kafka
  Issue Type: Improvement
Reporter: Hanyu Zheng


Update test and add test cases in IQv2StoreIntegrationTest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

2023-10-09 Thread Николай Ижиков

Hello. I want to participate in discussion.

> 9 окт. 2023 г., в 20:34, Greg Harris  
> написал(а):
> 
> Hey all,
> 
> I think it may be helpful to discuss the KIP draft more informally on
> the ASF slack #kafka channel.
> I've invited those of you who responded above, and for anyone else who
> wants an invite please just let me know on this thread.
> 
> Thanks!
> Greg
> 
> On Mon, Oct 2, 2023 at 1:39 PM Greg Harris  wrote:
>> 
>> Hi everyone,
>> 
>> Following up on this thread, I've opened a draft KIP for everyone to
>> contribute to:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-986%3A+Cross-Cluster+Replication
>> https://lists.apache.org/thread/4563fqdcwlqwjqlm7mbj8wftx1rpo0yd
>> 
>> Please consider yourselves owners of this feature, and make edits to
>> the KIP directly when you feel it is appropriate.
>> 
>> Thanks!
>> Greg Harris
>> 
>> On Mon, May 22, 2023 at 4:59 AM Andrew Otto  wrote:
>>> 
>>> Hello!
>>> 
>>> The Wikimedia Foundation would be very interested in this feature.  It
>>> would make deployment of multi DC/region streaming apps much simpler.  We'd
>>> like to investigate more the feasibility of using a Kafka stretch cluster
>>> to solve
>>> this problem, but something built into Kafka itself that still allows for
>>> distinct Kafka clusters would be really great!
>>> 
>>> -Andrew Otto
>>> 
>>> On Sat, May 20, 2023 at 7:42 AM hudeqi <16120...@bjtu.edu.cn> wrote:
>>> 
 Hi, I am also very excited to see this discussion, because I also
 implemented the "federation model" based on the kafka-0.10.2.1 version
 before in company and got benefits from going online. It solves the problem
 of seamlessly migrating the bytesIn/bytesOut of a topic to another kafka
 cluster without the client being aware. Therefore, it includes the
 cross-cluster synchronous topic data mentioned in this discussion, and
 ensures the consistency of the offset. It also provides the global
 coordinator service to ensure the consistency of the group offset, and the
 metadata service to ensure that the client is visible of the real traffic
 cluster of the topic. So I am very excited to see that the community has
 finally discussed this feature, and I especially hope to participate in the
 development and reviewer of this feature as a co-author.
 
 
 > -原始邮件-
 > 发件人: "Chia-Ping Tsai" 
 > 发送时间: 2023-05-20 18:57:31 (星期六)
 > 收件人: dev@kafka.apache.org
 > 抄送: t...@conduktor.io.invalid
 > 主题: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call
 for Interest & Co-Authors
 >

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



gharris1727 commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1350759025


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact
+your read committed consumers and prevent compacted 
logs from being compacted. KIP-890 helps
+address hanging transactions by verifying partition 
additions. Part 2 of KIP-890 will optimize
+verification, which currently adds an extra hop.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797:
+Accept duplicate listener on port for IPv4/IPv6: 
Until now, Kafka has not supported
+duplicate listeners on the same port. This works when 
using only a single IP stack, but presents
+an issue if you are working with both IPv4 and IPv6. 
With KIP-797, brokers can be configured
+with listeners that have the same port on different IP 
stacks. This update does not affect
+advertised listeners, which already have this feature.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863:
+Reduce CompletedFetch#parseRecord() memory copy: 
reduces memory allocation

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



gharris1727 commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1350763659


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact
+your read committed consumers and prevent compacted 
logs from being compacted. KIP-890 helps
+address hanging transactions by verifying partition 
additions. Part 2 of KIP-890 will optimize
+verification, which currently adds an extra hop.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797:
+Accept duplicate listener on port for IPv4/IPv6: 
Until now, Kafka has not supported
+duplicate listeners on the same port. This works when 
using only a single IP stack, but presents
+an issue if you are working with both IPv4 and IPv6. 
With KIP-797, brokers can be configured
+with listeners that have the same port on different IP 
stacks. This update does not affect
+advertised listeners, which already have this feature.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863:
+Reduce CompletedFetch#parseRecord() memory copy: 
reduces memory allocation

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-09 Thread Hanyu (Peter) Zheng

After our discussion, we discovered something intriguing. The definitions
for the range and reverseRange methods in the ReadOnlyKeyValueStore are as
follows:
/**
 * Get an iterator over a given range of keys. This iterator must be
closed after use.
 * The returned iterator must be safe from {@link
java.util.ConcurrentModificationException}s
 * and must not return null values.
 ** Order is not guaranteed as bytes lexicographical ordering might not
represent key order.*
 *
 * @param from The first key that could be in the range, where
iteration starts from.
 * A null value indicates that the range starts with the
first element in the store.
 * @param to   The last key that could be in the range, where iteration
ends.
 * A null value indicates that the range ends with the last
element in the store.
 * @return The iterator for this range, from smallest to largest bytes.
 * @throws InvalidStateStoreException if the store is not initialized
 */
KeyValueIterator range(K from, K to);

/**
 * Get a reverse iterator over a given range of keys. This iterator
must be closed after use.
 * The returned iterator must be safe from {@link
java.util.ConcurrentModificationException}s
 * and must not return null values.
 * *Order is not guaranteed as bytes lexicographical ordering might not
represent key order.*
 *
 * @param from The first key that could be in the range, where
iteration ends.
 * A null value indicates that the range starts with the
first element in the store.
 * @param to   The last key that could be in the range, where iteration
starts from.
 * A null value indicates that the range ends with the last
element in the store.
 * @return The reverse iterator for this range, from largest to
smallest key bytes.
 * @throws InvalidStateStoreException if the store is not initialized
 */
default KeyValueIterator reverseRange(K from, K to) {
throw new UnsupportedOperationException();
}

The query methods of RangeQuery ultimately invoke either the range method
or the reverseRange method. However, as per the JavaDoc: the order is not
guaranteed, since byte lexicographical ordering may not correspond to the
actual key order.

Sincerely,
Hanyu

On Fri, Oct 6, 2023 at 10:00 AM Hanyu (Peter) Zheng 
wrote:

> Thank you, Matthias, for the detailed implementation and explanation. As
> of now, our capability is limited to executing interactive queries on
> individual partitions. To illustrate:
>
> Consider the IQv2StoreIntegrationTest:
>
> We have two partitions:
> Partition0 contains key-value pairs: <0,0> and <2,2>.
> Partition1 contains key-value pairs: <1,1> and <3,3>.
> When executing RangeQuery.withRange(1,3), the results are:
>
> Partition0: [2]
> Partition1: [1, 3]
> To support functionalities like reverseRange and reverseAll, we can
> introduce the withDescendingKeys() method. For instance, using
> RangeQuery.withRange(1,3).withDescendingKeys(), the anticipated results are:
>
> Partition0: [2]
> Partition1: [3, 1]
>
> In response to Hao's inquiry about the boundary issue, please refer to the
> StoreQueryUtils class. The code snippet:
>
> iterator = kvStore.range(lowerRange.orElse(null), upperRange.orElse(null));
> indicates that when implementing range in each store, it's structured like:
>
> @Override
> public KeyValueIterator range(final Bytes from, final Bytes
> to) {
> if (from != null && to != null && from.compareTo(to) > 0) {
> This section performs the necessary checks.
>
> Sincerely,
> Hanyu
>
> On Thu, Oct 5, 2023 at 9:52 AM Hanyu (Peter) Zheng 
> wrote:
>
>> Hi, Hao,
>>
>> In this case, it will return an empty set or list in the end.
>>
>> Sincerely,
>> Hanyu
>>
>> On Wed, Oct 4, 2023 at 10:29 PM Matthias J. Sax  wrote:
>>
>>> Great discussion!
>>>
>>> It seems the only open question might be about ordering guarantees?
>>> IIRC, we had a discussion about this in the past.
>>>
>>>
>>> Technically (at least from my POV), existing `RangeQuery` does not have
>>> a guarantee that data is return in any specific order (not even on a per
>>> partitions bases). It just happens that RocksDB (and as pointed out by
>>> Hanyu already, also the built-in in-memory store that is base on a
>>> tree-map) allows us to return data ordered by key; as mentioned already,
>>> this guarantee is limited on a per partition basis.
>>>
>>> If there would be custom store base on a hashed key-value store, this
>>> store could implement RangeQuery and return data (even for a single
>>> partition) with no ordering, without violating the contract.
>>>
>>>
>>>
>>> Thus, it could actually make sense, to extend `RangeQuery` and allow
>>> three options: no-order, ascending, descending. For our existing
>>> Rocks/InMemory implementations, no-order could be equal to ascending and
>>> nothing changes effectively, but it might be a better API contract? --
>>> If we assume that there

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



jolshan commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1350841290


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact

Review Comment:
   We should add a note that we missed the fix for offset commits. That will be 
addressed in the next release.
   Something like. 
   
   In 3.6.0, transaction verification will prevent hanging transactions on data 
partitions. In the next release, transactional offset commits will also be 
covered.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-960: Support single-key_single-timestamp interactive queries (IQv2) for versioned state stores

2023-10-09 Thread Matthias J. Sax

One more nit: as discussed on the related KIP-698 thread, we should not 
use `get` as prefix for the getters.


So it should be `K key()` and `Optional asOfTimestamp()`.


Otherwise the KIP LGTM.


+1 (binding)


-Matthias

On 10/6/23 2:50 AM, Alieh Saeedi wrote:

Hi everyone,

Since KIP-960 is reduced to the simplest IQ type and all further comments
are related to the following-up KIPs, I decided to finalize it at this
point.


A huge thank you to everyone who has reviewed this KIP (and also the
following-up ones), and
participated in the discussion thread!

I'd also like to thank you in advance for taking the time to vote.

Best,
Alieh

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

2023-10-09 Thread Matthias J. Sax

Bruno and I had some background conversation about the `get` prefix 
question including a few other committers.


The official policy was never changed, and we should not add the 
`get`-prefix. It's a slip on our side in previous KIPs to add the 
`get`-prefix and we should actually clean it up doing a follow up KIP.



-Matthias


On 10/5/23 5:26 AM, Bruno Cadonna wrote:

Hi Matthias,

Given all the IQv2 KIPs that use getX and given recent PRs (internal 
interfaces mainly) that got merged, I was under the impression that we 
moved away from the strict no-getX policy.


I do not think it was an accident using getX in the IQv2 KIPs since 
somebody would have brought it up, otherwise.


I am fine with both types of getters.

If we think, we need to discuss this in a broader context, let's start a 
separate thread.



Best,
Bruno





On 10/5/23 7:44 AM, Matthias J. Sax wrote:

I agree to (almost) everything what Bruno said.


In general, we tend to move away from using getters without "get", 
recently. So I would keep the "get".


This is new to me? Can you elaborate on this point? Why do you think 
that's the case?


I actually did realize (after Walker mentioned it) that existing query 
types use `get` prefix, but to me it seems that it was by accident and 
we should consider correcting it? Thus, I would actually prefer to not 
add the `get` prefix for new methods query types.


IMHO, we should do a follow up KIP to deprecate all methods with `get` 
prefix and replace them with new ones without `get` -- it's of course 
always kinda "unnecessary" noise, but if we don't do it, we might get 
into more and more inconsistent naming what would result in a "bad" API.


If we indeed want to change the convention and use the `get` prefix, I 
would strongly advocate to bit the bullet and do KIP to pro-actively 
add the `get` "everywhere" it's missing... But overall, it seems to be 
a much broader decision, and we should get buy in from many committers 
about it -- as long as there is no broad consensus to add `get` 
everywhere, I would strongly prefer not to diverge from the current 
agreement to omit `get`.




-Matthias




On 10/4/23 2:36 AM, Bruno Cadonna wrote:

Hi,

Regarding tombstones:
As far as I understand, we need to add either a validTo field to 
VersionedRecord or we need to return tombstones, otherwise the result 
is not complete, because users could never know a record was deleted 
at some point before the second non-null value was put.
I like more adding the validTo field since it makes the result more 
concise and easier interpretable.


Extending on Victoria's example, with the following puts

put(k, v1, time=0)
put(k, null, time=5)
put(k, null, time=10)
put(k, null, time=15)
put(k, v2, time=20)

the result with tombstones would be

value, timestamp
(v1, 0)
(null, 5)
(null, 10)
(null, 15)
(v2, 20)

instead of

value, timestamp, validTo
(v1, 0, 5)
(v2, 20, null)

The benefit of conciseness would already apply to one single tombstone.

On the other hand, why would somebody write consecutive tombstones 
into a versioned state store? I guess if somebody does that on 
purpose, then there should be a way to retrieve each of those 
tombstones, right?
So maybe we need both -- validTo field and the option to return 
tombstones. The latter might be moved to a future KIP in case we see 
the need.



Regarding .within(fromTs, toTs):
I would keep it simple with .from() and .asOfTimestamp() (or 
.until()). If we go with .within(), I would opt for 
.withinTimeRange(fromTs, toTs), because the query becomes more readable:


MultiVersionedKeyQuery
   .withKey(1)
   .withinTimeRange(Instant.parse(2023-08-03T10:37:30.00Z), 
Instant.parse(2023-08-04T10:37:30.00Z))


If we stay with .from() and .until(), we should consider .fromTime() 
and .untilTime() (or .toTime()):


MultiVersionedKeyQuery
  .withKey(1)
  .fromTime(Instant.parse(2023-08-03T10:37:30.00Z))
  .untilTime(Instant.parse(2023-08-04T10:37:30.00Z))



Regarding asOf vs. until:
I think asOf() is more used in point in time queries as Walker 
mentioned where this KIP specifies a time range. IMO asOf() fits very 
well with KIP-960 where one version is queried, but here I think 
.until() fits better. That might just be a matter of taste and in the 
end I am fine with both as long as it is well documented.



Regarding getters without "get":
In the other IQv2 classes we used getters with "get". In general, we 
tend to move away from using getters without "get", recently. So I 
would keep the "get".



Best,
Bruno

On 10/3/23 7:49 PM, Walker Carlson wrote:

Hey Alieh thanks for the KIP,

Weighing in on the AsOf vs Until debate I think either is fine from a
natural language perspective. Personally AsOf makes more sense to me 
where
until gives me the idea that the query is making a change. It's 
totally a

connotative difference and not that important. I think as of is pretty
frequently used in point of time queries.

Also for these methods it makes sense to drop the

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-09 Thread Colt McNealy

Hanyu,

I like the attention to detail!

It is correct that the JavaDoc does not "guarantee" order. However, the
public API contract specified in the javadoc does mention lexicographical
ordering of the bytes, which is a useful API contract. In our Streams app
we make use of that contract during interactive queries (specifically, to
guarantee correctness when doing a paginated range scan. If the order
changes, then the "bookmark" we use for pagination would be meaningless).

As such, I still think the KIP as you proposed is a highly useful feature.
I would just make a note of the semantics in the JavaDoc and also in the
KIP.

Thanks,
Colt McNealy

*Founder, LittleHorse.dev*


On Mon, Oct 9, 2023 at 2:22 PM Hanyu (Peter) Zheng
 wrote:

> After our discussion, we discovered something intriguing. The definitions
> for the range and reverseRange methods in the ReadOnlyKeyValueStore are as
> follows:
> /**
>  * Get an iterator over a given range of keys. This iterator must be
> closed after use.
>  * The returned iterator must be safe from {@link
> java.util.ConcurrentModificationException}s
>  * and must not return null values.
>  ** Order is not guaranteed as bytes lexicographical ordering might not
> represent key order.*
>  *
>  * @param from The first key that could be in the range, where
> iteration starts from.
>  * A null value indicates that the range starts with the
> first element in the store.
>  * @param to   The last key that could be in the range, where iteration
> ends.
>  * A null value indicates that the range ends with the last
> element in the store.
>  * @return The iterator for this range, from smallest to largest bytes.
>  * @throws InvalidStateStoreException if the store is not initialized
>  */
> KeyValueIterator range(K from, K to);
>
> /**
>  * Get a reverse iterator over a given range of keys. This iterator
> must be closed after use.
>  * The returned iterator must be safe from {@link
> java.util.ConcurrentModificationException}s
>  * and must not return null values.
>  * *Order is not guaranteed as bytes lexicographical ordering might not
> represent key order.*
>  *
>  * @param from The first key that could be in the range, where
> iteration ends.
>  * A null value indicates that the range starts with the
> first element in the store.
>  * @param to   The last key that could be in the range, where iteration
> starts from.
>  * A null value indicates that the range ends with the last
> element in the store.
>  * @return The reverse iterator for this range, from largest to
> smallest key bytes.
>  * @throws InvalidStateStoreException if the store is not initialized
>  */
> default KeyValueIterator reverseRange(K from, K to) {
> throw new UnsupportedOperationException();
> }
>
> The query methods of RangeQuery ultimately invoke either the range method
> or the reverseRange method. However, as per the JavaDoc: the order is not
> guaranteed, since byte lexicographical ordering may not correspond to the
> actual key order.
>
> Sincerely,
> Hanyu
>
> On Fri, Oct 6, 2023 at 10:00 AM Hanyu (Peter) Zheng 
> wrote:
>
> > Thank you, Matthias, for the detailed implementation and explanation. As
> > of now, our capability is limited to executing interactive queries on
> > individual partitions. To illustrate:
> >
> > Consider the IQv2StoreIntegrationTest:
> >
> > We have two partitions:
> > Partition0 contains key-value pairs: <0,0> and <2,2>.
> > Partition1 contains key-value pairs: <1,1> and <3,3>.
> > When executing RangeQuery.withRange(1,3), the results are:
> >
> > Partition0: [2]
> > Partition1: [1, 3]
> > To support functionalities like reverseRange and reverseAll, we can
> > introduce the withDescendingKeys() method. For instance, using
> > RangeQuery.withRange(1,3).withDescendingKeys(), the anticipated results
> are:
> >
> > Partition0: [2]
> > Partition1: [3, 1]
> >
> > In response to Hao's inquiry about the boundary issue, please refer to
> the
> > StoreQueryUtils class. The code snippet:
> >
> > iterator = kvStore.range(lowerRange.orElse(null),
> upperRange.orElse(null));
> > indicates that when implementing range in each store, it's structured
> like:
> >
> > @Override
> > public KeyValueIterator range(final Bytes from, final
> Bytes
> > to) {
> > if (from != null && to != null && from.compareTo(to) > 0) {
> > This section performs the necessary checks.
> >
> > Sincerely,
> > Hanyu
> >
> > On Thu, Oct 5, 2023 at 9:52 AM Hanyu (Peter) Zheng 
> > wrote:
> >
> >> Hi, Hao,
> >>
> >> In this case, it will return an empty set or list in the end.
> >>
> >> Sincerely,
> >> Hanyu
> >>
> >> On Wed, Oct 4, 2023 at 10:29 PM Matthias J. Sax 
> wrote:
> >>
> >>> Great discussion!
> >>>
> >>> It seems the only open question might be about ordering guarantees?
> >>> IIRC, we had a discussion about this in the past.
> >>>
> >>>
>

Re: [DISCUSS] KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2

2023-10-09 Thread Hanyu (Peter) Zheng

Thank you Colt,

At first, we misinterpreted the JavaDoc. Upon further discussion, we
realized that after the key is converted to bytes, queries are based on the
key's byte order, not its intrinsic order.

Sincerely,
Hanyu

On Mon, Oct 9, 2023 at 6:55 PM Colt McNealy  wrote:

> Hanyu,
>
> I like the attention to detail!
>
> It is correct that the JavaDoc does not "guarantee" order. However, the
> public API contract specified in the javadoc does mention lexicographical
> ordering of the bytes, which is a useful API contract. In our Streams app
> we make use of that contract during interactive queries (specifically, to
> guarantee correctness when doing a paginated range scan. If the order
> changes, then the "bookmark" we use for pagination would be meaningless).
>
> As such, I still think the KIP as you proposed is a highly useful feature.
> I would just make a note of the semantics in the JavaDoc and also in the
> KIP.
>
> Thanks,
> Colt McNealy
>
> *Founder, LittleHorse.dev*
>
>
> On Mon, Oct 9, 2023 at 2:22 PM Hanyu (Peter) Zheng
>  wrote:
>
> > After our discussion, we discovered something intriguing. The definitions
> > for the range and reverseRange methods in the ReadOnlyKeyValueStore are
> as
> > follows:
> > /**
> >  * Get an iterator over a given range of keys. This iterator must be
> > closed after use.
> >  * The returned iterator must be safe from {@link
> > java.util.ConcurrentModificationException}s
> >  * and must not return null values.
> >  ** Order is not guaranteed as bytes lexicographical ordering might
> not
> > represent key order.*
> >  *
> >  * @param from The first key that could be in the range, where
> > iteration starts from.
> >  * A null value indicates that the range starts with the
> > first element in the store.
> >  * @param to   The last key that could be in the range, where
> iteration
> > ends.
> >  * A null value indicates that the range ends with the
> last
> > element in the store.
> >  * @return The iterator for this range, from smallest to largest
> bytes.
> >  * @throws InvalidStateStoreException if the store is not initialized
> >  */
> > KeyValueIterator range(K from, K to);
> >
> > /**
> >  * Get a reverse iterator over a given range of keys. This iterator
> > must be closed after use.
> >  * The returned iterator must be safe from {@link
> > java.util.ConcurrentModificationException}s
> >  * and must not return null values.
> >  * *Order is not guaranteed as bytes lexicographical ordering might
> not
> > represent key order.*
> >  *
> >  * @param from The first key that could be in the range, where
> > iteration ends.
> >  * A null value indicates that the range starts with the
> > first element in the store.
> >  * @param to   The last key that could be in the range, where
> iteration
> > starts from.
> >  * A null value indicates that the range ends with the
> last
> > element in the store.
> >  * @return The reverse iterator for this range, from largest to
> > smallest key bytes.
> >  * @throws InvalidStateStoreException if the store is not initialized
> >  */
> > default KeyValueIterator reverseRange(K from, K to) {
> > throw new UnsupportedOperationException();
> > }
> >
> > The query methods of RangeQuery ultimately invoke either the range method
> > or the reverseRange method. However, as per the JavaDoc: the order is not
> > guaranteed, since byte lexicographical ordering may not correspond to the
> > actual key order.
> >
> > Sincerely,
> > Hanyu
> >
> > On Fri, Oct 6, 2023 at 10:00 AM Hanyu (Peter) Zheng  >
> > wrote:
> >
> > > Thank you, Matthias, for the detailed implementation and explanation.
> As
> > > of now, our capability is limited to executing interactive queries on
> > > individual partitions. To illustrate:
> > >
> > > Consider the IQv2StoreIntegrationTest:
> > >
> > > We have two partitions:
> > > Partition0 contains key-value pairs: <0,0> and <2,2>.
> > > Partition1 contains key-value pairs: <1,1> and <3,3>.
> > > When executing RangeQuery.withRange(1,3), the results are:
> > >
> > > Partition0: [2]
> > > Partition1: [1, 3]
> > > To support functionalities like reverseRange and reverseAll, we can
> > > introduce the withDescendingKeys() method. For instance, using
> > > RangeQuery.withRange(1,3).withDescendingKeys(), the anticipated results
> > are:
> > >
> > > Partition0: [2]
> > > Partition1: [3, 1]
> > >
> > > In response to Hao's inquiry about the boundary issue, please refer to
> > the
> > > StoreQueryUtils class. The code snippet:
> > >
> > > iterator = kvStore.range(lowerRange.orElse(null),
> > upperRange.orElse(null));
> > > indicates that when implementing range in each store, it's structured
> > like:
> > >
> > > @Override
> > > public KeyValueIterator range(final Bytes from, final
> > Bytes
> > > to) {
> > > if (from != null && to != null && from.compareT

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1351381500


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact
+your read committed consumers and prevent compacted 
logs from being compacted. KIP-890 helps
+address hanging transactions by verifying partition 
additions. Part 2 of KIP-890 will optimize
+verification, which currently adds an extra hop.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797:
+Accept duplicate listener on port for IPv4/IPv6: 
Until now, Kafka has not supported
+duplicate listeners on the same port. This works when 
using only a single IP stack, but presents
+an issue if you are working with both IPv4 and IPv6. 
With KIP-797, brokers can be configured
+with listeners that have the same port on different IP 
stacks. This update does not affect
+advertised listeners, which already have this feature.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863:
+Reduce CompletedFetch#parseRecord() memory copy: 
reduces memory allocation and

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1351387052


##
blog.html:
##
@@ -22,6 +22,146 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+08 Oct 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+The ability to migrate Kafka clusters from a ZooKeeper 
metadata system to a KRaft metadata system is
+now ready for usage in production environments. See the 
ZooKeeper to KRaft migration
+https://kafka.apache.org/documentation/#kraft_zk_migration";>operations 
documentation for
+details. Note that support for JBOD is still not available 
for KRaft clusters, therefore clusters
+utilizing JBOD can not be migrated. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft";>KIP-858
+for details regarding KRaft and JBOD.
+
+Support for Delegation Tokens in KRaft (https://issues.apache.org/jira/browse/KAFKA-15219";>KAFKA-15219) was 
completed in 3.6, further reducing the gap of features between ZooKeeper-based 
Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to 
KRaft is also included in 3.6.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Release Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for ZooKeeper Deprecation
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405:
+Kafka Tiered Storage (Early Access): 
Introduces Tiered Storage to Kafka. Note that this
+is an early access feature only advised for use in 
non-production environments (see the https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>early
+access notes for more information). This 
feature provides a separation of computation
+and storage in the broker for pluggable storage 
tiering natively in Kafka Tiered Storage brings
+a seamless extension of storage to remote objects with 
minimal operational changes.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-890:
+Transactions Server Side Defense (Part 1): 
Hanging transactions can negatively impact
+your read committed consumers and prevent compacted 
logs from being compacted. KIP-890 helps
+address hanging transactions by verifying partition 
additions. Part 2 of KIP-890 will optimize
+verification, which currently adds an extra hop.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797:
+Accept duplicate listener on port for IPv4/IPv6: 
Until now, Kafka has not supported
+duplicate listeners on the same port. This works when 
using only a single IP stack, but presents
+an issue if you are working with both IPv4 and IPv6. 
With KIP-797, brokers can be configured
+with listeners that have the same port on different IP 
stacks. This update does not affect
+advertised listeners, which already have this feature.
+
+https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863:
+Reduce CompletedFetch#parseRecord() memory copy: 
reduces memory allocation and

Re: [PR] Added a blog entry for 3.6.0 release [kafka-site]

2023-10-09 Thread via GitHub



satishd commented on PR #547:
URL: https://github.com/apache/kafka-site/pull/547#issuecomment-1754340983

   @mimaison @gharris1727 @jolshan Addressed the review comments with the 
latest commit. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-10-09 Thread Ashwin

Hello Taras,

> Do you think that something needs to be corrected in KIP to make it more
understandable without PR? Do you have any advice?
Ha - no. I just wanted to thank you for sharing the PR which helped me as a
newbie.

> If I understood the question correctly:
I was referring to (and did not understand) the removal of L141
in clients/src/main/java/org/apache/kafka/common/security/ssl/SslFactory.java
in https://github.com/apache/kafka/pull/14203/files.
But you are right - this can be discussed in the final PR.

> There might be a better place for this public static method that creates
the SslEngineFactory.
Yes, I think this class should be moved to something like `server-common`
module - but would like any of the committers to comment on this.

Thanks,
Ashwin


On Fri, Sep 29, 2023 at 9:22 PM Taras Ledkov  wrote:

> Hi Ashwin,
>
> Thanks a lot for your review.
>
> > Thanks for the KIP and the PR (which helped me understand the change).
> Do you think that something needs to be corrected in KIP to make it more
> understandable without PR? Do you have any advice?
>
> > I could not understand one thing though - In
> https://github.com/apache/kafka/pull/14203/,
> > why did you have to remove the code which sets sslEngineFactoryConfig in
> instantiateSslEngineFactory?
> If I understood the question correctly:
> I've refactored this method a bit.
> SslFactory#instantiateSslEngineFactory was a private not-static method.
> I've separated the code that really creates new instance of the
> SslEngineFactory and place it into a public static method. There might be a
> better place for this public static method that creates the
> SslEngineFactory. I think we will discuss this at the final PR. Current PR
> is just a demo / prototype to play.
>

[PR] MINOR Updated 3.6 javadocs by replacing 3.6.0-SNAPSHOT with 3.6.0 [kafka-site]

2023-10-09 Thread via GitHub



satishd opened a new pull request, #557:
URL: https://github.com/apache/kafka-site/pull/557

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Apache Kafka 3.7.0 Release

2023-10-09 Thread Luke Chen

Thanks Stanislav!

On Tue, Oct 10, 2023 at 3:05 AM Josep Prat 
wrote:

> Thanks Stanislav!
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Oct 9, 2023, 20:05 Chris Egerton  wrote:
>
> > +1, thanks Stanislav!
> >
> > On Mon, Oct 9, 2023, 14:02 Bill Bejeck  wrote:
> >
> > > +1
> > >
> > > Thanks, Stanislav!
> > >
> > > -Bill
> > >
> > > On Mon, Oct 9, 2023 at 1:59 PM Ismael Juma  wrote:
> > >
> > > > Thanks for volunteering Stanislav!
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Oct 9, 2023 at 10:51 AM Stanislav Kozlovski
> > > >  wrote:
> > > >
> > > > > Hey all!
> > > > >
> > > > > I would like to volunteer to be the release manager driving the
> next
> > > > > release - Apache Kafka *3.7.0*.
> > > > >
> > > > > If there are no objections, I will start and share a release plan
> > soon
> > > > > enough!
> > > > >
> > > > > Cheers,
> > > > > Stanislav
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-09 Thread Tom Bentley

Hi Greg,

Many thanks for the KIP. Here are a few initial questions

1. Incomplete sentence: "But Connect is not situated to be able to manage
resources directly, as workers are given a fixed "
2. You explain how sessioned is now a subset of static, but what happens in
a cluster where some workers are using static and some are using either
eager or compatible?
3. "Assign each unassigned static job to a static worker which specifies
that job, choosing arbitrarily if there are multiple valid workers." I
think there might be less ambiguous words than "arbitrary" to specify this
behaviour. Hashing the task name would _appear_ pretty arbitrary to the
user, but would be deterministic. Picking at random would be
non-deterministic. Even better if you have a rationale.
4. You don't describe how a user, or automated system, starting with a set
of connectors, should find out the tasks that they want to run. This
relates to the contract of
org.apache.kafka.connect.connector.Connector#taskConfigs(int maxTasks).
AFAIK (please correct me if I'm wrong, because it's a long time since I
looked at this code) there's nothing that validates that the returned list
has at most the `maxTasks` and connectors can, of course, return fewer than
that many tasks. So without deeper knowledge of a particular connector it's
not clear to the user/operator how to configure their static workers and
static assignments.
5. Is there a lurking assumption that task indices are stable? E.g. that
the task with index 3 will always be the resource-intensive one. I can see
that that would often be a reliable assumption, but it's not clear to me
that it is always the case.

Thanks again,

Tom

On Fri, 6 Oct 2023 at 12:36, Greg Harris 
wrote:

> Hey everyone!
>
> I'd like to propose an improvement to Kafka Connect's scheduling
> algorithm, with the goal of improving the operability of connect
> clusters through resource isolation.
>
> Find the KIP here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-987%3A+Connect+Static+Assignments
>
> This feature is primarily intended to be consumed by cluster
> management systems, so I've opened a sister proposal in the Strimzi
> project that uses this feature to provide user-facing resource
> isolation. The base feature is generic, so it is usable manually and
> with cluster management systems other than Strimzi.
>
> Find the proposal here: https://github.com/strimzi/proposals/pull/96
>
> Thanks!
> Greg Harris
>
>

46 matches

Mail list logo