[jira] [Created] (KAFKA-14913) Migrate DistributedHerder Executor shutdown to use ThreadUtils#shutdownExecutorServiceQuietly

[Sagar Rao (Jira)]

Sagar Rao created KAFKA-14913:
-

 Summary: Migrate DistributedHerder Executor shutdown to use 
ThreadUtils#shutdownExecutorServiceQuietly
 Key: KAFKA-14913
 URL: https://issues.apache.org/jira/browse/KAFKA-14913
 Project: Kafka
  Issue Type: Improvement
Reporter: Sagar Rao






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Bruno Cadonna]

Thanks Satish!

+1

Best,
Bruno

On 16.04.23 20:02, Ismael Juma wrote:

Thanks for volunteering Satish. +1.

Ismael

On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
wrote:


Hi,
I would like to volunteer as release manager for the next release,
which will be Apache Kafka 3.6.0.

If there are no objections, I will start a release plan a week after
3.5.0 release(around early May).

Thanks,
Satish.

[jira] [Created] (KAFKA-14914) binarySearch in AbstactIndex may execute with infinite loop

[li xiangyuan (Jira)]

li xiangyuan created KAFKA-14914:


 Summary: binarySearch in AbstactIndex may execute with infinite 
loop
 Key: KAFKA-14914
 URL: https://issues.apache.org/jira/browse/KAFKA-14914
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.4.0
Reporter: li xiangyuan
 Attachments: stack.1.txt, stack.2.txt, stack.3.txt

Recently our servers in production environment may suddenly stop handle request 
frequently(for now 3 times in less than 10 days),   please check the stack file 
uploaded, it show that 1 ioThread(data-plane-kafka-request-handler-11) hold  
the ReadLock of Partition's leaderIsrUpdateLock and keep run the binarySearch 
function, once any thread(kafka-scheduler-2) need WriteMode Of this lock, then 
all requests read this partition need ReadMode Lock will use out all ioThreads 
and then this broker couldn't handle any request.

the 3 stack files are fetched with interval  about 6 minute, with my standpoint 
i just could think obviously the  binarySearch function cause dead lock and I 
presuppose maybe the index block values in offsetIndex (at least in mmap) are 
not sorted.

 

detail information:

this problem appear in 2 brokers

broker version: 2.4.0

jvm: openjdk 11

hardware: aws c7g 4xlarge, this is a arm64 server, we recently upgrade our 
servers from c6g 4xlarge to this type, when we use c6g haven't meet this 
problem, we don't know whether arm or aws c7g server have any problem.

other: once we restart broker, it will recover, so we doubt offset index file 
may not corrupted and maybe something wrong with mmap.

plz give any suggestion solve this problem, thx.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14915) Option to consume multiple partitions that have their data in remote storage for the target offsets.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-14915:
--

 Summary: Option to consume multiple partitions that have their 
data in remote storage for the target offsets.
 Key: KAFKA-14915
 URL: https://issues.apache.org/jira/browse/KAFKA-14915
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Chia-Ping Tsai]

+1 to Satish!

Thanks!

> Bruno Cadonna  於 2023年4月17日 下午5:09 寫道：
> 
> Thanks Satish!
> 
> +1
> 
> Best,
> Bruno
> 
>> On 16.04.23 20:02, Ismael Juma wrote:
>> Thanks for volunteering Satish. +1.
>> Ismael
>>> On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
>>> wrote:
>>> Hi,
>>> I would like to volunteer as release manager for the next release,
>>> which will be Apache Kafka 3.6.0.
>>> 
>>> If there are no objections, I will start a release plan a week after
>>> 3.5.0 release(around early May).
>>> 
>>> Thanks,
>>> Satish.
>>> 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Chris Egerton]

Hi Mickael,

It looks like we missed the feature freeze cutoff for part but not all of
KIP-875 [1]. The features we've been able to merge so far are the new
STOPPED state for connectors [2] and the API for reading offsets [3]. The
features we have not been able to merge yet are the APIs for altering and
resetting offsets.

The already-merged features are useful on their own and I believe it should
be acceptable to release them separately from the not-yet-merged ones, but
wanted to double-check with you that it's alright to split this KIP across
two or more releases, starting with 3.5.0.

Cheers,

Chris

[1] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
[2] - https://github.com/apache/kafka/pull/13424
[3] - https://github.com/apache/kafka/pull/13434

On Fri, Apr 14, 2023 at 10:13 AM Matthias J. Sax  wrote:

> Thanks a lot!
>
> On 4/14/23 5:32 AM, Mickael Maison wrote:
> > Hi Matthias,
> >
> > I merged the PR before cutting the 3.5 branch.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Apr 14, 2023 at 2:31 PM Mickael Maison 
> wrote:
> >>
> >> Hi David,
> >>
> >> I've created the 3.5 branch. Feel free to cherry pick these 2 commits
> >> when they are ready.
> >>
> >> Thanks,
> >> Mickael
> >>
> >> On Fri, Apr 14, 2023 at 11:23 AM Satish Duggana
> >>  wrote:
> >>>
> >>> Thanks Luke for helping with the reviews and adding a few tests in a
> >>> couple of PRs.
> >>>
> >>> Hi Mickael,
> >>> I raised 3 PRs recently for tiered storage, one is merged. The other 2
> >>> PRs are in the critical path of non-tiered storage changes also.
> >>> Especially, in consumer fetch and retention cleanup paths. These need
> >>> to be thoroughly reviewed and avoid any regressions in that area. We
> >>> should merge them to the trunk as soon as possible to make it easier
> >>> to work on follow-up PRs. IMO, we can avoid merging these PRs in 3.5
> >>> just before the release without baking for a longer duration. We can
> >>> take a call on this later after the reviews are done.
> >>>
> >>> Many of the individual functionalities related to tiered storage like
> >>> default topic based RLMM implementation, enhanced follower fetch
> >>> protocol implementation for tiered storage, copying remote log
> >>> segments are merged.
> >>> There are 2 PRs for consumer fetch for remote record reads, remote
> >>> retention cleanup and topic deletion functionality are under review.
> >>>
> >>> I do not think it can be considered as an early access review even
> >>> with the 2 PRs in review. Luke and I synced up and agreed on the same.
> >>> Most of the recent functionality is added with a few unit tests. We
> >>> plan to have follow-up PRs on the immediate pending items and also
> >>> raise PRs in the next few weeks on unit tests, integration test
> >>> framework and several integration tests for many of these
> >>> functionalities tying together.
> >>>
> >>> Thanks,
> >>> Satish.
> >>>
> >>>
> >>> On Fri, 14 Apr 2023 at 12:52, Matthias J. Sax 
> wrote:
> 
>  Hey Mickael,
> 
>  we have one open PR for KIP-914 left. Would be great if you could
> merge
>  it before cutting the 3.5 branch. If you don't want to merge it and
>  prefer that I cherry-pick it to 3.5 branch later, also works for me.
> 
>  I did close the ticket already as resolved. It's just a minor change
> the
>  KIP.
> 
>  https://github.com/apache/kafka/pull/13565
> 
> 
>  Thanks a lot!
>  -Matthias
> 
> 
>  On 4/13/23 4:32 AM, David Jacot wrote:
> > Hi Mickael,
> >
> > Thanks for the heads up. As raised by Jeff earlier in this thread,
> we would
> > like to get the two small patches [1][2] for KIP-915 in 3.5. The PRs
> are in
> > review and I should be able to merge them in the next few days. I
> will
> > cherry-pick them to the release branch in your create it in the
> meantime.
> >
> > [1] https://github.com/apache/kafka/pull/13511
> > [2] https://github.com/apache/kafka/pull/13526
> >
> > Best,
> > David
> >
> > On Thu, Apr 13, 2023 at 12:55 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> >> Hi Luke,
> >>
> >> Thanks for the heads up. This would be great to get Tiered Storage
> in
> >> Early Access. Let me know if you can't get everything done this
> week.
> >>
> >> Mickael
> >>
> >> On Thu, Apr 13, 2023 at 12:54 PM Mickael Maison
> >>  wrote:
> >>>
> >>> Hi,
> >>>
> >>> We've now reached feature freeze for 3.5.0. From now on, only bug
> >>> fixes and changes related to stabilizing the release should be
> merged.
> >>>
> >>> I plan to create the release branch tomorrow (Friday 14). After
> this
> >>> point, you'll have to cherry pick changes to the release branch
> when
> >>> merging a PR. I'll send another message once the branch has been
> >>> created.
> >>>
> >>> I've updated the rele

Re: Apache Kafka 3.6.0 release

[Chris Egerton]

Thanks Satish, +1!

On Mon, Apr 17, 2023 at 5:09 AM Bruno Cadonna  wrote:

> Thanks Satish!
>
> +1
>
> Best,
> Bruno
>
> On 16.04.23 20:02, Ismael Juma wrote:
> > Thanks for volunteering Satish. +1.
> >
> > Ismael
> >
> > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> >
> >> Hi,
> >> I would like to volunteer as release manager for the next release,
> >> which will be Apache Kafka 3.6.0.
> >>
> >> If there are no objections, I will start a release plan a week after
> >> 3.5.0 release(around early May).
> >>
> >> Thanks,
> >> Satish.
> >>
> >
>

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi Chris,

I was looking at that just now! As you said, the PRs merged provide
some functionality so I think it's fine to deliver the KIP across 2
releases.
I left a comment in https://issues.apache.org/jira/browse/KAFKA-14876
to document what's in 3.5.

Thanks,
Mickael


On Mon, Apr 17, 2023 at 5:05 PM Chris Egerton  wrote:
>
> Hi Mickael,
>
> It looks like we missed the feature freeze cutoff for part but not all of
> KIP-875 [1]. The features we've been able to merge so far are the new
> STOPPED state for connectors [2] and the API for reading offsets [3]. The
> features we have not been able to merge yet are the APIs for altering and
> resetting offsets.
>
> The already-merged features are useful on their own and I believe it should
> be acceptable to release them separately from the not-yet-merged ones, but
> wanted to double-check with you that it's alright to split this KIP across
> two or more releases, starting with 3.5.0.
>
> Cheers,
>
> Chris
>
> [1] -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> [2] - https://github.com/apache/kafka/pull/13424
> [3] - https://github.com/apache/kafka/pull/13434
>
> On Fri, Apr 14, 2023 at 10:13 AM Matthias J. Sax  wrote:
>
> > Thanks a lot!
> >
> > On 4/14/23 5:32 AM, Mickael Maison wrote:
> > > Hi Matthias,
> > >
> > > I merged the PR before cutting the 3.5 branch.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Fri, Apr 14, 2023 at 2:31 PM Mickael Maison 
> > wrote:
> > >>
> > >> Hi David,
> > >>
> > >> I've created the 3.5 branch. Feel free to cherry pick these 2 commits
> > >> when they are ready.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Fri, Apr 14, 2023 at 11:23 AM Satish Duggana
> > >>  wrote:
> > >>>
> > >>> Thanks Luke for helping with the reviews and adding a few tests in a
> > >>> couple of PRs.
> > >>>
> > >>> Hi Mickael,
> > >>> I raised 3 PRs recently for tiered storage, one is merged. The other 2
> > >>> PRs are in the critical path of non-tiered storage changes also.
> > >>> Especially, in consumer fetch and retention cleanup paths. These need
> > >>> to be thoroughly reviewed and avoid any regressions in that area. We
> > >>> should merge them to the trunk as soon as possible to make it easier
> > >>> to work on follow-up PRs. IMO, we can avoid merging these PRs in 3.5
> > >>> just before the release without baking for a longer duration. We can
> > >>> take a call on this later after the reviews are done.
> > >>>
> > >>> Many of the individual functionalities related to tiered storage like
> > >>> default topic based RLMM implementation, enhanced follower fetch
> > >>> protocol implementation for tiered storage, copying remote log
> > >>> segments are merged.
> > >>> There are 2 PRs for consumer fetch for remote record reads, remote
> > >>> retention cleanup and topic deletion functionality are under review.
> > >>>
> > >>> I do not think it can be considered as an early access review even
> > >>> with the 2 PRs in review. Luke and I synced up and agreed on the same.
> > >>> Most of the recent functionality is added with a few unit tests. We
> > >>> plan to have follow-up PRs on the immediate pending items and also
> > >>> raise PRs in the next few weeks on unit tests, integration test
> > >>> framework and several integration tests for many of these
> > >>> functionalities tying together.
> > >>>
> > >>> Thanks,
> > >>> Satish.
> > >>>
> > >>>
> > >>> On Fri, 14 Apr 2023 at 12:52, Matthias J. Sax 
> > wrote:
> > 
> >  Hey Mickael,
> > 
> >  we have one open PR for KIP-914 left. Would be great if you could
> > merge
> >  it before cutting the 3.5 branch. If you don't want to merge it and
> >  prefer that I cherry-pick it to 3.5 branch later, also works for me.
> > 
> >  I did close the ticket already as resolved. It's just a minor change
> > the
> >  KIP.
> > 
> >  https://github.com/apache/kafka/pull/13565
> > 
> > 
> >  Thanks a lot!
> >  -Matthias
> > 
> > 
> >  On 4/13/23 4:32 AM, David Jacot wrote:
> > > Hi Mickael,
> > >
> > > Thanks for the heads up. As raised by Jeff earlier in this thread,
> > we would
> > > like to get the two small patches [1][2] for KIP-915 in 3.5. The PRs
> > are in
> > > review and I should be able to merge them in the next few days. I
> > will
> > > cherry-pick them to the release branch in your create it in the
> > meantime.
> > >
> > > [1] https://github.com/apache/kafka/pull/13511
> > > [2] https://github.com/apache/kafka/pull/13526
> > >
> > > Best,
> > > David
> > >
> > > On Thu, Apr 13, 2023 at 12:55 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Hi Luke,
> > >>
> > >> Thanks for the heads up. This would be great to get Tiered Storage
> > in
> > >> Early Access. Let me know if you can't get everything done this
> > week.
> > >>
> > >> Mickael

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Alexandre Dupriez]

Hi Ivan,

Thank you for the KIP.

I think the KIP clearly explains the need for out-of-band metadata
authored and used by an implementation of the remote storage manager.
One question I would have is, what is the benefit of adding these
custom metadata in the rlmMetadata rather than letting the plugin
manage access and persistence to them?

Maybe one disadvantage and potential risk with the approach proposed
in the KIP is that the rlmMetadata is not of a predefined, relatively
constant size (although corner cases with thousands of leader epochs
in the leader epoch map are possible). It would be possible for a user
to use custom metadata large enough to adversely impact access to and
caching of the rlmMetadata by Kafka.

Thanks,
Alexandre

Le jeu. 6 avr. 2023 à 16:03, hzh0425  a écrit :
>
> I think it's a good idea as we may want to store remote segments in different 
> buckets
>
>
>
> | |
> hzhka...@163.com
> |
> |
> 邮箱：hzhka...@163.com
> |
>
>
>
>
>  回复的原邮件 
> | 发件人 | Ivan Yurchenko |
> | 日期 | 2023年04月06日 22:37 |
> | 收件人 | dev@kafka.apache.org |
> | 抄送至 | |
> | 主题 | [DISCUSS] KIP-917: Additional custom metadata for remote log segment |
> Hello!
>
> I would like to start the discussion thread on KIP-917: Additional custom
> metadata for remote log segment [1]
> This KIP is fairly small and proposes to add a new field to the remote
> segment metadata.
>
> Thank you!
>
> Best,
> Ivan
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment

Re: Apache Kafka 3.6.0 release

[Mickael Maison]

Thanks Satish!

On Sun, Apr 16, 2023 at 8:03 PM Ismael Juma  wrote:
>
> Thanks for volunteering Satish. +1.
>
> Ismael
>
> On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
> wrote:
>
> > Hi,
> > I would like to volunteer as release manager for the next release,
> > which will be Apache Kafka 3.6.0.
> >
> > If there are no objections, I will start a release plan a week after
> > 3.5.0 release(around early May).
> >
> > Thanks,
> > Satish.
> >

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1771

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 373841 lines...]
[2023-04-17T10:33:59.688Z] [INFO] Copying 2 resources
[2023-04-17T10:34:00.712Z] [INFO] Copying 3 resources
[2023-04-17T10:34:00.712Z] [INFO] 
[2023-04-17T10:34:00.712Z] [INFO] --- maven-archetype-plugin:2.2:jar 
(default-jar) @ streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] Building archetype jar: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] --- 
maven-archetype-plugin:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] --- maven-gpg-plugin:1.6:sign 
(sign-artifacts) @ streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.jar
[2023-04-17T10:34:04.223Z] [INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/java/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.pom
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] --- 
maven-archetype-plugin:2.2:update-local-catalog (default-update-local-catalog) 
@ streams-quickstart-java ---
[2023-04-17T10:34:04.223Z] [INFO] 

[2023-04-17T10:34:04.223Z] [INFO] Reactor Summary for Kafka Streams :: 
Quickstart 3.6.0-SNAPSHOT:
[2023-04-17T10:34:04.223Z] [INFO] 
[2023-04-17T10:34:04.223Z] [INFO] Kafka Streams :: Quickstart 
 SUCCESS [ 14.306 s]
[2023-04-17T10:34:04.223Z] [INFO] streams-quickstart-java 
 SUCCESS [  6.040 s]
[2023-04-17T10:34:04.223Z] [INFO] 

[2023-04-17T10:34:04.223Z] [INFO] BUILD SUCCESS
[2023-04-17T10:34:04.223Z] [INFO] 

[2023-04-17T10:34:04.223Z] [INFO] Total time:  21.196 s
[2023-04-17T10:34:04.223Z] [INFO] Finished at: 2023-04-17T10:34:03Z
[2023-04-17T10:34:04.223Z] [INFO] 

[Pipeline] dir
[2023-04-17T10:34:04.737Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
[2023-04-17T10:34:07.186Z] + echo Y
[2023-04-17T10:34:07.186Z] + mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.6.0-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps
[2023-04-17T10:34:09.404Z] [INFO] Scanning for projects...
[2023-04-17T10:34:10.596Z] [INFO] 
[2023-04-17T10:34:10.596Z] [INFO] --< 
org.apache.maven:standalone-pom >---
[2023-04-17T10:34:10.596Z] [INFO] Building Maven Stub Project (No POM) 1
[2023-04-17T10:34:10.596Z] [INFO] [ pom 
]-
[2023-04-17T10:34:10.596Z] [INFO] 
[2023-04-17T10:34:10.596Z] [INFO] >>> maven-archetype-plugin:3.2.1:generate 
(default-cli) > generate-sources @ standalone-pom >>>
[2023-04-17T10:34:10.596Z] [INFO] 
[2023-04-17T10:34:10.596Z] [INFO] <<< maven-archetype-plugin:3.2.1:generate 
(default-cli) < generate-sources @ standalone-pom <<<
[2023-04-17T10:34:10.596Z] [INFO] 
[2023-04-17T10:34:10.596Z] [INFO] 
[2023-04-17T10:34:10.596Z] [INFO] --- maven-archetype-plugin:3.2.1:generate 
(default-cli) @ standalone-pom ---
[2023-04-17T10:34:14.713Z] [INFO] Generating project in Interactive mode
[2023-04-17T10:34:14.713Z] [WARNING] Archetype not found in any catalog. 
Falling back to central repository.
[2023-04-17T10:34:14.713Z] [WARNING] Add a repository with id 'archetype' in 
your settings.xml if archetype's repository is elsewhere.
[2023-04-17T10:34:14.713Z] [INFO] Using property: groupId = streams.examples
[2023-04-17T10:34:14.713Z] [INFO] Using property: artifactId = streams.examples
[2023-04-17T10:34:14.713Z] [INFO] Using property: version = 0.1
[2023-04-17T10:34:14.713Z] [INFO] Using property: package = myapps
[2023-04-17

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

[Mickael Maison]

Hi Christo,

+1 (binding)

Thanks for the KIP

On Fri, Apr 14, 2023 at 7:32 PM Colin McCabe  wrote:
>
> On Sun, Apr 9, 2023, at 19:17, Ismael Juma wrote:
> >
> > On Sun, Apr 9, 2023 at 4:53 PM Colin McCabe  wrote:
> >
> >> We are going to deprecate ZK mode soon. So if this is indeed a requirement
> >> (no deprecated software in prod), perhaps those users will have to move to
> >> KRaft mode. (Independently of what we decide here)
> >>
> >
> > Not sure where "no deprecated software in prod" is coming from. The concern
> > is regarding end-of-life software - i.e. software that no longer receives
> > security fixes. If we don't upgrade beyond 3.6.x, we'll be in a tough
> > position when a CVE is fixed only in ZooKeeper 3.7.x, 3.8.x, etc. If it's a
> > serious security problem, then it's likely that an additional release of
> > ZooKeeper 3.6.x might be released. But the more likely case is that a
> > library dependency will have a CVE that will trigger the compliance checks
> > from enterprise users, but not warrant another ZooKeeper 3.6.x release.
>
> Hi Ismael,
>
> Fair enough. There is a difference between deprecated and unsupported. ZK 
> 3.6.x is unsupported which is worse than deprecated, since it means it will 
> not be updated.
>
> Overall, I agree with you that we're going to have to move to the new version 
> of ZK. This fits in with the overall timeline of one more year of Kafka 
> releases supporting ZK. If Apache Kafka 4.0 is April 2024, we'll need to be 
> getting security updates for ZK during this time.
>
> On Wed, Apr 12, 2023, at 08:45, Christo Lolov wrote:
> > Hello Colin,
> >
> > Thank you for the response!
> >
> > 1. I have attached the compatibility matrix in the KIP under the section
> > Compatibility, Deprecation, and Migration Plan.
>
> Hi Christo,
>
> Thanks for attaching the matrix to the KIP.
>
> I don't understand why Kafka clients are part of this matrix. The Kafka 
> client doesn't use ZK directly. (Well, certain very ancient pre-1.0 Kafka 
> clients did, but that was a long time ago). So let's remove this.
>
> If I understand this correctly, the main documentation that will be needed is 
> for pre-2.4 Kafka releases. Assuming they keep everything "stock" (which in 
> my experience most users do), the net-net is that pre-2.4 releases need to 
> make an extra hop through a post-2.4, pre-3.6 release. We will have to 
> document that as prominently as we can.
>
> I am +1 for this with the proviso that we do it in 3.6. We should update the 
> version as soon as we can post-3.5 so that any bugs shake out as soon as 
> possible.
>
> best,
> Colin

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Jun,

Thank you for sharing your questions, please find my answers below.


41. There can only be user partitions on `metadata.log.dir` if that log
dir is also listed in `log.dirs`.
`LogManager` does not specifically load contents from `metadata.log.dir`.

The broker will communicate UUIDs to the controller for all log dirs
configured in `log.dirs`. If the metadata directory happens to be one
of those, it may also contain user partitions, so the controller will
know about it. If it is a completely separate log dir, it cannot hold
user partitions, so there's no need to include it.


42. I'm not sure about what exactly you refer to with "decommission the
disk", so please let me know if I'm missing your point here.

A disk can be removed from `log.dirs` and removed from the system in a
single broker restart:

  1. Shutdown the broker
  2. Unmount, remove the disk
  3. Update `log.dirs` config
  4. Start the broker

Upon restart, the broker will update `directory.ids` in the
`meta.properties` for the remaining configured log dirs.

Log dir identity cannot be inferred from the path, because the same
storage device can be remounted under a different path, so the way we
identify storage directories is by looking at their contents – the
`directory.id` field in its `meta.properties`.
But this also means  that a log dir cannot be identified if it is not
available, and so it also means that the broker can only generate
`directory.ids` if all log directories listed under `log.dirs` happen
to be available.

Consider the following example, where `log.dirs=/a,/b/,/c`, and
the following `meta.properties` (non-relevant values omitted):

# /a/meta.properties
directory.id=1
directory.ids=1,2,3

# /b/meta.properties
directory.id=2
directory.ids=1,2,3

# /c/meta.properties
directory.id=3
directory.ids=1,2,3

If `log.dirs` is updated to remove `/c`, the broker will be able
to determine the new value for `directory.ids=1,2` by loading
`/a/meta.properties` and `/b/meta.properties`.
But if either `/a`, or `/b` happens to be unavailable, e.g. due to some
temporary disk failure we cannot determine `directory.ids`. e.g.
if `/b` is unavailable, the broker can't tell if `directory.ids` should be
`1,2`, `1,3`, or even `1,4`.

In a scenario where an operator wishes to remove a log dir from
configuration and some other log dir is also offline, the operator will have
a few options:

  a) Bring the offline log dir back online before restarting the broker.

  b) Edit `meta.properties` to remove the UUID for the deconfigured logdir
  from `directory.ids` in the remaining available log dirs. This will remove
  the need for the broker to regenerate `directory.ids` as the entry count
  for `directory.ids` and `log.dirs` will be equal.

  c) Also remove the offline log dir from `log.dirs`.


43. If the log dir was already failed at startup, indeed, the broker
will not know that. But in that case, there's no risk of a race or failure.

What I meant here relates rather to log dir failures at runtime.
I've updated this bit in the KIP to clarify.

When executing the log directory failure handler, the broker knows
which directory failed, which partitions resided there, and it can
check if any of those newly failed partitions refer to a different
log dir in the cluster metadata.

The assignment should be correct for all of them, as the broker
will be proactive in notifying the controller of any changes in
log dir assignment. But in case of some race condition, the broker
should nudge the controller to deal with the incorrectly
assigned partitions.


44. Tom Bentley and I have discussed this previously in this thread,
in emails dated Jan 10, 13, 23 and Feb 3.

When upgrading a JBOD enabled ZK cluster, we could piggyback on the ZK to
KRaft upgrade (as per KIP-866) and delay bumping `meta.properties` until the
ZK->KRaft upgrade is finalized. After then, we do not support downgrading.

But I'm not convinced we should do this, since there's another upgrade
scenario – when this change proposed in this KIP is applied to a KRaft
cluster that does not yet support JBOD.
In this scenario there are no multiple steps in which one of them is
considered final, and I'm not sure we'd want to introduce an
additional step – making the upgrade process more complex – just to
address this issue either.

I think the best approach is to keep using `version=1` in `meta.properties`.
The new properties introduced in this KIP will safely be ignored by previous
versions, in either ZK or KRaft mode, and we avoid creating conflicts with
unexpected declared versions. The presence of the extra fields is also
innocuous in the case of a second upgrade following a downgrade.

I've updated the KIP to reflect this. Let me know what you think.


Best,

--
Igor

Re: Apache Kafka 3.6.0 release

[Luke Chen]

Thanks for volunteering!

+1

Luke

On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  wrote:

> Thanks for volunteering Satish. +1.
>
> Ismael
>
> On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
> wrote:
>
> > Hi,
> > I would like to volunteer as release manager for the next release,
> > which will be Apache Kafka 3.6.0.
> >
> > If there are no objections, I will start a release plan a week after
> > 3.5.0 release(around early May).
> >
> > Thanks,
> > Satish.
> >
>

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Alexandre,

Thank you for your feedback!

> One question I would have is, what is the benefit of adding these
> custom metadata in the rlmMetadata rather than letting the plugin
> manage access and persistence to them?

Could you please elaborate? Do I understand correctly that the idea is that
the plugin will have its own storage for those custom metadata, for example
a special topic?

> It would be possible for a user
> to use custom metadata large enough to adversely impact access to and
> caching of the rlmMetadata by Kafka.

Since the custom metadata is 100% under control of the RSM plugin, the risk
is as big as the risk of running a third-party code (i.e. the RSM plugin).
The cluster admin must make the decision if they trust it.
To mitigate this risk and put it under control, the RSM plugin
implementations could document what custom metadata they use and estimate
their size.

Best,
Ivan


On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez 
wrote:

> Hi Ivan,
>
> Thank you for the KIP.
>
> I think the KIP clearly explains the need for out-of-band metadata
> authored and used by an implementation of the remote storage manager.
> One question I would have is, what is the benefit of adding these
> custom metadata in the rlmMetadata rather than letting the plugin
> manage access and persistence to them?
>
> Maybe one disadvantage and potential risk with the approach proposed
> in the KIP is that the rlmMetadata is not of a predefined, relatively
> constant size (although corner cases with thousands of leader epochs
> in the leader epoch map are possible). It would be possible for a user
> to use custom metadata large enough to adversely impact access to and
> caching of the rlmMetadata by Kafka.
>
> Thanks,
> Alexandre
>
> Le jeu. 6 avr. 2023 à 16:03, hzh0425  a écrit :
> >
> > I think it's a good idea as we may want to store remote segments in
> different buckets
> >
> >
> >
> > | |
> > hzhka...@163.com
> > |
> > |
> > 邮箱：hzhka...@163.com
> > |
> >
> >
> >
> >
> >  回复的原邮件 
> > | 发件人 | Ivan Yurchenko |
> > | 日期 | 2023年04月06日 22:37 |
> > | 收件人 | dev@kafka.apache.org |
> > | 抄送至 | |
> > | 主题 | [DISCUSS] KIP-917: Additional custom metadata for remote log
> segment |
> > Hello!
> >
> > I would like to start the discussion thread on KIP-917: Additional custom
> > metadata for remote log segment [1]
> > This KIP is fairly small and proposes to add a new field to the remote
> > segment metadata.
> >
> > Thank you!
> >
> > Best,
> > Ivan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment
>

[jira] [Resolved] (KAFKA-14775) Support SCRAM for broker to controller authentication

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-14775.
--
Fix Version/s: 3.5.0
 Assignee: Colin McCabe  (was: Proven Provenzano)
   Resolution: Fixed

> Support SCRAM for broker to controller authentication
> -
>
> Key: KAFKA-14775
> URL: https://issues.apache.org/jira/browse/KAFKA-14775
> Project: Kafka
>  Issue Type: Improvement
>  Components: kraft
>Reporter: Proven Provenzano
>Assignee: Colin McCabe
>Priority: Major
> Fix For: 3.5.0
>
>
> We need to apply SCRAM changes to controller nodes.
> We need to handle DescribeUserScramCredentialsRequest in the controller nodes.
> As part of this update I will split out the code from 
> {{BrokerMetadataPublisher.scala}} for applying the SCRAM  into a separate 
> {{{}MetadataPublisher{}}}, as we did with {{DynamicConfigPublisher}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1772

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14796) Migrate ZK ACLs to KRaft

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-14796.
--
Resolution: Fixed

Removed the 3.4.1 fix version since we're probably not back-porting this.

> Migrate ZK ACLs to KRaft
> 
>
> Key: KAFKA-14796
> URL: https://issues.apache.org/jira/browse/KAFKA-14796
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14916) Fix code that assumes transactional ID implies all records are transactional

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14916:
--

 Summary: Fix code that assumes transactional ID implies all 
records are transactional
 Key: KAFKA-14916
 URL: https://issues.apache.org/jira/browse/KAFKA-14916
 Project: Kafka
  Issue Type: Sub-task
Reporter: Justine Olshan
Assignee: Justine Olshan


KAFKA-14561 wrote code that assumed that if a transactional ID was included, 
all record batches were transactional and had the same producer ID.

This work with improve validation and fix the code that assumes all batches are 
transactional.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14917) Producer write while transaction is pending.

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14917:
--

 Summary: Producer write while transaction is pending.
 Key: KAFKA-14917
 URL: https://issues.apache.org/jira/browse/KAFKA-14917
 Project: Kafka
  Issue Type: Bug
Reporter: Justine Olshan
Assignee: Justine Olshan


As discovered in KAFKA-14904, we seem to get into a state where we try to write 
to a partition while the ongoing state is still pending.

This is likely a bigger issue than the test and worth looking in to.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1773

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 556376 lines...]
[2023-04-18T00:22:48.861Z] [INFO] Parameter: package, Value: myapps
[2023-04-18T00:22:48.861Z] [INFO] Parameter: version, Value: 0.1
[2023-04-18T00:22:48.861Z] [INFO] Parameter: groupId, Value: streams.examples
[2023-04-18T00:22:48.861Z] [INFO] Parameter: artifactId, Value: streams.examples
[2023-04-18T00:22:48.861Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/quickstart/test-streams-archetype/streams.examples
[2023-04-18T00:22:48.861Z] [INFO] 

[2023-04-18T00:22:48.861Z] [INFO] BUILD SUCCESS
[2023-04-18T00:22:48.861Z] [INFO] 

[2023-04-18T00:22:48.861Z] [INFO] Total time:  2.062 s
[2023-04-18T00:22:48.861Z] [INFO] Finished at: 2023-04-18T00:22:47Z
[2023-04-18T00:22:48.861Z] [INFO] 

[Pipeline] dir
[2023-04-18T00:22:49.375Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2023-04-18T00:22:52.024Z] + mvn compile
[2023-04-18T00:22:53.864Z] [INFO] Scanning for projects...
[2023-04-18T00:22:53.864Z] [INFO] 
[2023-04-18T00:22:53.864Z] [INFO] -< 
streams.examples:streams.examples >--
[2023-04-18T00:22:53.864Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2023-04-18T00:22:53.864Z] [INFO] [ jar 
]-
[2023-04-18T00:22:53.864Z] [INFO] 
[2023-04-18T00:22:53.864Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2023-04-18T00:22:54.892Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2023-04-18T00:22:54.892Z] [INFO] Copying 1 resource
[2023-04-18T00:22:54.892Z] [INFO] 
[2023-04-18T00:22:54.892Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2023-04-18T00:22:54.892Z] [INFO] Changes detected - recompiling the module!
[2023-04-18T00:22:54.892Z] [INFO] Compiling 3 source files to 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk@2/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2023-04-18T00:22:56.265Z] [INFO] 

[2023-04-18T00:22:56.265Z] [INFO] BUILD SUCCESS
[2023-04-18T00:22:56.265Z] [INFO] 

[2023-04-18T00:22:56.265Z] [INFO] Total time:  2.802 s
[2023-04-18T00:22:56.265Z] [INFO] Finished at: 2023-04-18T00:22:55Z
[2023-04-18T00:22:56.265Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-04-18T00:23:01.968Z] > Task :streams:compileTestJava
[2023-04-18T00:23:01.968Z] > Task :streams:testClasses
[2023-04-18T00:23:01.968Z] > Task :streams:testJar
[2023-04-18T00:23:01.968Z] > Task :streams:testSrcJar
[2023-04-18T00:23:01.968Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-04-18T00:23:01.968Z] > Task :streams:publishToMavenLocal
[2023-04-18T00:23:01.968Z] 
[2023-04-18T00:23:01.968Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-04-18T00:23:01.968Z] 
[2023-04-18T00:23:01.968Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-04-18T00:23:01.968Z] 
[2023-04-18T00:23:01.968Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-04-18T00:23:01.968Z] 
[2023-04-18T00:23:01.968Z] BUILD SUCCESSFUL in 5m 13s
[2023-04-18T00:23:01.968Z] 89 actionable tasks: 35 executed, 54 up-to-date
[Pipeline] sh
[2023-04-18T00:23:04.619Z] + grep ^version= gradle.properties
[2023-04-18T00:23:04.619Z] + cut -d= -f 2
[Pipeline] dir
[2023-04-18T00:23:05.304Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-04-18T00:23:07.441Z] + mvn clean install -Dgpg.skip
[2023-04-18T00:23:09.191Z] [INFO] Scanning for projects...
[2023-04-18T00:23:10.126Z] [INFO] 

[2023-04-18T00:23:10.126Z] [INFO] Reactor Build Order:
[2023-04-18T00:23:10.126Z] [INFO] 
[2023-04-18T00:23:10.126Z] [INFO] Kafka Streams ::