[jira] [Created] (KAFKA-17158) Method 'cleanupLogs' can not delete old logSegements after invoking ALTER_REPLICA_LOG_DIRS

[fujian.zfj (Jira)]

fujian.zfj created KAFKA-17158:
--

 Summary: Method 'cleanupLogs' can not delete old logSegements 
after invoking ALTER_REPLICA_LOG_DIRS
 Key: KAFKA-17158
 URL: https://issues.apache.org/jira/browse/KAFKA-17158
 Project: Kafka
  Issue Type: Bug
  Components: log cleaner
Affects Versions: 2.6.3
Reporter: fujian.zfj


After invoking ALTER_REPLICA_LOG_DIRS, partition flow_pageview-9 will be moved 
from /data1 to /data2, while ReplicaAlterLogDirsThread is created, leader of 
partition flow_pageview-9 change from broker 58 to broker 36. After that, 
logSegements and indexes on /data2/flow_pageview-9 are no longer being deleted.

the config of topic flow_pageview is:

cleanup.policy=delete

retention.ms=360



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Josep Prat]

I'm starting with the process now.

Best,

On Thu, Jul 18, 2024 at 2:12 AM Greg Harris 
wrote:

> Hi all,
>
> We found a blocker in 3.8.0-rc1
> https://issues.apache.org/jira/browse/KAFKA-17150 that is now merged to
> trunk and backported to 3.8.
> Additionally I've merged and backported the lower-severity
> https://issues.apache.org/jira/browse/KAFKA-17148 to trunk, 3.8, and 3.7
> because Dmitry Werner provided a very prompt fix.
> Thanks to Chia-Ping, Chris, and Josep for prompt reviews. At this time I am
> not aware of any more blockers for 3.8.0 and we can proceed with rc2.
>
> Thanks!
> Greg
>
> On Wed, Jul 10, 2024 at 10:49 PM Josep Prat 
> wrote:
>
> > Hi Greg,
> >
> > I'll make sure the PR with the fix (
> > https://github.com/apache/kafka/pull/16565) gets merged today. Thanks
> for
> > bringing this up!
> >
> > Best,
> >
> > --
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Wed, Jul 10, 2024, 22:55 Greg Harris 
> > wrote:
> >
> > > Hi Josep,
> > >
> > > A contributor just raised a regression [1] that I think should be
> > addressed
> > > in 3.8.0 prior to the release.
> > >
> > > Summary: This change [2] causes multiple ERROR logs to appear during
> > worker
> > > startup when operators install other, unrelated plugins that package
> the
> > > Jackson library.
> > > Severity: Workers will continue to operate normally and the errors will
> > be
> > > cosmetic. Operators and automatic systems that watch logs may roll-back
> > > upgrades due to the perception of a severe problem.
> > > Impact: I found 12 out of 250 third-party plugins that package the
> > Jackson
> > > library and trigger this error on upgrade. This will almost certainly
> > > affect several users upon upgrades, and does not require obscure
> setups.
> > > Risk: The contributor has opened a simple fix PR [3], and I have
> verified
> > > that it addresses the problem, and can be merged tomorrow. As an
> > > alternative, we can revert the performance change completely [4] but I
> > want
> > > to avoid this.
> > >
> > > With the above, what would you like to do for this release? Merge the
> > fix,
> > > revert, or leave as-is?
> > >
> > > Thanks,
> > > Greg
> > >
> > > [1] https://issues.apache.org/jira/browse/KAFKA-17111
> > > [2] https://issues.apache.org/jira/browse/KAFKA-15996
> > > [3] https://github.com/apache/kafka/pull/16565
> > > [4] https://github.com/apache/kafka/pull/16568
> > >
> > > On Wed, Jul 10, 2024 at 7:37 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Dongjin,
> > > >
> > > > It's great to see you back!
> > > > I hope what we did with KIP-390 matches your expectations. Looking
> > > > forward to seeing the reboot of KIP-780.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Wed, Jul 10, 2024 at 4:21 PM Dongjin Lee 
> > wrote:
> > > > >
> > > > > Hi Josep,
> > > > >
> > > > > OMG, what happened while I could not be involved with the Kafka
> > > > community?
> > > > > Thanks for digging down the whole situation.
> > > > >
> > > > > @Mickael I greatly appreciate your effort in finalizing the
> feature.
> > It
> > > > > seems like I only have to re-boot the KIP-780.
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > On Tue, Jul 9, 2024 at 12:52 AM Josep Prat
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Dongjin,
> > > > > >
> > > > > > KIP-390 is part of the 3.8 release because the JIRA associated
> with
> > > it:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-7632 is closed as
> > > > resolved,
> > > > > > hence the KIP is declared done and ready. I did some digging,
> and I
> > > saw
> > > > > > that Mickael was the one doing the PR that closed the JIRA
> ticket:
> > > > > > https://github.com/apache/kafka/pull/15516
> > > > > > This means that the KIP work is merged and unfortunately it is
> now
> > > > quite
> > > > > > late to perform a rollback for this feature.
> > > > > >
> > > > > > @Mickael Maison  let me know if
> > anything I
> > > > > > mentioned is not accurate (as you were the one bringing the KIP
> to
> > > > > > completion).
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > On Mon, Jul 8, 2024 at 5:38 PM Dongjin Lee 
> > > wrote:
> > > > > >
> > > > > > > Hi Josep,
> > > > > > >
> > > > > > > Thanks for managing the 3.8 release. I have a request: could
> you
> > > > please
> > > > > > > move the KIP-390 into the 3.9 release?
> > > > > > >
> > > > > > > Here is the background: KIP-390 was adopted first but hasn't
> been
> > > > > > released
> > > > > > > for a long time. After some time, I proposed KIP-780 with
> further
> > > > > > > improvements and also corrected an obvious design error
> > > > > > > (`compression.level` → `compression.(gzip|lz4|zstd). level`),
> but
> > 

[VOTE] 3.8.0 RC2

[Josep Prat]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 3.8.0.

Some of the major features included in this release are:
* KIP-1028: Docker Official Image for Apache Kafka
* KIP-974: Docker Image for GraalVM based Native Kafka Broker
* KIP-1036: Extend RecordDeserializationException exception
* KIP-1019: Expose method to determine Metric Measurability
* KIP-1004: Enforce tasks.max property in Kafka Connect
* KIP-989: Improved StateStore Iterator metrics for detecting leaks
* KIP-993: Allow restricting files accessed by File and Directory
ConfigProviders
* KIP-924: customizable task assignment for Streams
* KIP-813: Shareable State Stores
* KIP-719: Deprecate Log4J Appender
* KIP-390: Support Compression Level
* KIP-1018: Introduce max remote fetch timeout config for
DelayedRemoteFetch requests
* KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission
* KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace
kafka.serializer.Decoder
* KIP-899: Allow producer and consumer clients to rebootstrap

Release notes for the 3.8.0 release:
https://home.apache.org/~jlprat/kafka-3.8.0-rc2/RELEASE_NOTES.html

 Please download, test and vote by *

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~jlprat/kafka-3.8.0-rc2/


* Docker release artifact to be voted upon(apache/kafka-native is supported
from 3.8+ release.):
apache/kafka:3.8.0-rc2
apache/kafka-native:3.8.0-rc2

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~jlprat/kafka-3.8.0-rc2/javadoc/

* Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag:
https://github.com/apache/kafka/releases/tag/3.8.0-rc2

* Documentation:
https://kafka.apache.org/38/documentation.html

* Protocol:
https://kafka.apache.org/38/protocol.html

* Successful Jenkins builds for the 3.8 branch:
Unit/integration tests:
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/70
(latest one is still building, but PR passed without related failures)
System tests: (Same as before)
https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html

* Successful Docker Image Github Actions Pipeline for 3.8 branch:
Docker Build Test Pipeline (JVM):
https://github.com/apache/kafka/actions/runs/9987484582
Docker Build Test Pipeline (Native):
https://github.com/apache/kafka/actions/runs/9987487177



Thanks,

-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Resolved] (KAFKA-17141) "DescribeTopicPartitions API is not supported" warning message confuses users

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17141.

Fix Version/s: 3.9.0
   Resolution: Fixed

> "DescribeTopicPartitions API is not supported" warning message confuses users
> -
>
> Key: KAFKA-17141
> URL: https://issues.apache.org/jira/browse/KAFKA-17141
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: PoAn Yang
>Priority: Major
>  Labels: newbie
> Fix For: 3.9.0
>
>
> When running describeTopics admin API, we'll try to invoke 
> DescribeTopicPartitions request, and fallback to metadata request if not 
> supported, and then log a warning message:
> {code:java}
> 2024-07-15 19:03:34 WARN  KafkaAdminClient:2293 - [AdminClient 
> clientId=adminclient-17] The DescribeTopicPartitions API is not supported, 
> using Metadata API to describe topics.{code}
>  
> When seeing this warning message, users might think something is wrong with 
> this call. We should try to improve it. Maybe:
> 1. log it with an INFO level or DEBUG level
> 2. Make the message clear to not to confuse users
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #71

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-17135) Add unit test for `ProducerStateManager#readSnapshot` and `ProducerStateManager#writeSnapshot`

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17135.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Add unit test for `ProducerStateManager#readSnapshot` and 
> `ProducerStateManager#writeSnapshot`
> --
>
> Key: KAFKA-17135
> URL: https://issues.apache.org/jira/browse/KAFKA-17135
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 3.9.0
>
>
> We are going to introduce generated code to `ProducerStateManager`, so it 
> would be nice to increase the test converge for now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15773) Group protocol configuration should be validated

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-15773.

Resolution: Fixed

> Group protocol configuration should be validated
> 
>
> Key: KAFKA-15773
> URL: https://issues.apache.org/jira/browse/KAFKA-15773
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: PoAn Yang
>Priority: Minor
>  Labels: kip-848-client-support
> Fix For: 3.9.0
>
>
> If the user specifies using the generic group, or not specifying the 
> group.protocol config at all, we should invalidate all group.remote.assignor
>  
> If group.local.assignor and group.remote.assignor are both configured, we 
> should also invalidate the configuration
>  
> This is an optimization/user experience improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 3.6 #1

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 65 lines...]
[2024-07-18T10:39:41.098Z] Cloning with configured refspecs honoured and 
without tags
[2024-07-18T10:39:41.586Z] ERROR: Error cloning remote repo 'origin'
[2024-07-18T10:39:41.586Z] hudson.plugins.git.GitException: Command "git fetch 
--no-tags --force --progress -- https://github.com/mimaison/kafka.git 
+refs/heads/3.6:refs/remotes/origin/3.6" returned status code 128:
[2024-07-18T10:39:41.586Z] stdout: 
[2024-07-18T10:39:41.586Z] stderr: fatal: couldn't find remote ref 
refs/heads/3.6
[2024-07-18T10:39:41.586Z] 
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.Request$2.run(Request.java:377)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.lang.Thread.run(Thread.java:829)
[2024-07-18T10:39:41.586Z]  Suppressed: 
hudson.remoting.Channel$CallSiteStackTrace: Remote call to builds27
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2024-07-18T10:39:41.586Z]  at 
hudson.remoting.Channel.call(Channel.java:1003)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
[2024-07-18T10:39:41.586Z]  at 
jdk.internal.reflect.GeneratedMethodAccessor954.invoke(Unknown Source)
[2024-07-18T10:39:41.586Z]  at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
[2024-07-18T10:39:41.586Z]  at 
com.sun.proxy.$Proxy232.execute(Unknown Source)
[2024-07-18T10:39:41.586Z]  at 
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)
[2024-07-18T10:39:41.586Z]  at 
hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
[2024-07-18T10:39:41.586Z]  at 
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2024-07-18T10:39:41.586Z]  at 
java.base/java.lang.Thread.run(Thread.java:829)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipelin

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 3.8 #1

[Apache Jenkins Server]

See 


Changes:


--
Branch indexing
Connecting to https://api.github.com using 76764/** (ASF Cloudbees Jenkins 
ci-builds)
Obtained Jenkinsfile from 017228042716f9e76e4064a72b7e5070ce591335
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] parallel
[Pipeline] { (Branch: JDK 8 and Scala 2.12)
[Pipeline] { (Branch: JDK 11 and Scala 2.13)
[Pipeline] { (Branch: JDK 17 and Scala 2.13)
[Pipeline] { (Branch: JDK 21 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 8 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 11 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 17 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 21 and Scala 2.13)
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[2024-07-18T10:39:43.857Z] Running on builds29 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
[2024-07-18T10:39:44.048Z] Running on builds44 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8
[2024-07-18T10:39:44.054Z] Running on builds27 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.8
[2024-07-18T10:39:44.421Z] Running on builds36 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8
[Pipeline] {
[Pipeline] {
[Pipeline] {
[Pipeline] checkout
[Pipeline] checkout
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] envVarsForTool
[Pipeline] tool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] envVarsForTool
[Pipeline] tool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
[2024-07-18T10:39:45.703Z] + ./retry_zinc ./gradlew -PscalaVersion=2.13 clean 
check -x test --profile --continue -PxmlSpotBugsReport=true 
-PkeepAliveMode=session
[2024-07-18T10:39:45.703Z] 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8@tmp/durable-bf68bd62/script.sh:
 2: ./retry_zinc: not found
[Pipeline] sh
[2024-07-18T10:39:47.341Z] + ./retry_zinc ./gradlew -PscalaVersion=2.13 clean 
check -x test --profile --continue -PxmlSpotBugsReport=true 
-PkeepAliveMode=session
[2024-07-18T10:39:47.341Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.8@tmp/durable-69224ed0/script.sh:
 2: ./retry_zinc: not found
[Pipeline] sh
[2024-07-18T10:39:48.200Z] + ./retry_zinc ./gradlew -PscalaVersion=2.12 clean 
check -x test --profile --continue -PxmlSpotBugsReport=true 
-PkeepAliveMode=session
[2024-07-18T10:39:48.200Z] 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8@tmp/durable-5d9fbe4e/script.sh:
 2: ./retry_zinc: not found
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] }
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 11 and Scala 2.13
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.12
[Pipeline] }
Failed in branch JDK 17 and Scala 2.13
[2024-07-18T10:39:50.224Z] + ./retry_zinc ./gradlew -PscalaVersion=2.13 clean 
check -x test --profile --continue -PxmlSpotBugsReport=true 
-PkeepAliveMode=session
[2024-07-18T10:39:50.224Z] 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.8@tmp/durable-8bac78b3/script.sh:
 2: ./retry_zinc: not found
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 21 and Scala 2.13
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on builds40 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.8
[Pipeline] {
[Pipeline] step

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 3.5 #1

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 9 lines...]
[Pipeline] { (Branch: JDK 17 and Scala 2.13)
[Pipeline] { (Branch: JDK 8 and Scala 2.13)
[Pipeline] { (Branch: JDK 11 and Scala 2.12)
[Pipeline] { (Branch: JDK 17 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 8 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 11 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 17 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 8 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 11 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 17 and Scala 2.12)
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[2024-07-18T10:39:38.571Z] Running on builds29 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[Pipeline] {
[2024-07-18T10:39:38.628Z] Running on builds44 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[2024-07-18T10:39:38.632Z] Running on builds58 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[2024-07-18T10:39:38.645Z] Running on builds47 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[Pipeline] checkout
[Pipeline] {
[Pipeline] {
[Pipeline] {
[2024-07-18T10:39:38.986Z] Running on builds32 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[2024-07-18T10:39:39.077Z] The recommended git tool is: NONE
[2024-07-18T10:39:39.162Z] Running on builds24 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.5
[Pipeline] checkout
[Pipeline] checkout
[2024-07-18T10:39:39.477Z] The recommended git tool is: NONE
[Pipeline] checkout
[Pipeline] {
[Pipeline] {
[2024-07-18T10:39:39.541Z] The recommended git tool is: NONE
[2024-07-18T10:39:39.690Z] The recommended git tool is: NONE
[2024-07-18T10:39:39.889Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:39.890Z] Wiping out workspace first.
[Pipeline] checkout
[Pipeline] checkout
[2024-07-18T10:39:40.126Z] The recommended git tool is: NONE
[2024-07-18T10:39:40.168Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:40.168Z] Wiping out workspace first.
[2024-07-18T10:39:40.337Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:40.338Z] Wiping out workspace first.
[2024-07-18T10:39:40.408Z] The recommended git tool is: NONE
[2024-07-18T10:39:40.561Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:40.562Z] Wiping out workspace first.
[2024-07-18T10:39:40.731Z] Cloning the remote Git repository
[2024-07-18T10:39:40.732Z] Cloning with configured refspecs honoured and 
without tags
[2024-07-18T10:39:41.139Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:41.140Z] Wiping out workspace first.
[2024-07-18T10:39:41.160Z] Cloning the remote Git repository
[2024-07-18T10:39:41.160Z] Cloning with configured refspecs honoured and 
without tags
[2024-07-18T10:39:41.202Z] Cloning the remote Git repository
[2024-07-18T10:39:41.202Z] Cloning with configured refspecs honoured and 
without tags
[2024-07-18T10:39:41.412Z] Cloning the remote Git repository
[2024-07-18T10:39:41.412Z] Cloning with configured refspecs honoured and 
without tags
[2024-07-18T10:39:40.990Z] Cloning repository 
https://github.com/mimaison/kafka.git
[2024-07-18T10:39:40.991Z]  > git init 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.5 # timeout=10
[2024-07-18T10:39:41.002Z] Fetching upstream changes from 
https://github.com/mimaison/kafka.git
[2024-07-18T10:39:41.002Z]  > git --version # timeout=10
[2024-07-18T10:39:41.008Z]  > git --version # 'git version 2.34.1'
[2024-07-18T10:39:41.008Z] using GIT_ASKPASS to set credentials ASF Cloudbees 
Jenkins ci-builds
[2024-07-18T10:39:41.009Z]  > git fetch --no-tags --force --progress -- 
https://github.com/mimaison/kafka.git +refs/heads/3.5:refs/remotes/origin/3.5 # 
timeout=10
[2024-07-18T10:39:41.513Z] using credential ASF_Cloudbees_Jenkins_ci-builds
[2024-07-18T10:39:41.514Z] Wiping out workspace first.
[2024-07-18T10:39:41.394Z] Cloning repository 
https://github.com/mimaison/kafka.git
[2024-07-18T10:39:41.394Z]  > git init 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.5 # timeout=10
[2024-07-18T10:39:41.505Z] Fetching upstream changes from 
https://github.com/mimaison/kafka.git
[2024-07-1

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 3.1 #1

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 3 lines...]
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] parallel
[Pipeline] { (Branch: JDK 8 and Scala 2.12)
[Pipeline] { (Branch: JDK 11 and Scala 2.13)
[Pipeline] { (Branch: JDK 17 and Scala 2.13)
[Pipeline] { (Branch: ARM)
[Pipeline] { (Branch: JDK 8 and Scala 2.13)
[Pipeline] { (Branch: JDK 11 and Scala 2.12)
[Pipeline] { (Branch: JDK 17 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 8 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 11 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 17 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (ARM)
[Pipeline] stage
[Pipeline] { (JDK 8 and Scala 2.13)
[Pipeline] stage
[Pipeline] { (JDK 11 and Scala 2.12)
[Pipeline] stage
[Pipeline] { (JDK 17 and Scala 2.12)
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 2 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[2024-07-18T10:39:47.319Z] Running on builds23 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
[2024-07-18T10:39:49.111Z] Running on builds49 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
[2024-07-18T10:39:49.212Z] + ./gradlew -PscalaVersion=2.12 clean compileJava 
compileScala compileTestJava compileTestScala spotlessScalaCheck checkstyleMain 
checkstyleTest spotbugsMain rat --profile --no-daemon --continue 
-PxmlSpotBugsReport=true
[2024-07-18T10:39:49.212Z] 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.1@tmp/durable-51441c46/script.sh:
 2: ./gradlew: not found
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.12
[2024-07-18T10:39:50.996Z] Running on builds44 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[2024-07-18T10:39:51.056Z] Running on builds29 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[Pipeline] {
[Pipeline] sh
[2024-07-18T10:39:51.330Z] + ./gradlew -PscalaVersion=2.13 clean compileJava 
compileScala compileTestJava compileTestScala spotlessScalaCheck checkstyleMain 
checkstyleTest spotbugsMain rat --profile --no-daemon --continue 
-PxmlSpotBugsReport=true
[2024-07-18T10:39:51.330Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.1@tmp/durable-1b1dbb5c/script.sh:
 2: ./gradlew: not found
[2024-07-18T10:39:51.376Z] Running on builds27 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[2024-07-18T10:39:52.719Z] Running on builds36 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.1
[Pipeline] checkout
[Pipeline] }
[Pipeline] {
[Pipeline] {
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] checkout
[Pipeline] checkout
[Pipeline] // node
[Pipeline] }
[Pipeline] withEnv
[Pipeline] {
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] tool
[Pipeline] // timeout
[Pipeline] }
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 11 and Scala 2.13
[Pipeline] tool
[Pipeline] sh
[2024-07-18T10:39:53.301Z] Still waiting to schedule task
[2024-07-18T10:39:53.302Z] There are no nodes with the label ‘arm4’
[2024-07-18T10:39:53.765Z] + ./gradlew -PscalaVersion=2.13 clean compileJava 
compileScala compileTestJava compileTestScala spotlessScalaCheck checkstyleMain 
checkstyleTest spotbugsMain rat --profile --no-daemon --continue 
-PxmlSpotBugsReport=true
[2024-07-18T10:39:53.765Z] 
/home/jenkins

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 3.0 #1

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 196 lines...]
[2024-07-18T10:39:42.875Z]  at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2024-07-18T10:39:42.875Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
[2024-07-18T10:39:42.875Z]  at 
com.sun.proxy.$Proxy232.execute(Unknown Source)
[2024-07-18T10:39:42.875Z]  at 
hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)
[2024-07-18T10:39:42.875Z]  at 
hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
[2024-07-18T10:39:42.875Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
[2024-07-18T10:39:42.875Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
[2024-07-18T10:39:42.875Z]  at 
org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
[2024-07-18T10:39:42.875Z]  at 
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2024-07-18T10:39:42.875Z]  at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[2024-07-18T10:39:42.875Z]  at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2024-07-18T10:39:42.875Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2024-07-18T10:39:42.875Z]  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2024-07-18T10:39:42.875Z]  at 
java.base/java.lang.Thread.run(Thread.java:829)
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 8 and Scala 2.12
[2024-07-18T10:39:42.781Z] Cloning repository 
https://github.com/mimaison/kafka.git
[2024-07-18T10:39:42.782Z]  > git init 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_3.0 # timeout=10
[2024-07-18T10:39:42.792Z] Fetching upstream changes from 
https://github.com/mimaison/kafka.git
[2024-07-18T10:39:42.792Z]  > git --version # timeout=10
[2024-07-18T10:39:42.796Z]  > git --version # 'git version 2.34.1'
[2024-07-18T10:39:42.796Z] using GIT_ASKPASS to set credentials ASF Cloudbees 
Jenkins ci-builds
[2024-07-18T10:39:42.798Z]  > git fetch --no-tags --force --progress -- 
https://github.com/mimaison/kafka.git +refs/heads/3.0:refs/remotes/origin/3.0 # 
timeout=10
[2024-07-18T10:39:43.128Z] ERROR: Error cloning remote repo 'origin'
[2024-07-18T10:39:43.128Z] hudson.plugins.git.GitException: Command "git fetch 
--no-tags --force --progress -- https://github.com/mimaison/kafka.git 
+refs/heads/3.0:refs/remotes/origin/3.0" returned status code 128:
[2024-07-18T10:39:43.128Z] stdout: 
[2024-07-18T10:39:43.128Z] stderr: fatal: couldn't find remote ref 
refs/heads/3.0
[2024-07-18T10:39:43.128Z] 
[2024-07-18T10:39:43.128Z] ERROR: Error cloning remote repo 'origin'
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:871)
[2024-07-18T10:39:43.128Z] hudson.plugins.git.GitException: Command "git fetch 
--no-tags --force --progress -- https://github.com/mimaison/kafka.git 
+refs/heads/3.0:refs/remotes/origin/3.0" returned status code 128:
[2024-07-18T10:39:43.128Z] stdout: 
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
[2024-07-18T10:39:43.128Z] stderr: fatal: couldn't find remote ref 
refs/heads/3.0
[2024-07-18T10:39:43.128Z] 
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
[2024-07-18T10:39:43.128Z]  at 
hudson.remoting.UserRequest.perform(UserRequest.java:211)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
[2024-07-18T10:39:43.128Z]  at 
hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2024-07-18T10:39:43.128Z]  at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java

Build failed in Jenkins: Kafka » Kafka PowerPC Daily » 2.8 #1

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 39 lines...]
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 8 hr 0 min
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[Pipeline] node
[2024-07-18T10:39:44.267Z] Running on builds40 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[2024-07-18T10:39:44.294Z] Running on builds48 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[Pipeline] {
[Pipeline] tool
[Pipeline] checkout
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
[2024-07-18T10:39:44.336Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8/.gradle
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
[2024-07-18T10:39:44.367Z] Running in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_2.8/.gradle
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] }
[Pipeline] // dir
[2024-07-18T10:39:44.518Z] Running on builds23 in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[Pipeline] sh
[2024-07-18T10:39:44.636Z] Running on builds28 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[Pipeline] {
[Pipeline] }
[Pipeline] {
[Pipeline] // dir
[Pipeline] sh
[2024-07-18T10:39:46.073Z] Running on builds27 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[2024-07-18T10:39:46.350Z] + ./gradlew -version
[2024-07-18T10:39:46.351Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8@tmp/durable-eab4804b/script.sh:
 1: ./gradlew: not found
[Pipeline] checkout
[Pipeline] checkout
[Pipeline] {
[Pipeline] checkout
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] tool
[Pipeline] tool
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] envVarsForTool
[Pipeline] tool
[Pipeline] envVarsForTool
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] envVarsForTool
[Pipeline] dir
[2024-07-18T10:39:46.748Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8/.gradle
[Pipeline] {
[Pipeline] dir
[2024-07-18T10:39:46.749Z] Running in 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_2.8/.gradle
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] deleteDir
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
[2024-07-18T10:39:46.763Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8/.gradle
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh
[2024-07-18T10:39:47.372Z] Running on builds49 in 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8
[2024-07-18T10:39:47.387Z] + ./gradlew -version
[2024-07-18T10:39:47.388Z] 
/home/jenkins/workspace/Kafka_Kafka_PowerPC_Daily_2.8@tmp/durable-c3ecd197/script.sh:
 1: ./gradlew: not found
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] {
[Pipeline] // withEnv
[Pipeline] // dir
[Pipeline] // dir
[Pipeline] }
[Pipeline] sh
[2024-07-18T10:39:48.747Z] + ./gradlew -version
[2024-07-18T10:39:48.747Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8@tmp/durable-ad96a73e/script.sh:
 1: ./gradlew: not found
[Pipeline] sh
[2024-07-18T10:39:50.670Z] + ./gradlew -version
[2024-07-18T10:39:50.670Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_Kafka_PowerPC_Daily_2.8@tmp/durable-7b0f4eea/script.sh:
 1: ./gradlew: not found
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] checkout
[Pipeline] // node
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] // withEnv
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // timeout
[Pipeline] // node
[Pipeline] // node
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] withEnv
[Pipeline] {
[Pipeline] // stage
[Pipeline] // timestamps
[Pipeline] // timestamps
[Pipeline] }
Failed in branch JDK 15 and Scala 2.13
[Pipeline] }
[Pipeline] }
[Pipeline] tool
[Pipeline] // timeout
[Pipeline] // timeout
[Pipeline] }
[Pipeline] }
[Pipeline] envVarsForTool
[Pipeline] // stage
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 11 and Scala 2.13
[Pipeline] }
Failed in branch JDK 8 and Scala 2.13
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
[2024-07-18T10:39:51.394Z] Running in

[jira] [Resolved] (KAFKA-17142) Fix deadlock caused by LogManagerTest#testLogRecoveryMetrics

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17142.

Fix Version/s: 3.9.0
 Assignee: PoAn Yang  (was: Chia-Ping Tsai)
   Resolution: Fixed

> Fix deadlock caused by LogManagerTest#testLogRecoveryMetrics
> 
>
> Key: KAFKA-17142
> URL: https://issues.apache.org/jira/browse/KAFKA-17142
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Major
> Fix For: 3.9.0
>
> Attachments: log
>
>
> `testLogRecoveryMetrics` uses mock scheduler to create UnifiedLog [0], and 
> the mock scheduler does NOT have true thread poll, and that is the root cause!
> We create recovery threads [1] for each data folder, and they will pass the 
> action: `writeToFileForTruncation` to scheduler [2]. The action requires the 
> read lock [3], so the deadlock is produced when one thread executes the 
> action from another thread. For example:
> 1. thread_a is handling dir_a, and it holds the writelock_a
> 2. thread_a pass action_a: `writeToFileForTruncation` to mock scheduler
> 3. thread_b is handling dir_b, and it holds the writelock_b
> 4. thread_b pass action_b: `writeToFileForTruncation` to mock scheduler
> 5. thread_b (holding writelock_b) handle the action_a, so it requires the 
> readlock_a
> 6. thread_a (holding writelock_a) handle the action_b, so it requires the 
> readlock_b
> so lucky we have a deadlock :cry
> This deadlock happens due to the mock scheduler, so that is a issue belonging 
> to test. We can fix it by a simple solution - add some delay when creating 
> next UnifiedLog
> Or we can do a bit refactor to production code: create snapshot of `epochs` 
> when we are holding write lock! That means `writeToFileForTruncation` does 
> not take lock anymore. For example:
> {code:java}
> public void truncateFromStartAsyncFlush(long startOffset) {
> lock.writeLock().lock();
> try {
> List removedEntries = truncateFromStart(epochs, 
> startOffset);
> if (!removedEntries.isEmpty()) {
> ...
> List entries = new ArrayList<>(epochs.values());
> scheduler.scheduleOnce("leader-epoch-cache-flush-" + 
> topicPartition, () -> checkpoint.writeForTruncation(entries));
> ...
> }
> } finally {
> lock.writeLock().unlock();
> }
> }
> {code}
> [0] 
> https://github.com/apache/kafka/blob/808498e9391dab292a7ccd8a0bf3713f444f9d2f/core/src/test/scala/unit/kafka/log/LogManagerTest.scala#L968
> [1] 
> https://github.com/apache/kafka/blob/808498e9391dab292a7ccd8a0bf3713f444f9d2f/core/src/main/scala/kafka/log/LogManager.scala#L434
> [2] 
> https://github.com/apache/kafka/blob/808498e9391dab292a7ccd8a0bf3713f444f9d2f/storage/src/main/java/org/apache/kafka/storage/internals/epoch/LeaderEpochFileCache.java#L351
> [3] 
> https://github.com/apache/kafka/blob/808498e9391dab292a7ccd8a0bf3713f444f9d2f/storage/src/main/java/org/apache/kafka/storage/internals/epoch/LeaderEpochFileCache.java#L528
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3119

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16975) The error message of creating `__cluster_metadata` should NOT be "Authorization failed"

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16975.

Fix Version/s: 3.9.0
   Resolution: Fixed

> The error message of creating `__cluster_metadata` should NOT be 
> "Authorization failed" 
> 
>
> Key: KAFKA-16975
> URL: https://issues.apache.org/jira/browse/KAFKA-16975
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 3.9.0
>
>
> The error message of creating "__cluster_metadata" by admin is "Authorization 
> failed.". That could confuse users that it implies you "can" create it in 
> using root. However, the fact is that we DISALLOW users to create it as a 
> regular topic even though you are the boss :)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

[Kamal Chandraprakash]

Hi Harry,

Thanks for the updates!

Yes, the proposed metric looks good.

If the user runs the kafka-reassign-partitions script with throttle set,
then the static throttle gets overwritten
until the reassignment gets completed. Can you clarify this on the KIP?

--
Kamal



On Sun, Jul 14, 2024 at 9:59 PM Harry Fallows
 wrote:

> Hi Kamal,
>
> Thank you for reading KIP-1051!
>
> Yes, it's true that it can impact regular replication traffic. However,
> network throughput is bounded so regardless of whether we allow it as a
> config in Kafka or not, there is always a chance that replication traffic
> will get throttled. Having it as a config will at least ensure that the
> entire bandwidth is not taken up by replication traffic.
>
> I agree, the nature of the leader replication throttling is dependent on
> how many followers there are, however, I don't think it's dependent on the
> partition assignment strategy or the number of brokers; it should only be
> dependent on the replication factor. I think it's key to point out here
> that these configurations do not need to be "optimised" for use cases with
> different replication factors, they just need to be set to match the
> infrastructure that they are deployed in. For example if you have a maximum
> network bandwidth of 200MB/s and a replication factor of 3, you may set
> follower.replication.throttled.replicas to 150MB/s, to reserve some
> bandwidth for other traffic (e.g. producing and consuming). In this case,
> if you start with all replicas in sync, I don't think it's possible for the
> follower throttling to be the sole cause of a replica falling out of sync.
> It may be the case that it takes longer for an out-of-sync replica to
> become in sync, but in that case the replication throttling just serves to
> mitigate other traffic from getting throttled (e.g. producer traffic to a
> different partition). Even so, it is possible that misconfiguring these
> values could cause issues, so the potential consequences should be clearly
> documented.
>
> I think the concern about producing spikes causing ISR issues is only an
> issue if these values are poorly configured. I think in general if these
> values are always configured as >=
> (replicationFactor/(replicationFactor+1))*maxBandwidth (e.g. like the above
> example: 3/(3+1) * 200 = 150), then even if 100% of the non-replication
> traffic is producer traffic, all followers should be able to stay in sync.
>
> I like the idea of emitting a metric for when a quota is breached, what do
> you think about having it as a gauge for number of partitions that are
> currently leader of follower throttled (similar to the URP metric)?
>
> Kind regards,
> Harry
>
> On Thursday, 11 July 2024 at 19:02, Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Harry Fallows,
> >
> > Thanks for the KIP!
> >
> > I went over both the KIP-1051 and KIP-1009. Assuming that the
> > leader.replication.throttled.replicas
> > and follower.replication.throttled.replicas are set to Wildcard (*) to
> > apply for all the partitions in the
> > broker. If we set a static value for leader and follower replication
> > throttled rate, then it might impact
> > the normal replication traffic.
> >
> > Throttling rate depends on the number of brokers in the cluster. If the
> > cluster contains 100+ brokers, then
> > the leader.replication.throttled.rate is shared across all the followers.
> > The number of followers reading
> > data from the leader depends on the partition assignment strategy. If the
> > leader replication throttle is breached,
> > then the follower might fail to catch-up with the leader.
> >
> > If there are sudden spikes in a specific set of topics/partitions in the
> > cluster, then the replicas might fail to join
> > the isr and can impact the cluster reliability. If we are going with this
> > proposal, then we may also have to emit
> > a metric to inform the administrator that the leader/follower replication
> > quota is breached.
> >
> > --
> > Kamal
> >
> > On Thu, Jul 4, 2024 at 8:10 PM Harry Fallows
> > harryfall...@protonmail.com.invalid wrote:
> >
> > > Hi everyone,
> > >
> > > Bumping this one last time before I call a vote. Please take a look if
> > > you're interested in replication throttling and/or static/dynamic
> config.
> > >
> > > Kind regards,
> > > Harry
> > >
> > > On Thursday, 13 June 2024 at 19:39, Harry Fallows <
> > > harryfall...@protonmail.com.INVALID> wrote:
> > >
> > > > Hi Hector,
> > > >
> > > > I did see your colleague's KIP, and I actually mentioned it in the
> KIP
> > > > that I have written. As I see it, both of these KIPs move towards
> more
> > > > easily configurable replication throttling and both should be
> implemented.
> > > > KIP-1009 makes it easier to enable throttling and KIP-1051 makes it
> easier
> > > > to apply a throttle rate. I did try to look at supporting KIP-1009
> in the
> > > > discussion thread, however, I only subscribed to the mailing list
> after it
>

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

[Andrew Schofield]

Hi Lucas and Bruno,

Thanks for the great KIP.

I've read through the document and have some initial comments.

AS1: I suppose that there is a new o.a.k.common.GroupType.STREAMS enumeration
constant. This is a change to the public interface and should be called out.

AS2: Since streams groups are no longer consumer groups, how does the user
manipulate them, observe lag and so on? Will you add `kafka-streams-groups.sh`
or extend `kafka-streams-application-reset.sh`? Of course, KIP-1043 can easily
be extended to support streams groups, but that only lets the user see the
groups, not manipulate them.

AS3: I wonder whether the streams group state of UNINITIALIZED would be
better expressed as INITIALIZING.

AS4: In StreamsGroupInitializeRequest, Topology[].SourceTopicRegex should
be nullable.

AS5: Why does StreamsGroupInitialize require CREATE permission on the
cluster resource? I imagine that this is one of the ways that the request might
be granted permission to create the StateChangelogTopics and
RepartitionSourceTopics, but if it is granted permission to create those topics
with specific ACLs, would CREATE on the cluster resource still be required?

AS6: StreamsGroupInitialize can also fail with TOPIC_AUTHORIZATION_FAILED
and (subject to AS5) CLUSTER_AUTHORIZATION_FAILED.

AS7: A tiny nit. You've used TopologyID (capitals) in 
StreamsGroupHeartbeatRequest
and a few others, but in all other cases the fields which are ids are spelled 
Id.
I suggest TopologyId.

Also, "interal" is probably meant to be "interval”.

AS8: For consumer groups, the `group.consumer.assignors` configuration is a
list of class names. The assignors do have names too, but the configuration 
which
enables them is in terms of class names. I wonder whether the broker’s
group.streams.assignor could actually be `group.streams.assignors` and specified
as a list of the class names of the supplied assignors. I know you're not 
supporting
other assignors yet, but when you do, I expect you would prefer to have used 
class
names from the start.

The use of assignor names in the other places looks good to me.

AS9: I'd find it really helpful to have a bit of a description about the 
purpose and
lifecycle of the 9 record types you've introduced on the __consumer_offsets 
topic.
I did a cursory review but without really understanding what's written when,
I can't do a thorough job of reviewing.

AS10: In the definitions of the record keys, such as
StreamsGroupCurrentMemberAssignmentKey, the versions of the fields must
match the versions of the types.

Thanks,
Andrew

> On 12 Jul 2024, at 09:04, Lucas Brutschy  
> wrote:
>
> Hi all,
>
> I would like to start a discussion thread on KIP-1071: Streams
> Rebalance Protocol. With this KIP, we aim to bring the principles laid
> down by KIP-848 to Kafka Streams, to make rebalances more reliable and
> scalable, and make Kafka Streams overall easier to deploy and operate.
> The KIP proposed moving the assignment logic to the broker, and
> introducing a dedicated group type and dedicated RPCs for Kafka
> Streams.
>
> The KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1071%3A+Streams+Rebalance+Protocol
>
> This is joint work with Bruno Cadonna.
>
> Please take a look and let us know what you think.
>
> Best,
> Lucas

[jira] [Created] (KAFKA-17159) Make sure kafka-cluster-test-kit-executor get down when closing KafkaClusterTestKit

[PoAn Yang (Jira)]

PoAn Yang created KAFKA-17159:
-

 Summary: Make sure kafka-cluster-test-kit-executor get down when 
closing KafkaClusterTestKit
 Key: KAFKA-17159
 URL: https://issues.apache.org/jira/browse/KAFKA-17159
 Project: Kafka
  Issue Type: Sub-task
Reporter: PoAn Yang
Assignee: PoAn Yang


When trying to remove `TestUtils.waitForCondition` in 
[https://github.com/apache/kafka/pull/16499], kraft may have thread leak.

We can use ThreadUtils#shutdownExecutorServiceQuietly [0] to replace the block 
in KafkaClusterTestKie#close [1].

 

[0] 
[https://github.com/apache/kafka/blob/f595802cc752ed01dc74e9ab932209fe25a9d10b/clients/src/main/java/org/apache/kafka/common/utils/ThreadUtils.java#L71-L92]

[1] 
[https://github.com/apache/kafka/blob/f595802cc752ed01dc74e9ab932209fe25a9d10b/core/src/test/java/kafka/testkit/KafkaClusterTestKit.java#L640-L643]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17160) Always import `mockito-inline` if the mockito version is 4.x

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17160:
--

 Summary: Always import `mockito-inline` if the mockito version is 
4.x
 Key: KAFKA-17160
 URL: https://issues.apache.org/jira/browse/KAFKA-17160
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Currently, we pick up mockito artifact according to java version[0]. It can 
cause mock error on final class when we run JDK 11+ with scala 2.12, since it 
imports the 4.x mockito-core which does not include features of mockito-line. 

In short, the artifact should be based on mockito version rather than java 
version. Otherwise, some tests (for example, `PCollectionsImmutableSetTest`) 
never pass

[0] https://github.com/apache/kafka/blob/trunk/gradle/dependencies.gradle#L74



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17161) REST Jersey metric names are inconsistent

[Doug Hoard (Jira)]

Doug Hoard created KAFKA-17161:
--

 Summary: REST Jersey metric names are inconsistent
 Key: KAFKA-17161
 URL: https://issues.apache.org/jira/browse/KAFKA-17161
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.6.2
Reporter: Doug Hoard


*Summary*

When exporting JMX metrics from a server, REST Jersey metric names are 
inconsistent, resulting in two metrics different metrics with the same value.
{code:java}
v3.topics-partitions-reassignment.list.request-total
v3.topics.partitions-reassignment.list.request-total{code}
Dash (-) versus period ({{{}.{}}}) between {{topics}} and {{partitions}}

This most likely affects older versions.

*Test scenario*

Install Kafka 3.6.2 with the Prometheus JMX Exporter v1.0.1 and collect 
metrics. The exporter adds an "_objectname" for metrics that map to the same 
Prometheus name.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17162) DefaultTaskManagerTest may leak AwaitingRunnable thread

[Ao Li (Jira)]

Ao Li created KAFKA-17162:
-

 Summary: DefaultTaskManagerTest may leak AwaitingRunnable thread
 Key: KAFKA-17162
 URL: https://issues.apache.org/jira/browse/KAFKA-17162
 Project: Kafka
  Issue Type: Bug
  Components: streams, unit tests
Affects Versions: 3.9.0
Reporter: Ao Li


The `DefaultTaskManagerTest#shouldReturnFromAwaitOnInterruption` will fail with 
the following patch:

{code}
```
diff --git 
a/streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
 
b/streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
index 5d2db3c279..b87a82b85b 100644
--- 
a/streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
+++ 
b/streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManager.java
@@ -348,6 +348,10 @@ public class DefaultTaskManager implements TaskManager {
 }
 
 private  T returnWithTasksLocked(final Supplier action) {
+try {
+Thread.sleep(1000);
+} catch (final Exception e) {
+}
 tasksLock.lock();
 try {
 return action.get();
diff --git 
a/streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java
 
b/streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java
index 98065eae7d..0d8dde7156 100644
--- 
a/streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java
+++ 
b/streams/src/test/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskManagerTest.java
@@ -114,6 +114,10 @@ public class DefaultTaskManagerTest {
 @Override
 public void run() {
 while (!shutdownRequested.get()) {
+try {
+Thread.sleep(1000);
+} catch (final Exception e) {
+}
 try {
 taskManager.awaitProcessableTasks();
 } catch (final InterruptedException ignored) {
@@ -151,6 +155,8 @@ public class DefaultTaskManagerTest {
 assertTrue(awaitingRunnable.awaitDone.await(VERIFICATION_TIMEOUT, 
TimeUnit.MILLISECONDS));
 
 awaitingRunnable.shutdown();
+Thread.sleep(5000);
+assertFalse(awaitingThread.isAlive());
 }
 
 @Test
```
{code}

Because awatingThread is left unclosed because it was waiting for the signal

{code}
"Thread-3" #25 [26371] prio=5 os_prio=31 cpu=9.68ms elapsed=74.89s 
tid=0x0001250d8600 nid=26371 waiting on condition  [0x000173d4e000]
   java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@21.0.2/Native Method)
- parking to wait for  <0x0007dcd49b88> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at 
java.util.concurrent.locks.LockSupport.park(java.base@21.0.2/LockSupport.java:371)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21.0.2/AbstractQueuedSynchronizer.java:519)
at 
java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21.0.2/ForkJoinPool.java:3780)
at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@21.0.2/ForkJoinPool.java:3725)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21.0.2/AbstractQueuedSynchronizer.java:1707)
at 
org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.lambda$awaitProcessableTasks$1(DefaultTaskManager.java:142)
at 
org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager$$Lambda/0x007001305428.get(Unknown
 Source)
at 
org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.returnWithTasksLocked(DefaultTaskManager.java:357)
at 
org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.awaitProcessableTasks(DefaultTaskManager.java:129)
at 
org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManagerTest$AwaitingRunnable.run(DefaultTaskManagerTest.java:122)
at java.lang.Thread.runWith(java.base@21.0.2/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.2/Thread.java:1583)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1043: Administration of groups

[Apoorv Mittal]

Hi Andrew,
Thanks for the KIP. The group administration is getting difficult with new
types of groups being added and certainly the proposal looks great.
I have some questions:

AM1: The current proposal defines the behaviour for listing and describing
groups, simplifying create for `kafka-share-groups.sh`. Do we want to leave
the other group administration like delete to respective groups or shall
have common behaviour defined i.e. leave to respective
kafka-consumer-groups.sh or kafka-share-groups.sh?

AM2: Reading the notes on KIP-848,
https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes,
which requires `--consumer-property group.protocol=consumer` to enable
modern consumer group. But the listing for `classic` "type" also defines
"protocol" as `consumer` in some scenarios. Is it intended or `classic`
type should different protocol?

AM3: The KIP adds behaviour for  `kafka-share-groups.sh` which defines the
`--create` option. Though it simplifies group creation, what should be the
error behaviour when the group with the same name exists but not of "share"
group type?

AM4: The GroupMetadataManager.java stores all groups in the same data
structure which means the name has to be unique across different group
types. Do you think we should also change the error message for existing
kafka-consumer-groups.sh and kafka-share-groups.sh to recommend using new
kafka-groups.sh for extensive groups list? As currently the individual
scripts will result in "Group already exists" while creating new groups but
listing with respective utility will not yield the group.

AM5: The KIP defines compatibility considerations for the ListGroups RPC.
But it's unclear to me why it's needed as the client and server supporting
`kafka-groups.sh` will be post ListGroups v5 API anyways hence TypesFilter
will be supported in both client and server. Am I missing something here?

Regards,
Apoorv Mittal
+44 7721681581


On Wed, Jul 17, 2024 at 6:26 PM Andrew Schofield 
wrote:

> Hi Lianet,
> Thanks for your comments.
>
> LM5. Unfortunately, the protocol type has to be a string rather than
> an enumeration. This is because when people have created their own
> extensions of the classic consumer group protocol, they have chosen
> their own protocol strings. For example, the Confluent schema registry
> uses “sr” and there are other examples in the wild.
>
> LM6.1. It’s because of the difference between a parameterised
> type and a raw type.
>
> If I use:
> public class ListGroupsResult
> public class ListConsumerGroupsResult
>
> then ListGroupsResult (no type variable) is a raw type which does
> not provide a type for the type variable. This causes compiler warnings
> when the type is used, unless it’s used as ListGroupsResult.
>
> However, this works better.
> public class AbstractListGroupsResult
> public class ListGroupsResult extends
> AbstractListGroupsResult
> public class ListConsumerGroupsResult extends
> AbstractListGroupsResult
>
> I’ll change the KIP to use this.
>
> LM6.2. I was just pointing out a difference and you’re happy
> with it. That’s good.
>
> LM7. If you have a cluster with a mixture of classic and modern
> consumer groups, to list them all you could use this:
>
> bin/kafka-groups.sh --protocol consumer
>
> When there are no classic consumer groups, you could do:
>
> bin/kafka-groups.sh --group-type consumer
>
> However, this only gives a complete list if you don’t have any classic
> consumer groups.
>
> As a result, I suggested --consumer so you don’t need to know
> or care about the existence of classic and modern consumer groups.
> I think it helps, but you aren’t convinced I think, which tells me
> more thinking needed here.
>
> Maybe adding --share would help, so only power users would
> use --group-type or --protocol to deal with the more complicated
> cases.
>
> LM8. It’s just not clear. I was trying to make the output the same
> whether the group was created, or whether it already existed. In
> either case, there’s a share group in existence. The
> AlterShareGroupOffsets RPC doesn’t distinguish between the
> two cases in its response.
>
> Thanks,
> Andrew
>
> > On 16 Jul 2024, at 21:24, Lianet M.  wrote:
> >
> > Hello Andrew, thanks for considering the feedback. Some follow-ups and
> > other comments:
> >
> > LM4. Good point about the older RPC versions and therefore the
> > Optional, agreed.
> >
> > LM5. In GroupListing, should we use the existing
> > org.apache.kafka.clients.ProtocolType to represent the protocol (instead
> of
> > String). I don’t quite like the fact that the enum is inside the
> > GroupRebalanceConfig though, feels like it should be a first level
> citizen.
> >
> >
> > LM6. Regarding the changes around ListGroupResults with generics.
> > - LM6.1. What’s the need for keeping both, a base
> > AbstractListGroupsResult and the ListGroupsResult
> > extends AbstractListGroupsResult? Would it work if ins

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3120

[Apache Jenkins Server]

See 

Support Needed

[MIMOUNI WAFAA]

Dear Dev Kafka Team,

I hope this message finds you well. I am writing to seek your assistance 
regarding an issue I am facing with real-time logs.

I am currently encountering an issue with the compact log functionality while 
using the Debezium connector with Apache Kafka to capture data changes from a 
PostgreSQL database.

Specifically, the compact log feature I have implemented does not appear to be 
functioning correctly.

I would greatly appreciate it if someone from your team could assist me in 
resolving this matter at your earliest convenience. Please let me know a 
suitable time for us to discuss this further or if there are specific steps I 
should take to provide more information.

Thank you very much for your attention to this matter. I look forward to your 
prompt response.

Best regards,

[jira] [Created] (KAFKA-17163) Revisit testSubscriptionOnInvalidTopic for AsyncKafkaConsumer

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-17163:
--

 Summary: Revisit testSubscriptionOnInvalidTopic for 
AsyncKafkaConsumer
 Key: KAFKA-17163
 URL: https://issues.apache.org/jira/browse/KAFKA-17163
 Project: Kafka
  Issue Type: Task
  Components: consumer
Reporter: Lianet Magrans


testSubscriptionOnInvalidTopic expects the consumer throws
InvalidTopicException on poll(ZERO), which fails for the new AsyncConsumer. We 
should investigate if this needs the same fix applied to other tests, to do 
continuous poll(ZERO) to allow for the background thread to run after a fetch 
event has been added in the app thread, so that related errors can be 
propagated in poll. See linked Jira with similar fix. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17164) Consider to enforce application.server : format at config level

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-17164:
---

 Summary: Consider to enforce application.server : 
format at config level
 Key: KAFKA-17164
 URL: https://issues.apache.org/jira/browse/KAFKA-17164
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


KafkaStreams support configuration parameter `application.server` which must be 
of format `:`.

However, in `StreamsConfig` we accept any `String` w/o validation, but only 
validate the format inside `KafkaStreams` constructor.

It might be better to add an `AbstactConfig.Validator` and move this validation 
into `StreamsConfig` directly.

This would be a semantic change because `new StreamsConfig(...)` might now 
throw an exception. Thus we need a KIP for this change, and it's technically 
backward incompatible... (So not sure if we can do this at all – expect for a 
major release? – But 4.0 is close...)

The ROI is unclear to be fair. Filing this ticket mainly for documentation and 
to collect feedback if people think it would be a worthwhile thing to do or not.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3121

[Apache Jenkins Server]

See 

Re: [PR] Update powered-by_adding SPITHA.html [kafka-site]

[via GitHub]

mjsax merged PR #611:
URL: https://github.com/apache/kafka-site/pull/611


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Update powered-by_adding SPITHA.html [kafka-site]

[via GitHub]

mjsax commented on PR #611:
URL: https://github.com/apache/kafka-site/pull/611#issuecomment-2237776445

   Ah thanks. It's there now. Did not see it yesterday. Not sure why...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-17165) Revisit LeaderEpochFileCache#writeToFileForTruncation

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17165:
--

 Summary: Revisit LeaderEpochFileCache#writeToFileForTruncation
 Key: KAFKA-17165
 URL: https://issues.apache.org/jira/browse/KAFKA-17165
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


see origin discussion: 
https://github.com/apache/kafka/pull/16614#discussion_r1683340380



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17166) Use `NoOpScheduler` to rewrite LogManagerTest#testLogRecoveryMetrics

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17166:
--

 Summary: Use `NoOpScheduler` to rewrite 
LogManagerTest#testLogRecoveryMetrics
 Key: KAFKA-17166
 URL: https://issues.apache.org/jira/browse/KAFKA-17166
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


`LogManagerTest#testLogRecoveryMetrics` does not require the scheduler, so we 
can introduce the `NoOpScheduler` to handle the test case.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17167) Moving the flush call inside the read lock when writing truncation

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17167:
--

 Summary: Moving the flush call inside the read lock when writing 
truncation
 Key: KAFKA-17167
 URL: https://issues.apache.org/jira/browse/KAFKA-17167
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


copy the discussion  
(https://github.com/apache/kafka/pull/16614#discussion_r1683340380) to here
{quote}
Consider the following sequence: we take a snapshot of the epoch entries here; 
a new epoch entry is added and is flushed to disk; the scheduler then writes 
the snapshot to disk. This can lead to the case where the leader epoch file 
doesn't contain all entries up to the recovery point.
{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17118) Remove StorageTool#buildMetadataProperties

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17118.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Remove StorageTool#buildMetadataProperties
> --
>
> Key: KAFKA-17118
> URL: https://issues.apache.org/jira/browse/KAFKA-17118
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: kangning.li
>Priority: Minor
> Fix For: 3.9.0
>
>
> It is useless in production after 
> https://github.com/apache/kafka/commit/7060c08d6f9b0408e7f40a90499caf2e636fac61



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17122) Change the type of `clusterId` from `UUID` to `String`

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17122.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Change the type of `clusterId` from `UUID` to `String`
> --
>
> Key: KAFKA-17122
> URL: https://issues.apache.org/jira/browse/KAFKA-17122
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Volk Huang
>Priority: Minor
> Fix For: 3.9.0
>
>
> see discussion: 
> [https://github.com/apache/kafka/pull/14628#discussion_r1673749083]
> There is no uuid restriction to "cluster id", so we don't need to declare the 
> "uuid" type



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17077) the node.id is inconsistent to broker.id when "broker.id.generation.enable=true"

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17077.

Fix Version/s: 3.9.0
   Resolution: Fixed

> the node.id is inconsistent to broker.id when 
> "broker.id.generation.enable=true"
> 
>
> Key: KAFKA-17077
> URL: https://issues.apache.org/jira/browse/KAFKA-17077
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Critical
> Fix For: 3.9.0
>
>
> We change the broker id of `KafkaConfig` directly when 
> `broker.id.generation.enable=true` [0]. However, the update is NOT sync to 
> node.id of `KafkaConfig`. It results in following issues:
> 1. we can see many "-1" in the log. for example:
> {code:sh}
> [2024-07-03 19:23:08,453] INFO [ExpirationReaper--1-AlterAcls]: Starting 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> {code}
> 2.  `KafkaRaftManager` will use uninitialized node.id to create 
> `KafkaRaftClient` in migration [1], and the error sequentially happens
> [0] 
> https://github.com/apache/kafka/blob/27220d146c5d043da4adc3d636036bd6e7b112d2/core/src/main/scala/kafka/server/KafkaServer.scala#L261
> [1] 
> https://github.com/apache/kafka/blob/27220d146c5d043da4adc3d636036bd6e7b112d2/core/src/main/scala/kafka/raft/RaftManager.scala#L230



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17095) Fix the typo: CreateableTopicConfig -> CreatableTopicConfig

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17095.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Fix the typo: CreateableTopicConfig -> CreatableTopicConfig
> ---
>
> Key: KAFKA-17095
> URL: https://issues.apache.org/jira/browse/KAFKA-17095
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: Ming-Yen Chung
>Priority: Minor
> Fix For: 3.9.0
>
>
> source: 
> https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/CreateTopicsRequest.json#L51



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17168) The logPrefix of RemoteLogReader/RemoteStorageThreadPool is not propagated correctly

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-17168:
--

 Summary: The logPrefix of RemoteLogReader/RemoteStorageThreadPool 
is not propagated correctly
 Key: KAFKA-17168
 URL: https://issues.apache.org/jira/browse/KAFKA-17168
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


RemoteLogReader [0] and RemoteStorageThreadPool [1] override the method 
`logPrefix` instead of passing to constructor. However, `LogContext` does not 
use the method `logPrefix` to set the prefix inner. That means the override 
method is no-op and the expected prefix is never propagated.
[0] 
https://github.com/apache/kafka/blob/66655ab49a9401a62e1e99e637e0c5bd5a1ba550/core/src/main/java/kafka/log/remote/RemoteLogReader.java#L59
[1] 
https://github.com/apache/kafka/blob/66655ab49a9401a62e1e99e637e0c5bd5a1ba550/storage/src/main/java/org/apache/kafka/storage/internals/log/RemoteStorageThreadPool.java#L49



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Artem Livshits]

Hey folks,

Hopefully not to make this KIP go for another spin :-), but I thought of a
modification that might actually address safety concerns over using flags
to ignore a vaguely specified class of errors.

What if we had the following overload of .send method:

  void send(ProducerRecord record, Callback callback, boolean
throwImmediately)

and if throwImmediately=false, then we behave the same way as now (return
errors via Future and poison transaction) and if throwImmediately=true then
we just throw errors immediately from the send function.  This way we don't
ignore the error, we're just changing the method they are delivered.  Then
KStreams can catch the error for send(record, callback, true) and do
whatever it needs to do.

-Artem


On Mon, Jul 15, 2024 at 4:30 PM Greg Harris 
wrote:

> Matthias,
>
> Thank you for rejecting my suggested alternatives. Your responses are the
> sorts of things I expected to see summarized in the text of the KIP.
>
> I agree with most of your rejections, except this one:
>
> > "Estimation" is not sufficient, but we would need to know it exactly.
> > And that's an impl detail, given that the message format could change
> > and we could add new internal fields increasing the message size.
>
> An estimate is certainly going to have an error. But an estimate shouldn't
> be treated as exact anyway, there should be an error bound, or "safety
> factor" used when interpreting it. For example, if the broker side limit is
> 1MB, and an estimate could be wrong by ~10%, then computing an estimate and
> dropping records >900kb should be sufficient to prevent RTLEs.
> This is the sort of estimation that I would expect application developers
> could implement, without knowing the exact serialization and protocol
> overhead. This could prevent user-originated oversize records from making
> it to the producer.
>
> Thanks,
> Greg
>
>
> On Mon, Jul 15, 2024 at 4:08 PM Matthias J. Sax  wrote:
>
> > I agree with Alieh and Artem -- in the end, why buffer records twice? We
> > effectively want to allow to push some error handling (which I btw
> > consider "business logic") into the producer. IMHO, there is nothing
> > wrong with it. Dropping a poison pill record is no really a violation of
> > atomicity from my POV, but a business logic decision to not include a
> > record in a transaction -- the proposed API just makes it much simpler
> > to achieve this business logic goal.
> >
> >
> >
> > For memory size estimation, throughput or message size is actually not
> > relevant, right? We would need to look at producer buffer size, ie,
> > `batch.size`, `max.in.flight.request.per.connection` and guesstimate the
> > number of connections there might be? At least for KS, we don't need to
> > buffer everything until commit, but only until we get a successful "ack"
> > back.
> >
> > Note that KS application not only need to write to (a single) user
> > result topic, but multiple output topics, as well as repartition and
> > changelog topics, across all tasks assigned to a thread (ie, producer),
> > which can easily be 10 tasks or more. If we assume topics with 30
> > partitions (topics with 50 or more partitions are not uncommon either),
> > and a producer who must write to 10 different topics, the number of
> > required connections is very quickly very high, and thus the required
> > "application buffer space" would be significant.
> >
> >
> >
> > Your others ideas seems not to be viable alternatives:
> >
> > > Streams users that specifically want to drop oversize records can
> > > estimate the size of their data and drop records which are too
> > > large, enforcing their own limits that are lower than the Kafka limits.
> >
> > "Estimation" is not sufficient, but we would need to know it exactly.
> > And that's an impl detail, given that the message format could change
> > and we could add new internal fields increasing the message size. The
> > idea to add some `producer.serializedRecordSize()` helper method was
> > discussed, but it's a very ugly API and clumsy to use -- also, the user
> > code would need to know the producer config which it might not have
> > access to (as it might get passed in from some config file; and it might
> > also be changed).
> >
> > Some other alternative we also discussed was, to let `send()` throw an
> > exception for a "record too large" case directly. However, this solution
> > raises backward compatibly concerns, and it might also not help us to
> > extend the solution in the future (eg, tackle broker side errors). So we
> > discarded this idea.
> >
> >
> >
> > > Streams users that want CONTINUE semantics can use at_least_once
> > > semantics
> >
> > Not really. EOS is mainly about not having duplicates in the result, but
> > at-least-once cannot provide this guarantee. (Even if I repeat my self:
> > but dropping a poison pill record based on a business logic decision is
> > not data loss, but effectively a business logic filter...)
> >
> >
> >
> > > Streams itse