[jira] [Created] (KAFKA-15923) Implement changes proposed in KIP-994

[Raman Verma (Jira)]

Raman Verma created KAFKA-15923:
---

 Summary: Implement changes proposed in KIP-994
 Key: KAFKA-15923
 URL: https://issues.apache.org/jira/browse/KAFKA-15923
 Project: Kafka
  Issue Type: Task
Reporter: Raman Verma
Assignee: Raman Verma


1. Introduce durationFilter to ListTransactionsRequest as proposed in KIP-994

Add DurationFilter to ListTransactionsRequest and make corresponding broker 
side changes. Also, make changes to `kafka-transactions.sh --list` tooling to 
use this new field in the API

2. Introduce TransactionLastUpdateTimeMs tagged field to 
DescribeTransactionsResponse. Make broker side changes to send this bit of 
information. Also, make changes to `kafka-transactions.sh --describe` tooling 
to display this new piece of information to the output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15924) Flaky test - QuorumControllerTest.testFatalMetadataReplayErrorOnActive

[Haruki Okada (Jira)]

Haruki Okada created KAFKA-15924:


 Summary: Flaky test - 
QuorumControllerTest.testFatalMetadataReplayErrorOnActive
 Key: KAFKA-15924
 URL: https://issues.apache.org/jira/browse/KAFKA-15924
 Project: Kafka
  Issue Type: Bug
Reporter: Haruki Okada


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14242/15/tests]

 
{code:java}
Error
org.opentest4j.AssertionFailedError: expected:  
but was: 
Stacktrace
org.opentest4j.AssertionFailedError: expected:  
but was: 
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at 
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at 
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
at 
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
at 
app//org.apache.kafka.controller.QuorumControllerTest.testFatalMetadataReplayErrorOnActive(QuorumControllerTest.java:1132)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at 
app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at 
app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at 
app//org.junit.platform.engine.support.hie

[jira] [Created] (KAFKA-15925) Flaky test testReplicateSourceDefault - MirrorConnectorsIntegrationTransactionsTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15925:
--

 Summary: Flaky test testReplicateSourceDefault - 
MirrorConnectorsIntegrationTransactionsTest
 Key: KAFKA-15925
 URL: https://issues.apache.org/jira/browse/KAFKA-15925
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing. 

[https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=*MirrorConnectorsIntegrationTransactionsTest]

[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14842/10/tests/]

 
{code:java}
org.opentest4j.AssertionFailedError: `delete.retention.ms` should be different, 
because it's in exclude filter!  ==> expected: not equal but was: <8640>
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
   at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
   at 
app//org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:277)  
 at 
app//org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:263)
 at app//org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2828) 
 at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testReplicateSourceDefault$9(MirrorConnectorsIntegrationBaseTest.java:826)
   at 
app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
   at 
app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)   
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)   
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)   
 at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicateSourceDefault(MirrorConnectorsIntegrationBaseTest.java:813){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15926) Flaky test - testReplicateSourceDefault - MirrorConnectorsIntegrationSSLTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15926:
--

 Summary: Flaky test - testReplicateSourceDefault - 
MirrorConnectorsIntegrationSSLTest
 Key: KAFKA-15926
 URL: https://issues.apache.org/jira/browse/KAFKA-15926
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing.

See 
[https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=*MirrorConnectorsIntegrationSSLTest]
 for failed runs
{code:java}
org.opentest4j.AssertionFailedError: `delete.retention.ms` should be different, 
because it's in exclude filter!  ==> expected: not equal but was: <8640>
    at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
    at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
    at 
app//org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:277)
    at 
app//org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:263)
    at 
app//org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2828)
    at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testReplicateSourceDefault$9(MirrorConnectorsIntegrationBaseTest.java:826)
    at 
app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
    at 
app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
    at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)
    at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)
    at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)
    at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicateSourceDefault(MirrorConnectorsIntegrationBaseTest.java:813)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15927) Flaky test - testReplicateSourceDefault - MirrorConnectorsIntegrationExactlyOnceTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15927:
--

 Summary: Flaky test - testReplicateSourceDefault - 
MirrorConnectorsIntegrationExactlyOnceTest
 Key: KAFKA-15927
 URL: https://issues.apache.org/jira/browse/KAFKA-15927
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing

[https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=*MirrorConnectorsIntegrationExactlyOnceTest]
{code:java}
org.opentest4j.AssertionFailedError: `delete.retention.ms` should be different, 
because it's in exclude filter!  ==> expected: not equal but was: <8640>
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
   at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
   at 
app//org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:277)  
 at 
app//org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:263)
 at app//org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2828) 
 at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testReplicateSourceDefault$9(MirrorConnectorsIntegrationBaseTest.java:826)
   at 
app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
   at 
app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)   
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)   
 at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)   
 at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testReplicateSourceDefault(MirrorConnectorsIntegrationBaseTest.java:813)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2427

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 433105 lines...]

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteNewTopicConfigs() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteNewTopicConfigs() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testDelegationTokens() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testDelegationTokens() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteNewClientQuotas() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteNewClientQuotas() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteExistingTopicConfigs() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteExistingTopicConfigs() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteExistingClientQuotas() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testWriteExistingClientQuotas() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testScramAndQuotaChangesInSnapshot() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkConfigMigrationClientTest > testScramAndQuotaChangesInSnapshot() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > VerificationGuardTest > 
testEqualsAndHashCode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > VerificationGuardTest > 
testEqualsAndHashCode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > VerificationGuardTest > 
testVerify() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > VerificationGuardTest > 
testVerify() PASSED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor(String) > 
testRackAwareRangeAssignor(String).quorum=zk STARTED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor(String) > 
testRackAwareRangeAssignor(String).quorum=zk PASSED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor(String) > 
testRackAwareRangeAssignor(String).quorum=kraft STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [3] 
Type=ZK, MetadataVersion=3.6-IV2, Security=PLAINTEXT SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [4] 
Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT STARTED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
FetchFromFollowerIntegrationTest > testRackAwareRangeAssignor(String) > 
testRackAwareRangeAssignor(String).quorum=kraft PASSED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, String).quorum=zk.groupProtocol=generic STARTED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, String).quorum=zk.groupProtocol=generic PASSED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, String).quorum=kraft.groupProtocol=generic 
STARTED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, String).quorum=kraft.groupProtocol=generic 
PASSED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, 
String).quorum=kraft+kip848.groupProtocol=generic STARTED

Gradle Test Run :core:test > Gradle Test Executor 101 > 
SaslPlainPlaintextConsumerTest > testCoordinatorFailover(String, String) > 
testCoordinatorFailover(String, 
String).quorum=kraft+kip848.groupProtocol=generic PASSED

5305 tests completed, 2 failed, 15 skipped
There were failing tests. See the report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/core/build/reports/tests/test/index.html

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > [4] 
Type=ZK, MetadataVersion=3.7-IV0, Security=PLAINTEXT SKIPPED

Gradle Test Run :core:test > Gradle Test Executor 92 > 
ZkMigrationIntegrationTest >

[jira] [Created] (KAFKA-15928) Flaky test - testBalancePartitionLeaders - QuorumControllerTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15928:
--

 Summary: Flaky test - testBalancePartitionLeaders - 
QuorumControllerTest
 Key: KAFKA-15928
 URL: https://issues.apache.org/jira/browse/KAFKA-15928
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing.

[https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=org.apache.kafka.controller.QuorumControllerTest&tests.test=testBalancePartitionLeaders()]

 

```
org.opentest4j.AssertionFailedError: Condition not met within timeout 1. 
Leaders were not balanced after unfencing all of the brokers ==> expected: 
 but was: 
 at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
 at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
 at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
 at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
 at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
 at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
 at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)
 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)
 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)
 at 
org.apache.kafka.controller.QuorumControllerTest.testBalancePartitionLeaders(QuorumControllerTest.java:490)
```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15929) Flaky test - testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress - TopicCommandIntegrationTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15929:
--

 Summary: Flaky test - 
testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress - 
TopicCommandIntegrationTest
 Key: KAFKA-15929
 URL: https://issues.apache.org/jira/browse/KAFKA-15929
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing.

https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=org.apache.kafka.tools.TopicCommandIntegrationTest&tests.test=testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress(String)%5B1%5D
{code:java}
java.lang.NullPointerException: Cannot invoke 
"org.apache.kafka.clients.admin.PartitionReassignment.addingReplicas()" because 
"reassignments" is null
    at 
org.apache.kafka.tools.TopicCommandIntegrationTest.testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress(TopicCommandIntegrationTest.java:800)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-996: Pre-Vote

[Luke Chen]

Hi Alyssa,

Thanks for the KIP!
This is an important improvement for KRaft quorum.

Some comments:
1. Follower transitions to: Prospective: After expiration of the election
timeout
-> Is this the fetch timeout, not election timeout?

2. I also agree we don't bump the epoch in prospective state.
 A candidate will now send a VoteRequest with the PreVote field set to true
and CandidateEpoch set to its [epoch + 1] when its election timeout
expires.
-> What is "CandidateEpoch"? And I thought you've agreed to not set [epoch
+ 1] ?

Thanks.
Luke

On Wed, Nov 29, 2023 at 2:06 AM Alyssa Huang 
wrote:

> Thanks Jose, I've updated the KIP to reflect your and Jason's suggestions!
>
> On Tue, Nov 28, 2023 at 9:54 AM José Armando García Sancio
>  wrote:
>
> > Hi Alyssa
> >
> > On Mon, Nov 27, 2023 at 1:40 PM Jason Gustafson
> >  wrote:
> > > 2. Do you think the pretend epoch bump is necessary? Would it be
> simpler
> > to
> > > change the prevote acceptance check to assert a greater than or equal
> > epoch?
> >
> > I agree with Jason it would be better if all of the requests always
> > sent the current epoch. For the VoterRequest, it should be correct for
> > the prospective node to not increase the epoch and send the current
> > epoch and id. Since there are two states (prospective and candidate)
> > that can send a VoteRequest, maybe we can change the field name to
> > just ReplicaEpoch and ReplicaId.
> >
> > Thanks,
> > --
> > -José
> >
>

[jira] [Created] (KAFKA-15930) Flaky test - testWithGroupId - TransactionsBounceTest

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15930:
--

 Summary: Flaky test - testWithGroupId - TransactionsBounceTest
 Key: KAFKA-15930
 URL: https://issues.apache.org/jira/browse/KAFKA-15930
 Project: Kafka
  Issue Type: Bug
Reporter: Lucas Brutschy


Test is intermittently failing.

[https://ge.apache.org/scans/tests?search.names=Git%20branch&search.relativeStartTime=P28D&search.rootProjectNames=kafka&search.timeZoneId=Europe%2FVienna&search.values=trunk&tests.container=kafka.api.TransactionsBounceTest&tests.test=testWithGroupId()]

 
{code:java}
org.apache.kafka.common.KafkaException: Unexpected error in 
TxnOffsetCommitResponse: The server disconnected before a response was received.
    at 
app//org.apache.kafka.clients.producer.internals.TransactionManager$TxnOffsetCommitHandler.handleResponse(TransactionManager.java:1702)
    at 
app//org.apache.kafka.clients.producer.internals.TransactionManager$TxnRequestHandler.onComplete(TransactionManager.java:1236)
    at 
app//org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)
    at 
app//org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:594)
    at app//org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:586)
    at 
app//org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:460)
    at 
app//org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:337)
    at 
app//org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:252)
    at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15887) Autocommit during close consistently fails with exception in background thread

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-15887.

Resolution: Fixed

> Autocommit during close consistently fails with exception in background thread
> --
>
> Key: KAFKA-15887
> URL: https://issues.apache.org/jira/browse/KAFKA-15887
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Lucas Brutschy
>Assignee: Philip Nee
>Priority: Blocker
>
> when I run {{AsyncKafkaConsumerTest}} I get this every time I call close:
> {code:java}
> java.lang.IndexOutOfBoundsException: Index: 0
>   at java.base/java.util.Collections$EmptyList.get(Collections.java:4483)
>   at 
> java.base/java.util.Collections$UnmodifiableList.get(Collections.java:1310)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.findCoordinatorSync(ConsumerNetworkThread.java:302)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.ensureCoordinatorReady(ConsumerNetworkThread.java:288)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.maybeAutoCommitAndLeaveGroup(ConsumerNetworkThread.java:276)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:257)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:101)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.1 RC0

[Josep Prat]

Hi Mickael,
This PR[1] made me realize NOTICE-binary is missing the notice for
commons-io. I don't know if it's a blocker or not. I can cherry pick the
commit to the 3.6 branch if you want.

Best,


[1]: https://github.com/apache/kafka/pull/14865

On Tue, Nov 28, 2023 at 10:25 AM Josep Prat  wrote:

> Hi Mickael,
> Thanks for running the release. It's a +1 for me (non-binding).
> I did the following:
> - Verified artifact's signatures and hashes
> - Checked JavaDoc (with navigation to Oracle JavaDoc)
> - Compiled source code
> - Run unit tests and integration tests
> - Run getting started with ZK and KRaft
>
> Best,
>
> On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
>> +1 (non-binding)
>>
>> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
>> 2. Ran all the unit and integration tests.
>> 3. Ran quickstart and verified the produce-consume on a 3 node cluster.
>> 4. Verified the tiered storage functionality with local-tiered storage.
>>
>> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri 
>> wrote:
>>
>> > Hi Mickael,
>> >
>> > - Build from source (Java 17, Scala 2.13)
>> > - Run unit and integration tests
>> > - Run custom client apps using staging artifacts
>> >
>> > +1 (non binding)
>> >
>> > Thanks
>> > Fede
>> >
>> >
>> >
>> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz  wrote:
>> > >
>> > > +1 non-binding. I used the staged Scala 2.13 artifacts and the staged
>> > Maven
>> > > repo for my tests. All seems to work fine.
>> > >
>> > > Thanks
>> > > Jakub
>> > >
>> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison 
>> > wrote:
>> > >
>> > > > Hello Kafka users, developers and client-developers,
>> > > >
>> > > > This is the first candidate for release of Apache Kafka 3.6.1.
>> > > >
>> > > > This is a bugfix release with several fixes, including dependency
>> > > > version bumps for CVEs.
>> > > >
>> > > > Release notes for the 3.6.1 release:
>> > > >
>> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
>> > > >
>> > > > *** Please download, test and vote by Friday, December 1
>> > > >
>> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > > https://kafka.apache.org/KEYS
>> > > >
>> > > > * Release artifacts to be voted upon (source and binary):
>> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
>> > > >
>> > > > * Maven artifacts to be voted upon:
>> > > >
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > > >
>> > > > * Javadoc:
>> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
>> > > >
>> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
>> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
>> > > >
>> > > > PR for updating docs:
>> > > > https://github.com/apache/kafka-site/pull/568
>> > > >
>> > > > * Documentation:
>> > > > https://kafka.apache.org/36/documentation.html
>> > > >
>> > > > * Protocol:
>> > > > https://kafka.apache.org/36/protocol.html
>> > > >
>> > > > * Successful Jenkins builds for the 3.6 branch:
>> > > > Unit/integration tests: We still have a lot of flaky tests in the
>> 3.6
>> > > > branch. Looking at the last few 3.6 builds in
>> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it seems
>> all
>> > > > tests passed at least once apart from
>> > > > ClusterConnectionStatesTest.testSingleIP(). There's
>> > > > https://issues.apache.org/jira/browse/KAFKA-15762 to fix that test.
>> > > > System tests: Still running I'll post an update once they complete.
>> > > >
>> > > > Thanks,
>> > > > Mickael
>> > > >
>> >
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-15931:
--

 Summary: Cached transaction index gets closed if tiered storage 
read is interrupted
 Key: KAFKA-15931
 URL: https://issues.apache.org/jira/browse/KAFKA-15931
 Project: Kafka
  Issue Type: Bug
  Components: Tiered-Storage
Affects Versions: 3.6.0
Reporter: Ivan Yurchenko


This reproduces when reading from remote storage with the default 
{{fetch.max.wait.ms}} (500) or lower. This error is logged

 
{noformat}
[2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data 
for topic1-0 (kafka.log.remote.RemoteLogReader)
org.apache.kafka.common.KafkaException: Failed read position from the 
transaction index 
    at 
org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235)
    at 
org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171)
    at 
kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359)
    at 
kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341)
    at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310)
    at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62)
    at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.channels.ClosedChannelException
    at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150)
    at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325)
    at 
org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233)
    ... 10 more
{noformat}
and after that this txn index becomes unusable until the process is restarted.

I suspect, it's caused by the reading thread being interrupted due to the fetch 
timeout. At least [this 
code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160]
 in {{AbstractInterruptibleChannel}} is called.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15932:


 Summary: Flaky test - 
PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
 Key: KAFKA-15932
 URL: https://issues.apache.org/jira/browse/KAFKA-15932
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.7.0
Reporter: Andrew Schofield


Intermittently failing test for the new consumer.

https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14859/1/tests/

```Error
org.opentest4j.AssertionFailedError: Timed out before consuming expected 1 
records. The number consumed was 0.
Stacktrace
org.opentest4j.AssertionFailedError: Timed out before consuming expected 1 
records. The number consumed was 0.
at 
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
at 
app//kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:161)
at 
app//kafka.api.AbstractConsumerTest.consumeAndVerifyRecords(AbstractConsumerTest.scala:128)
at 
app//kafka.api.PlaintextConsumerTest.testSeek(PlaintextConsumerTest.scala:616)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at 
app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at 
app//org.junit.platform.engine.suppor

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Taras Ledkov]

Hi team,

Ping for review / vote for KIP-967 [1].
Voting thread is here [2]

[1]. 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
[2]. https://github.com/apache/kafka/pull/14203
[2]. https://lists.apache.org/thread/wc4v5v3pynl15g1q547m2croqsqgmzpw

--
With best regards,
Taras Ledkov

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2428

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 424064 lines...]

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnEmptyMapWhenUnsupportedVersionFailure STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnEmptyMapWhenUnsupportedVersionFailure PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
shouldRetryWhenTopicCreateThrowsWrappedTimeoutException STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
shouldRetryWhenTopicCreateThrowsWrappedTimeoutException PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnTopicConfigWhenTopicExists STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnTopicConfigWhenTopicExists PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
retryEndOffsetsShouldRetryWhenTopicNotFound STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
retryEndOffsetsShouldRetryWhenTopicNotFound PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
createShouldReturnFalseWhenSuppliedNullTopicDescription STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
createShouldReturnFalseWhenSuppliedNullTopicDescription PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeShouldReturnEmptyWhenTopicDoesNotExist STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeShouldReturnEmptyWhenTopicDoesNotExist PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
verifyingTopicCleanupPolicyShouldReturnFalseWhenTopicAuthorizationError STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
verifyingTopicCleanupPolicyShouldReturnFalseWhenTopicAuthorizationError PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnEmptyMapWhenNoTopicsAreSpecified STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
describeTopicConfigShouldReturnEmptyMapWhenNoTopicsAreSpecified PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
shouldRetryCreateTopicWhenAvailableBrokersAreNotEnoughForReplicationFactor 
STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
shouldRetryCreateTopicWhenAvailableBrokersAreNotEnoughForReplicationFactor 
PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
throwsWithApiVersionMismatchOnDescribe STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
throwsWithApiVersionMismatchOnDescribe PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
returnEmptyWithApiVersionMismatchOnCreate STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
returnEmptyWithApiVersionMismatchOnCreate PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
endOffsetsShouldFailWhenAnyTopicPartitionHasError STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
endOffsetsShouldFailWhenAnyTopicPartitionHasError PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
endOffsetsShouldReturnEmptyMapWhenPartitionsSetIsNull STARTED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
endOffsetsShouldReturnEmptyMapWhenPartitionsSetIsNull PASSED

Gradle Test Run :connect:runtime:test > Gradle Test Executor 47 > 
org.apache.kafka.connect.util.TopicAdminTest > 
shouldCreateTopicWithPartitionsWhenItDoesNot

Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery

[Bruno Cadonna]

Hi,

Thanks for the updates!


1.
Could you please link the correct ticket in the KIP?

2.
Could you please adapt the motivation section and the title to the 
updated goal of the KIP? There is no fetch() or fetchAll() method in the 
query class.


3.
Could you please add the "// newly added" comment to all parts that were 
newly added? That is methods lowerKeyBound() and upperKeyBound().


4.
We should use a more fluent API as I proposed in my last e-mail:

Here again

WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2);
WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2);
WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2);

5.
We should also consider the order of the results similar as we did in 
KIP-968. Alternatively, we do not guarantee any order and postpone the 
order guarantees to a future KIP.



Best,
Bruno



On 11/17/23 3:11 AM, Matthias J. Sax wrote:

Thanks for the KIP.

Given how `WindowRangeQuery` works right now, it's really time to 
improve it.



1) Agree. It's not clear what will be added right now. I think we should 
deprecate existing `getKey()` w/o an actually replacement? For 
`getFromKey` and `getToKey` we should actually be `lowerKeyBound()` and 
`upperKeyBound()` to align to KIP-969?


Also wondering if we should deprecate existing `withKey()` and 
`withWindowStartRange`? `withKey` only works for SessionStores and 
implements a single-key full-time-range query. Similarly, 
`withWindowStartRange` only works for WindowedStore and implements an 
all-key time-range query. Thus, both are rather special and it seems the 
aim of this KIP is to generalize `WindowRangeQuery` to arbitrary 
key-range/time-range queries?


What raises one question about time-range semantics, given that we query 
windows with different semantics.


  - The current `WindowStore` semantics used for 
`WindowRangeQuery#withWindowStartRange` is considering only the window 
start time, ie, the window-start time must fall into the query 
time-range to be returned.


  - In contrast, `SessionStore` time ranges base on `findSession` use 
earliest-session-end-time and latest-session-end-time and thus implement 
an "window-bounds / search-time-range overlap query".


Is there any concern about semantic differences? I would also be 
possible to use the same semantics for both query types, and maybe even 
let the user pick with semantics they want (let users chose might 
actually be the best thing to do)? -- We can also do this incrementally, 
and limit the scope of this KIP (or keep the full KIP scope but 
implement it incrementally only)


Btw: I think we should not add any ordering at this point, and 
explicitly state that no ordering is guarantee whatsoever at this point.




2) Agreed. We should deprecate `getFromTime` and `getToTime` and add new 
method `fromTime` and `toTime`.




3) About the API. If we move forward with general key-range/time-range I 
agree that a more modular approach might be nice. Not sure right now, 
what the best approach would be for this? Looking into KIP-969, we might 
want to have:


  - static withKeyRange
  - static withLowerKeyBound
  - static withUpperKeyBound
  - static withAllKeys (KIP-969 actually uses `allKeys` ?)
  - fromTime
  - toTime

with default-time range would be "all / unbounded" ?



10: you mentioned that `WindowKeyQuery` functionality can be covered by 
`WindowRangeQuery`. I agree. For this case, it seems we want to 
deprecate `WindowKeyQuery` entirely?




-Matthias

On 11/16/23 1:19 AM, Bruno Cadonna wrote:

Hi Hanyu,

Thanks for the KIP!

1)
Could you please mark the pieces that you want to add to the API in 
the code listing in the KIP? You can add a comment like "// newly 
added" or similar. That would make reading the KIP a bit easier 
because one does not need to compare your code with the code in the 
current codebase.


2)
Could you -- as a side cleanup -- also change the getters to not use 
the get-prefix anymore, please? That was apparently an oversight when 
those methods were added. Although the API is marked as Evolving, I 
think we should still deprecate the getX() methods, since it does not 
cost us anything.


3)
I propose to make the API a bit more fluent. For example, something like

WindowRangeQuery.withKey(key).fromTime(t1).toTime(t2)

and

WindowRangeQuery.withAllKeys().fromTime(t1).toTime(t2)

and

WindowRangeQuery.withKeyRange(key1, key2).fromTime(t1).toTime(t2)

and maybe even in addition to the above add also the option to start 
with the time range


WindowRangeQuery.withWindowStartRange(t1, t2).fromKey(key1).toKey(key2)


4)
Could you also add some usage examples? Alieh did quite a nice job 
regarding usage examples in KIP-986.



Best,
Bruno

On 11/8/23 8:02 PM, Hanyu (Peter) Zheng wrote:

Hello everyone,

I would like to start the discussion for KIP-997: Support fetch(fromKey,
toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and
WindowRangeQuery
The KIP can be found here:
https://cwiki.a

Re: [VOTE] 3.6.1 RC0

[Mickael Maison]

Hi Josep,

Good catch!
If it's the only issue we find, I don't think we should block the
release just to fix that.

If we find another issue, I'll backport it before running another RC,
otherwise I'll backport it once 3.6.1 is released.

Thanks,
Mickael

On Wed, Nov 29, 2023 at 11:55 AM Josep Prat  wrote:
>
> Hi Mickael,
> This PR[1] made me realize NOTICE-binary is missing the notice for
> commons-io. I don't know if it's a blocker or not. I can cherry pick the
> commit to the 3.6 branch if you want.
>
> Best,
>
>
> [1]: https://github.com/apache/kafka/pull/14865
>
> On Tue, Nov 28, 2023 at 10:25 AM Josep Prat  wrote:
>
> > Hi Mickael,
> > Thanks for running the release. It's a +1 for me (non-binding).
> > I did the following:
> > - Verified artifact's signatures and hashes
> > - Checked JavaDoc (with navigation to Oracle JavaDoc)
> > - Compiled source code
> > - Run unit tests and integration tests
> > - Run getting started with ZK and KRaft
> >
> > Best,
> >
> > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> >> +1 (non-binding)
> >>
> >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
> >> 2. Ran all the unit and integration tests.
> >> 3. Ran quickstart and verified the produce-consume on a 3 node cluster.
> >> 4. Verified the tiered storage functionality with local-tiered storage.
> >>
> >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri 
> >> wrote:
> >>
> >> > Hi Mickael,
> >> >
> >> > - Build from source (Java 17, Scala 2.13)
> >> > - Run unit and integration tests
> >> > - Run custom client apps using staging artifacts
> >> >
> >> > +1 (non binding)
> >> >
> >> > Thanks
> >> > Fede
> >> >
> >> >
> >> >
> >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz  wrote:
> >> > >
> >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and the staged
> >> > Maven
> >> > > repo for my tests. All seems to work fine.
> >> > >
> >> > > Thanks
> >> > > Jakub
> >> > >
> >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison 
> >> > wrote:
> >> > >
> >> > > > Hello Kafka users, developers and client-developers,
> >> > > >
> >> > > > This is the first candidate for release of Apache Kafka 3.6.1.
> >> > > >
> >> > > > This is a bugfix release with several fixes, including dependency
> >> > > > version bumps for CVEs.
> >> > > >
> >> > > > Release notes for the 3.6.1 release:
> >> > > >
> >> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
> >> > > >
> >> > > > *** Please download, test and vote by Friday, December 1
> >> > > >
> >> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> >> > > > https://kafka.apache.org/KEYS
> >> > > >
> >> > > > * Release artifacts to be voted upon (source and binary):
> >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
> >> > > >
> >> > > > * Maven artifacts to be voted upon:
> >> > > >
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > > >
> >> > > > * Javadoc:
> >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
> >> > > >
> >> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
> >> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
> >> > > >
> >> > > > PR for updating docs:
> >> > > > https://github.com/apache/kafka-site/pull/568
> >> > > >
> >> > > > * Documentation:
> >> > > > https://kafka.apache.org/36/documentation.html
> >> > > >
> >> > > > * Protocol:
> >> > > > https://kafka.apache.org/36/protocol.html
> >> > > >
> >> > > > * Successful Jenkins builds for the 3.6 branch:
> >> > > > Unit/integration tests: We still have a lot of flaky tests in the
> >> 3.6
> >> > > > branch. Looking at the last few 3.6 builds in
> >> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it seems
> >> all
> >> > > > tests passed at least once apart from
> >> > > > ClusterConnectionStatesTest.testSingleIP(). There's
> >> > > > https://issues.apache.org/jira/browse/KAFKA-15762 to fix that test.
> >> > > > System tests: Still running I'll post an update once they complete.
> >> > > >
> >> > > > Thanks,
> >> > > > Mickael
> >> > > >
> >> >
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 101

[jira] [Reopened] (KAFKA-14089) Flaky ExactlyOnceSourceIntegrationTest.testSeparateOffsetsTopic

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal reopened KAFKA-14089:
---
  Assignee: (was: Chris Egerton)

Failure occurred on PR build: 
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/

 
{code:java}
org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Could not 
execute PUT request. Error response: {"error_code":500,"message":"Request timed 
out"}Stacktraceorg.apache.kafka.connect.runtime.rest.errors.ConnectRestException:
 Could not execute PUT request. Error response: 
{"error_code":500,"message":"Request timed out"}  at 
org.apache.kafka.connect.util.clusters.EmbeddedConnect.putConnectorConfig(EmbeddedConnect.java:272)
  at 
org.apache.kafka.connect.util.clusters.EmbeddedConnect.configureConnector(EmbeddedConnect.java:196)
  at 
org.apache.kafka.connect.util.clusters.EmbeddedConnectCluster.configureConnector(EmbeddedConnectCluster.java:48)
 at 
org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testSeparateOffsetsTopic(ExactlyOnceSourceIntegrationTest.java:776)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)   
 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)  at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
   at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)  at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)  at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)  at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) {code}

> Flaky ExactlyOnceSourceIntegrationTest.testSeparateOffsetsTopic
> ---
>
> Key: KAFKA-14089
> URL: https://issues.apache.org/jira/browse/KAFKA-14089
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Mickael Maison
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: failure.txt, 
> org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testSeparateOffsetsTopic.test.stdout
>
>
> It looks like the sequence got broken around "65535, 65537, 65536, 65539, 
> 65538, 65541, 65540, 65543"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15933) Flaky test: testRestartReplication() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTransactionsTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15933:
-

 Summary: Flaky test: testRestartReplication() – 
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTransactionsTest
 Key: KAFKA-15933
 URL: https://issues.apache.org/jira/browse/KAFKA-15933
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
Build / JDK 11 and Scala 2.13 / testRestartReplication() – 
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTransactionsTest3m
 21sErrororg.apache.kafka.common.errors.TimeoutException: Timeout expired after 
6ms while awaiting 
InitProducerIdStacktraceorg.apache.kafka.common.errors.TimeoutException: 
Timeout expired after 6ms while awaiting InitProducerId {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15934) Flaky test: testMultiConsumerStickyAssignor(String, String).quorum=kraft+kip848.groupProtocol=generic – kafka.api.PlaintextConsumerTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15934:
-

 Summary: Flaky test: testMultiConsumerStickyAssignor(String, 
String).quorum=kraft+kip848.groupProtocol=generic – 
kafka.api.PlaintextConsumerTest
 Key: KAFKA-15934
 URL: https://issues.apache.org/jira/browse/KAFKA-15934
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/

 
{code:java}
Errororg.opentest4j.AssertionFailedError: Timeout waiting for controller 
metadata propagating to brokersStacktraceorg.opentest4j.AssertionFailedError: 
Timeout waiting for controller metadata propagating to brokers   at 
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)   at 
app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)  at 
app//kafka.utils.TestUtils$.ensureConsistentKRaftMetadata(TestUtils.scala:1141) 
 at 
app//kafka.utils.TestUtils$.$anonfun$createTopicWithAdmin$1(TestUtils.scala:499)
 at 
app//kafka.utils.TestUtils$.$anonfun$createTopicWithAdmin$1$adapted(TestUtils.scala:499)
 at app//scala.collection.immutable.List.foreach(List.scala:333) at 
app//kafka.utils.TestUtils$.createTopicWithAdmin(TestUtils.scala:499)at 
app//kafka.integration.KafkaServerTestHarness.$anonfun$createTopic$1(KafkaServerTestHarness.scala:184)
   at 
app//kafka.integration.KafkaServerTestHarness.createTopic(KafkaServerTestHarness.scala:176)
  at 
app//kafka.api.AbstractConsumerTest.createTopicAndSendRecords(AbstractConsumerTest.scala:176)
at 
app//kafka.api.PlaintextConsumerTest.testMultiConsumerStickyAssignor(PlaintextConsumerTest.scala:1001)
   at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method) at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566) at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
  at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
   at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
 at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
  at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
   at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
 at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15935) Flaky test: testRestartReplication() – org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15935:
-

 Summary: Flaky test: testRestartReplication() – 
org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest
 Key: KAFKA-15935
 URL: https://issues.apache.org/jira/browse/KAFKA-15935
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
Errororg.opentest4j.AssertionFailedError: Condition not met within timeout 
3. Topic: mm2-status.backup.internal didn't get created in the cluster ==> 
expected:  but was: Stacktraceorg.opentest4j.AssertionFailedError: 
Condition not met within timeout 3. Topic: mm2-status.backup.internal 
didn't get created in the cluster ==> expected:  but was:  at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) 
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)  at 
org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)   
 at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379) 
 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328) at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312) at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302) at 
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.waitForTopicCreated(MirrorConnectorsIntegrationBaseTest.java:1041)
   at 
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.startClusters(MirrorConnectorsIntegrationBaseTest.java:224)
  at 
org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest.startClusters(IdentityReplicationIntegrationTest.java:40)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15936) Flaky tests for class: ConnectorTopicsIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15936:
-

 Summary: Flaky tests for class: ConnectorTopicsIntegrationTest
 Key: KAFKA-15936
 URL: https://issues.apache.org/jira/browse/KAFKA-15936
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]
 
Build / JDK 8 and Scala 2.12 / testGetActiveTopics – 
org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest1m 50s
{code:java}
Errorjava.lang.RuntimeException: java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: The request timed 
out.Stacktracejava.lang.RuntimeException: 
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: The request timed out. at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:427)
   at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:401)
   at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:392)
   at 
org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest.testGetActiveTopics(ConnectorTopicsIntegrationTest.java:115)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)   
 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)  at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
   at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) {code}
 
Build / JDK 8 and Scala 2.12 / testTopicTrackingResetIsDisabled – 
org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest2m 37s
{code:java}
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: The request timed 
out.Stacktracejava.lang.RuntimeException: 
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: The request timed out.  at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:427)
   at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:401)
   at 
org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.createTopic(EmbeddedKafkaCluster.java:392)
   at 
org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest.testTopicTrackingResetIsDisabled(ConnectorTopicsIntegrationTest.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
  at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)   
 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)  at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
 {code}
 
Build / JDK 8 and Scala 2.12 / testTopicTrackingIsDisabled – 
org.apache.kafka.connect.integration.ConnectorTopicsIntegrationTest
{code:java}
java.lang.RuntimeException: java.util.co

[jira] [Created] (KAFKA-15937) Flaky test: testTopicDeletion(String).quorum=kraft – kafka.admin.RemoteTopicCrudTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15937:
-

 Summary: Flaky test: testTopicDeletion(String).quorum=kraft – 
kafka.admin.RemoteTopicCrudTest
 Key: KAFKA-15937
 URL: https://issues.apache.org/jira/browse/KAFKA-15937
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
java.lang.RuntimeException: Received a fatal error while waiting for the 
controller to acknowledge that we are caught 
upStacktracejava.lang.RuntimeException: Received a fatal error while waiting 
for the controller to acknowledge that we are caught up  at 
org.apache.kafka.server.util.FutureUtils.waitWithLogging(FutureUtils.java:68)   
 at kafka.server.BrokerServer.startup(BrokerServer.scala:472)at 
kafka.integration.KafkaServerTestHarness.$anonfun$createBrokers$1(KafkaServerTestHarness.scala:356)
  at 
kafka.integration.KafkaServerTestHarness.$anonfun$createBrokers$1$adapted(KafkaServerTestHarness.scala:352)
  at scala.collection.Iterator.foreach(Iterator.scala:943)at 
scala.collection.Iterator.foreach$(Iterator.scala:943)   at 
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)   at 
scala.collection.IterableLike.foreach(IterableLike.scala:74) at 
scala.collection.IterableLike.foreach$(IterableLike.scala:73)at 
scala.collection.AbstractIterable.foreach(Iterable.scala:56) at 
kafka.integration.KafkaServerTestHarness.createBrokers(KafkaServerTestHarness.scala:352)
 at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:121)
 at 
kafka.api.IntegrationTestHarness.doSetup(IntegrationTestHarness.scala:134)   at 
kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:114) at 
kafka.admin.RemoteTopicCrudTest.setUp(RemoteTopicCrudTest.scala:71)  at 
sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
   at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15938) Flaky test: testCreateRemoteTopicWithValidRetentionTime(String).quorum=kraft – kafka.admin.RemoteTopicCrudTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15938:
-

 Summary: Flaky test:  
testCreateRemoteTopicWithValidRetentionTime(String).quorum=kraft – 
kafka.admin.RemoteTopicCrudTest
 Key: KAFKA-15938
 URL: https://issues.apache.org/jira/browse/KAFKA-15938
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
org.opentest4j.AssertionFailedError: Topic [__consumer_offsets] metadata not 
propagated after 6 msStacktraceorg.opentest4j.AssertionFailedError: Topic 
[__consumer_offsets] metadata not propagated after 6 ms  at 
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)at 
org.junit.jupiter.api.Assertions.fail(Assertions.java:134)   at 
kafka.utils.TestUtils$.waitForAllPartitionsMetadata(TestUtils.scala:1141)at 
kafka.utils.TestUtils$.createTopicWithAdmin(TestUtils.scala:498) at 
kafka.utils.TestUtils$.createOffsetsTopicWithAdmin(TestUtils.scala:540)  at 
kafka.integration.KafkaServerTestHarness.$anonfun$createOffsetsTopic$1(KafkaServerTestHarness.scala:155)
 at 
kafka.integration.KafkaServerTestHarness.createOffsetsTopic(KafkaServerTestHarness.scala:154)
at 
kafka.api.IntegrationTestHarness.doSetup(IntegrationTestHarness.scala:153)   at 
kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:114) at 
kafka.admin.RemoteTopicCrudTest.setUp(RemoteTopicCrudTest.scala:71)  at 
sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
   at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
  at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15939) Flaky test: testInvalidAlterConfigs(String).quorum=kraft – kafka.api.AdminClientWithPoliciesIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15939:
-

 Summary: Flaky test: testInvalidAlterConfigs(String).quorum=kraft 
– kafka.api.AdminClientWithPoliciesIntegrationTest
 Key: KAFKA-15939
 URL: https://issues.apache.org/jira/browse/KAFKA-15939
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
org.opentest4j.AssertionFailedError: Topic [invalid-alter-configs-topic-2] 
metadata not propagated after 6 
msStacktraceorg.opentest4j.AssertionFailedError: Topic 
[invalid-alter-configs-topic-2] metadata not propagated after 6 msat 
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)at 
org.junit.jupiter.api.Assertions.fail(Assertions.java:134)   at 
kafka.utils.TestUtils$.waitForAllPartitionsMetadata(TestUtils.scala:1141)at 
kafka.utils.TestUtils$.createTopicWithAdmin(TestUtils.scala:498) at 
kafka.api.PlaintextAdminIntegrationTest$.checkInvalidAlterConfigs(PlaintextAdminIntegrationTest.scala:2699)
  at 
kafka.api.AdminClientWithPoliciesIntegrationTest.testInvalidAlterConfigs(AdminClientWithPoliciesIntegrationTest.scala:104)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
   at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15940) Flaky test: testLogCleanerConfig(String).quorum=kraft – kafka.server.DynamicBrokerReconfigurationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15940:
-

 Summary: Flaky test: testLogCleanerConfig(String).quorum=kraft – 
kafka.server.DynamicBrokerReconfigurationTest
 Key: KAFKA-15940
 URL: https://issues.apache.org/jira/browse/KAFKA-15940
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/]

 
{code:java}
org.opentest4j.AssertionFailedError: expected: <8000> but was: 
<6000>Stacktraceorg.opentest4j.AssertionFailedError: expected: <8000> but was: 
<6000>at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at 
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)   at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)   at 
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)   at 
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)  at 
kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfigOnServer$1(DynamicBrokerReconfigurationTest.scala:1640)
  at 
kafka.server.DynamicBrokerReconfigurationTest.waitForConfigOnServer(DynamicBrokerReconfigurationTest.scala:1640)
 at 
kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1(DynamicBrokerReconfigurationTest.scala:1635)
  at 
kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1$adapted(DynamicBrokerReconfigurationTest.scala:1635)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)   
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)  
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)   at 
kafka.server.DynamicBrokerReconfigurationTest.waitForConfig(DynamicBrokerReconfigurationTest.scala:1635)
 at 
kafka.server.DynamicBrokerReconfigurationTest.reconfigureServers(DynamicBrokerReconfigurationTest.scala:1580)
at 
kafka.server.DynamicBrokerReconfigurationTest.testLogCleanerConfig(DynamicBrokerReconfigurationTest.scala:568)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15941) Flaky test: shouldRestoreNullRecord() – org.apache.kafka.streams.integration.RestoreIntegrationTest

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15941:
-

 Summary: Flaky test: shouldRestoreNullRecord() – 
org.apache.kafka.streams.integration.RestoreIntegrationTest
 Key: KAFKA-15941
 URL: https://issues.apache.org/jira/browse/KAFKA-15941
 Project: Kafka
  Issue Type: Bug
Reporter: Apoorv Mittal


https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14699/24/tests/
{code:java}
org.opentest4j.AssertionFailedError: Condition not met within timeout 6. 
Did not receive all [KeyValue(2, \x00\x00\x00)] records from topic output (got 
[]) ==> expected:  but was: 
Stacktraceorg.opentest4j.AssertionFailedError: Condition not met within 
timeout 6. Did not receive all [KeyValue(2, \x00\x00\x00)] records from 
topic output (got []) ==> expected:  but was: at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63) 
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)  at 
org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210) at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)   
 at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379) 
 at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328) at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312) at 
org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302) at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:878)
 at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:827)
 at 
org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilFinalKeyValueRecordsReceived(IntegrationTestUtils.java:790)
 at 
org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreNullRecord(RestoreIntegrationTest.java:244)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15942) Implement ConsumerInterceptor

[Philip Nee (Jira)]

Philip Nee created KAFKA-15942:
--

 Summary: Implement ConsumerInterceptor
 Key: KAFKA-15942
 URL: https://issues.apache.org/jira/browse/KAFKA-15942
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Philip Nee


As title, we need to implement ConsumerInterceptor in the AsyncKafkaConsumer

 

This is the current code. The implementation would be very similar
{code:java}
if (interceptors != null)
interceptors.onCommit(offsets); {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15943) Flaky test - testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15943:


 Summary: Flaky test - 
testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – 
org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest
 Key: KAFKA-15943
 URL: https://issues.apache.org/jira/browse/KAFKA-15943
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 3.7.0
Reporter: Andrew Schofield


Error
org.opentest4j.AssertionFailedError: Timed out waiting for verifyAssignment 
result org.apache.kafka.tools.reassign.VerifyAssignmentResult@9c4dd302.  The 
latest result was org.apache.kafka.tools.reassign.VerifyAssignmentResult@cc845dc
Stacktrace
org.opentest4j.AssertionFailedError: Timed out waiting for verifyAssignment 
result org.apache.kafka.tools.reassign.VerifyAssignmentResult@9c4dd302.  The 
latest result was org.apache.kafka.tools.reassign.VerifyAssignmentResult@cc845dc
at 
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
at app//kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:1141)
at app//kafka.utils.TestUtils.waitUntilTrue(TestUtils.scala)
at 
app//org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest.waitForVerifyAssignment(ReassignPartitionsIntegrationTest.java:685)
at 
app//org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest.testHighWaterMarkAfterPartitionReassignment(ReassignPartitionsIntegrationTest.java:216)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at 
app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at 
app//org.junit.platform.e

[jira] [Created] (KAFKA-15944) Flaky test - verifyStore[cache=false, log=true, supplier=ROCKS_KV, kind=DSL] – org.apache.kafka.streams.integration.PositionRestartIntegrationTest

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15944:


 Summary: Flaky test - verifyStore[cache=false, log=true, 
supplier=ROCKS_KV, kind=DSL] – 
org.apache.kafka.streams.integration.PositionRestartIntegrationTest
 Key: KAFKA-15944
 URL: https://issues.apache.org/jira/browse/KAFKA-15944
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.7.0
Reporter: Andrew Schofield


Error
org.apache.kafka.common.errors.TimeoutException: The query never returned 
within the bound. Last result: 
StateQueryResult{partitionResults={0=SucceededQueryResult{result=org.apache.kafka.streams.state.internals.MeteredKeyValueStore$MeteredKeyValueTimestampedIterator@61b360a4,
 executionInfo=[], position=Position{position={input-topic={0=1, 
1=FailedQueryResult{failureReason=NOT_PRESENT, failure='The requested partition 
was not present at the time of the query.', executionInfo=[], position=null}}, 
globalResult=null}
Stacktrace
org.apache.kafka.common.errors.TimeoutException: The query never returned 
within the bound. Last result: 
StateQueryResult{partitionResults={0=SucceededQueryResult{result=org.apache.kafka.streams.state.internals.MeteredKeyValueStore$MeteredKeyValueTimestampedIterator@61b360a4,
 executionInfo=[], position=Position{position={input-topic={0=1, 
1=FailedQueryResult{failureReason=NOT_PRESENT, failure='The requested partition 
was not present at the time of the query.', executionInfo=[], position=null}}, 
globalResult=null}
Standard Output
[2023-11-28 22:52:47,353] INFO [Producer clientId=producer-129] Instantiated an 
idempotent producer. (org.apache.kafka.clients.producer.KafkaProducer:587)
[2023-11-28 22:52:47,466] INFO [Producer clientId=producer-129] ProducerId set 
to 0 with epoch 0 
(org.apache.kafka.clients.producer.internals.TransactionManager:502)
[2023-11-28 22:52:47,473] INFO [Producer clientId=producer-129] Closing the 
Kafka producer with timeoutMillis = 9223372036854775807 ms. 
(org.apache.kafka.clients.producer.KafkaProducer:1332)
[2023-11-28 22:52:47,531] INFO stream-client 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
 Kafka Streams version: test-version (org.apache.kafka.streams.KafkaStreams:914)
[2023-11-28 22:52:47,531] INFO stream-client 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
 Kafka Streams commit ID: test-commit-ID 
(org.apache.kafka.streams.KafkaStreams:915)
[2023-11-28 22:52:47,532] INFO stream-thread 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
 Creating restore consumer client 
(org.apache.kafka.streams.processor.internals.StreamThread:365)
[2023-11-28 22:52:47,537] INFO stream-thread 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
 Creating thread producer client 
(org.apache.kafka.streams.processor.internals.StreamThread:105)
[2023-11-28 22:52:47,538] INFO [Producer 
clientId=app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-producer]
 Instantiated an idempotent producer. 
(org.apache.kafka.clients.producer.KafkaProducer:587)
[2023-11-28 22:52:47,545] INFO stream-thread 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
 Creating consumer client 
(org.apache.kafka.streams.processor.internals.StreamThread:432)
[2023-11-28 22:52:47,545] INFO state-updater 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StateUpdater-1]
 State updater thread started 
(org.apache.kafka.streams.processor.internals.DefaultStateUpdater:135)
[2023-11-28 22:52:47,547] INFO [Producer 
clientId=app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-producer]
 ProducerId set to 1 with epoch 0 
(org.apache.kafka.clients.producer.internals.TransactionManager:502)
[2023-11-28 22:52:47,550] INFO stream-thread 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-consumer]
 Cooperative rebalancing protocol is enabled now 
(org.apache.kafka.streams.processor.internals.assignment.AssignorConfiguration:141)
[2023-11-28 22:52:47,552] INFO stream-client 
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
 State transi

[jira] [Created] (KAFKA-15945) Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15945:


 Summary: Flaky test - testSyncTopicConfigs() – 
org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest
 Key: KAFKA-15945
 URL: https://issues.apache.org/jira/browse/KAFKA-15945
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.7.0
Reporter: Andrew Schofield


Last seen: 
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14811/7/tests

Error
org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, 
because it's explicitly defined on the target topic!  ==> expected: <2000> but 
was: <8640>
Stacktrace
org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, 
because it's explicitly defined on the target topic!  ==> expected: <2000> but 
was: <8640>
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at 
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at 
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
at 
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1152)
at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testSyncTopicConfigs$8(MirrorConnectorsIntegrationBaseTest.java:780)
at 
app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331)
at 
app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379)
at 
app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328)
at 
app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312)
at 
app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302)
at 
app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testSyncTopicConfigs(MirrorConnectorsIntegrationBaseTest.java:774)
at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at 
java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at 
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at 
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.ex

[jira] [Resolved] (KAFKA-15046) Produce performance issue under high disk load

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-15046.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

merged the PR to trunk.

> Produce performance issue under high disk load
> --
>
> Key: KAFKA-15046
> URL: https://issues.apache.org/jira/browse/KAFKA-15046
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 3.3.2
>Reporter: Haruki Okada
>Assignee: Haruki Okada
>Priority: Major
>  Labels: performance
> Fix For: 3.7.0
>
> Attachments: image-2023-06-01-12-46-30-058.png, 
> image-2023-06-01-12-52-40-959.png, image-2023-06-01-12-54-04-211.png, 
> image-2023-06-01-12-56-19-108.png, image-2023-08-18-19-23-36-597.png, 
> image-2023-08-18-19-29-56-377.png
>
>
> * Phenomenon:
>  ** !image-2023-06-01-12-46-30-058.png|width=259,height=236!
>  ** Producer response time 99%ile got quite bad when we performed replica 
> reassignment on the cluster
>  *** RequestQueue scope was significant
>  ** Also request-time throttling happened at the incidental time. This caused 
> producers to delay sending messages in the mean time.
>  ** The disk I/O latency was higher than usual due to the high load for 
> replica reassignment.
>  *** !image-2023-06-01-12-56-19-108.png|width=255,height=128!
>  * Analysis:
>  ** The request-handler utilization was much higher than usual.
>  *** !image-2023-06-01-12-52-40-959.png|width=278,height=113!
>  ** Also, thread time utilization was much higher than usual on almost all 
> users
>  *** !image-2023-06-01-12-54-04-211.png|width=276,height=110!
>  ** From taking jstack several times, for most of them, we found that a 
> request-handler was doing fsync for flusing ProducerState and meanwhile other 
> request-handlers were waiting Log#lock for appending messages.
>  * 
>  ** 
>  *** 
> {code:java}
> "data-plane-kafka-request-handler-14" #166 daemon prio=5 os_prio=0 
> cpu=51264789.27ms elapsed=599242.76s tid=0x7efdaeba7770 nid=0x1e704 
> runnable  [0x7ef9a12e2000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.force0(java.base@11.0.17/Native 
> Method)
> at 
> sun.nio.ch.FileDispatcherImpl.force(java.base@11.0.17/FileDispatcherImpl.java:82)
> at 
> sun.nio.ch.FileChannelImpl.force(java.base@11.0.17/FileChannelImpl.java:461)
> at 
> kafka.log.ProducerStateManager$.kafka$log$ProducerStateManager$$writeSnapshot(ProducerStateManager.scala:451)
> at 
> kafka.log.ProducerStateManager.takeSnapshot(ProducerStateManager.scala:754)
> at kafka.log.UnifiedLog.roll(UnifiedLog.scala:1544)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.maybeRoll(UnifiedLog.scala:1523)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.append(UnifiedLog.scala:919)
> - locked <0x00060d75d820> (a java.lang.Object)
> at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:760)
> at 
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1170)
> at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1158)
> at 
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:956)
> at 
> kafka.server.ReplicaManager$$Lambda$2379/0x000800b7c040.apply(Unknown 
> Source)
> at 
> scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
> at 
> scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
> at scala.collection.mutable.HashMap.map(HashMap.scala:35)
> at 
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:944)
> at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:602)
> at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:666)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:175)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:75)
> at java.lang.Thread.run(java.base@11.0.17/Thread.java:829) {code}
>  * 
>  ** Also there were bunch of logs that writing producer snapshots took 
> hundreds of milliseconds.
>  *** 
> {code:java}
> ...
> [2023-05-01 11:08:36,689] INFO [ProducerStateManager partition=xxx-4] Wrote 
> producer snapshot at offset 1748817854 with 8 producer ids in 809 ms. 
> (kafka.log.ProducerStateManager)
> [2023-05-01 11:08:37,319] INFO [ProducerStateManager partition=yyy-34] Wrote 
> producer snapshot at offset 247996937813 with 0 producer ids in 547 ms. 
> (kafka.log.ProducerStateManager)
> [2023-05-01 11:08:38,887] INFO [ProducerStateManager partition=zzz-9] Wrote 
> producer snapshot at offset 226222355404 with 0 producer ids in 576 ms

[jira] [Resolved] (KAFKA-5046) Support file rotation in FileStreamSource Connector

[Konstantine Karantasis (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-5046.
---
Resolution: Won't Fix

> Support file rotation in FileStreamSource Connector
> ---
>
> Key: KAFKA-5046
> URL: https://issues.apache.org/jira/browse/KAFKA-5046
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Minor
>
> Currently when a source file is moved (for file rotation purposes, or between 
> restarts of Kafka Connect) the FileStreamSource Connector can not detect the 
> change, because it only uses the filename as key to its offset tracking. 
> Nevertheless, file rotation can be detected easily by checking basic file 
> attributes such as the {{fileKey}} in platforms that this attribute is 
> supported (for instance file key includes the device id and the inode in unix 
> based filesystems) and the file's creation time.
> Such checks need to take place when the task starts and when no more records 
> are read during a call to {{poll}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15946) AsyncKafkaConsumer should retry commits on the application thread instead of autoretry

[Philip Nee (Jira)]

Philip Nee created KAFKA-15946:
--

 Summary: AsyncKafkaConsumer should retry commits on the 
application thread instead of autoretry
 Key: KAFKA-15946
 URL: https://issues.apache.org/jira/browse/KAFKA-15946
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Philip Nee


The original design was that the network thread always completes the future 
whether succeeds or fails.  However, in the current patch, I mis-added 
auto-retry functionality because commitSync wasn't retrying.  What we should be 
doing is, the commit sync API should catch the RetriableExceptions and resend 
another commit until timesout.

```

if (error.exception() instanceof RetriableException) {
log.warn("OffsetCommit failed on partition {} at offset {}: {}", tp, offset, 
error.message());
handleRetriableError(error, response);
retry(responseTime);  <--- We probably shouldn't do this.
return;
}

```

 

and 

 

```

@Override
public void commitSync(Map offsets, Duration 
timeout) {
acquireAndEnsureOpen();
long commitStart = time.nanoseconds();
try {
CompletableFuture commitFuture = commit(offsets, true); <-- we probably 
should retry here
ConsumerUtils.getResult(commitFuture, time.timer(timeout));
} finally {
wakeupTrigger.clearTask();
kafkaConsumerMetrics.recordCommitSync(time.nanoseconds() - commitStart);
release();
}
}

```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15947) Null pointer on LZ4 compression since Kafka 3.6

[Ludo (Jira)]

Ludo created KAFKA-15947:


 Summary: Null pointer on LZ4 compression since Kafka 3.6
 Key: KAFKA-15947
 URL: https://issues.apache.org/jira/browse/KAFKA-15947
 Project: Kafka
  Issue Type: Bug
  Components: compression
Affects Versions: 3.6.0
Reporter: Ludo


I have a Kafka Stream application running well since month using client version 
{{3.5.1 }}with 3.5.1 (bitnami image: {{bitnami/3.5.1-debian-11-r44)}} using{{ 
compression.type: "lz4"}}

I've recently updated a my kafka server to kafka 3.6 (bitnami image: 
{{{}bitnami/kafka:3.6.0-debian-11-r0){}}}.
 
The startup is working well for days, and after some time, Kafka Stream crash 
and Kafka output a lot of NullPointerException on the console: 
 
{code:java}
org.apache.kafka.common.KafkaException: java.lang.NullPointerException: Cannot 
invoke "java.nio.ByteBuffer.hasArray()" because "this.intermediateBufRef" is 
null
at 
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:134)
at 
org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
at 
org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
at 
org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
at 
org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
at 
org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
at kafka.log.UnifiedLog.$anonfun$append$2(UnifiedLog.scala:805)
at kafka.log.UnifiedLog.append(UnifiedLog.scala:1845)
at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
at 
kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
at 
kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at 
scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at 
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
at 
kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:754)
at 
kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke 
"java.nio.ByteBuffer.hasArray()" because "this.intermediateBufRef" is null
at 
org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
at 
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:132)
... 25 more {code}
At the same time the Kafka Stream raise this error:

 
{code:java}
org.apache.kafka.streams.errors.StreamsException: Error encountered sending 
record to topic kestra_workertaskresult for task 3_6 due 
to:org.apache.kafka.common.errors.UnknownServerException: The server 
experienced an unexpected error when processing the request.Written offsets 
would not be recorded and no more records would be sent since this is a fatal 
error.at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:297)at
 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:284)at
 
org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505)at
 
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)at
 
org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)at
 
org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)at
 
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:772)at 
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:757)at 
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:709)at 
org.apache.kafka.clients.producer.internals.Sender.co

[jira] [Created] (KAFKA-15948) Refactor AsyncKafkaConsumer shutdown

[Philip Nee (Jira)]

Philip Nee created KAFKA-15948:
--

 Summary: Refactor AsyncKafkaConsumer shutdown
 Key: KAFKA-15948
 URL: https://issues.apache.org/jira/browse/KAFKA-15948
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Philip Nee


Upon closing we need a round trip from the network thread to the application 
thread and then back to the network thread to complete the callback invocation. 
 Currently, we don't have any of that.  I think we need to refactor our closing 
mechanism.  There are a few points to the refactor:
 # The network thread should know if there's a custom user callback to trigger 
or not.  If there is, it should wait for the callback completion to send a 
leave group.  If not, it should proceed with the shutdown.
 # The application thread sends a closing signal to the network thread and 
continuously polls the background event handler until time runs out.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2429

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15947) Null pointer on LZ4 compression since Kafka 3.6

[Ludo (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ludo resolved KAFKA-15947.
--
Fix Version/s: 3.6.1
   Resolution: Duplicate

> Null pointer on LZ4 compression since Kafka 3.6
> ---
>
> Key: KAFKA-15947
> URL: https://issues.apache.org/jira/browse/KAFKA-15947
> Project: Kafka
>  Issue Type: Bug
>  Components: compression
>Affects Versions: 3.6.0
>Reporter: Ludo
>Priority: Major
> Fix For: 3.6.1
>
>
> I have a Kafka Stream application running well since month using client 
> version {{3.5.1 }}with 3.5.1 (bitnami image: {{bitnami/3.5.1-debian-11-r44)}} 
> using{{ compression.type: "lz4"}}
> I've recently updated a my kafka server to kafka 3.6 (bitnami image: 
> {{{}bitnami/kafka:3.6.0-debian-11-r0){}}}.
>  
> The startup is working well for days, and after some time, Kafka Stream crash 
> and Kafka output a lot of NullPointerException on the console: 
>  
> {code:java}
> org.apache.kafka.common.KafkaException: java.lang.NullPointerException: 
> Cannot invoke "java.nio.ByteBuffer.hasArray()" because 
> "this.intermediateBufRef" is null
>   at 
> org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:134)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
>   at 
> org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
>   at 
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
>   at 
> org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
>   at kafka.log.UnifiedLog.$anonfun$append$2(UnifiedLog.scala:805)
>   at kafka.log.UnifiedLog.append(UnifiedLog.scala:1845)
>   at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
>   at 
> kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
>   at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
>   at 
> kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
>   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>   at 
> scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
>   at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
>   at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
>   at scala.collection.TraversableLike.map(TraversableLike.scala:286)
>   at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>   at 
> kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
>   at 
> kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:754)
>   at 
> kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
>   at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.lang.NullPointerException: Cannot invoke 
> "java.nio.ByteBuffer.hasArray()" because "this.intermediateBufRef" is null
>   at 
> org.apache.kafka.common.utils.ChunkedBytesStream.(ChunkedBytesStream.java:89)
>   at 
> org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:132)
>   ... 25 more {code}
> At the same time the Kafka Stream raise this error:
>  
> {code:java}
> org.apache.kafka.streams.errors.StreamsException: Error encountered sending 
> record to topic kestra_workertaskresult for task 3_6 due 
> to:org.apache.kafka.common.errors.UnknownServerException: The server 
> experienced an unexpected error when processing the request.Written offsets 
> would not be recorded and no more records would be sent since this is a fatal 
> error.at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:297)at
>  
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:284)at
>  
> org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505)at
>  
> org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)at
>  
> org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)at
>  
> org.apache.kafka.clients

Re: [VOTE] 3.6.1 RC0

[Justine Olshan]

I built from source and ran a simple transactional produce bench. I ran a
handful of unit tests as well.
I scanned the docs and everything looked reasonable.

I was wondering if we got the system test results mentioned > System tests:
Still running I'll post an update once they complete.

Justine

On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison 
wrote:

> Hi Josep,
>
> Good catch!
> If it's the only issue we find, I don't think we should block the
> release just to fix that.
>
> If we find another issue, I'll backport it before running another RC,
> otherwise I'll backport it once 3.6.1 is released.
>
> Thanks,
> Mickael
>
> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat 
> wrote:
> >
> > Hi Mickael,
> > This PR[1] made me realize NOTICE-binary is missing the notice for
> > commons-io. I don't know if it's a blocker or not. I can cherry pick the
> > commit to the 3.6 branch if you want.
> >
> > Best,
> >
> >
> > [1]: https://github.com/apache/kafka/pull/14865
> >
> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat  wrote:
> >
> > > Hi Mickael,
> > > Thanks for running the release. It's a +1 for me (non-binding).
> > > I did the following:
> > > - Verified artifact's signatures and hashes
> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
> > > - Compiled source code
> > > - Run unit tests and integration tests
> > > - Run getting started with ZK and KRaft
> > >
> > > Best,
> > >
> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
> > >> 2. Ran all the unit and integration tests.
> > >> 3. Ran quickstart and verified the produce-consume on a 3 node
> cluster.
> > >> 4. Verified the tiered storage functionality with local-tiered
> storage.
> > >>
> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri <
> fedeval...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Mickael,
> > >> >
> > >> > - Build from source (Java 17, Scala 2.13)
> > >> > - Run unit and integration tests
> > >> > - Run custom client apps using staging artifacts
> > >> >
> > >> > +1 (non binding)
> > >> >
> > >> > Thanks
> > >> > Fede
> > >> >
> > >> >
> > >> >
> > >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz 
> wrote:
> > >> > >
> > >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and the
> staged
> > >> > Maven
> > >> > > repo for my tests. All seems to work fine.
> > >> > >
> > >> > > Thanks
> > >> > > Jakub
> > >> > >
> > >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison <
> mimai...@apache.org>
> > >> > wrote:
> > >> > >
> > >> > > > Hello Kafka users, developers and client-developers,
> > >> > > >
> > >> > > > This is the first candidate for release of Apache Kafka 3.6.1.
> > >> > > >
> > >> > > > This is a bugfix release with several fixes, including
> dependency
> > >> > > > version bumps for CVEs.
> > >> > > >
> > >> > > > Release notes for the 3.6.1 release:
> > >> > > >
> > >> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
> > >> > > >
> > >> > > > *** Please download, test and vote by Friday, December 1
> > >> > > >
> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > >> > > > https://kafka.apache.org/KEYS
> > >> > > >
> > >> > > > * Release artifacts to be voted upon (source and binary):
> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
> > >> > > >
> > >> > > > * Maven artifacts to be voted upon:
> > >> > > >
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > > >
> > >> > > > * Javadoc:
> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
> > >> > > >
> > >> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
> > >> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
> > >> > > >
> > >> > > > PR for updating docs:
> > >> > > > https://github.com/apache/kafka-site/pull/568
> > >> > > >
> > >> > > > * Documentation:
> > >> > > > https://kafka.apache.org/36/documentation.html
> > >> > > >
> > >> > > > * Protocol:
> > >> > > > https://kafka.apache.org/36/protocol.html
> > >> > > >
> > >> > > > * Successful Jenkins builds for the 3.6 branch:
> > >> > > > Unit/integration tests: We still have a lot of flaky tests in
> the
> > >> 3.6
> > >> > > > branch. Looking at the last few 3.6 builds in
> > >> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it
> seems
> > >> all
> > >> > > > tests passed at least once apart from
> > >> > > > ClusterConnectionStatesTest.testSingleIP(). There's
> > >> > > > https://issues.apache.org/jira/browse/KAFKA-15762 to fix that
> test.
> > >> > > > System tests: Still running I'll post an update once they
> complete.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Mickael
> > >> > > >
> > >> >
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +49171

[jira] [Created] (KAFKA-15949) Improve the KRaft metadata version related messages

[Jakub Scholz (Jira)]

Jakub Scholz created KAFKA-15949:


 Summary: Improve the KRaft metadata version related messages
 Key: KAFKA-15949
 URL: https://issues.apache.org/jira/browse/KAFKA-15949
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 3.6.0
Reporter: Jakub Scholz


Various error messages related to KRaft seem to use very different style and 
formatting. Just for example in the {{StorageTool}} Scala class, there are two 
different examples:
 * {{Must specify a valid KRaft metadata version of at least 3.0.}}
 ** Refers to "metadata version"
 ** Refers to the version as 3.0 (although strictly speaking 3.0-IV0 is not 
valid for KRaft)
 * {{SCRAM is only supported in metadataVersion IBP_3_5_IV2 or later.}}
 ** Talks about "metadataVersion"
 ** Refers to "IBP_3_5_IV2" instead of "3.5" or "3.5-IV2"

Other pieces of Kafka code seem to also talk about "metadata.version" for 
example.

For users, it would be nice if the style and formats used were the same 
everywhere. Would it be worth unifying messages like this? If yes, what would 
be the preferred style to use?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15311) Fix docs about reverting to ZooKeeper mode during KRaft migration

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-15311.
--
Fix Version/s: 3.7.0
   Resolution: Fixed

> Fix docs about reverting to ZooKeeper mode during KRaft migration
> -
>
> Key: KAFKA-15311
> URL: https://issues.apache.org/jira/browse/KAFKA-15311
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin McCabe
>Assignee: Colin McCabe
>Priority: Minor
> Fix For: 3.7.0
>
>
> The cocs incorrectly state that reverting to ZooKeeper mode during KRaft 
> migration is not possible



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15950) CommunicationEvent should be scheduled with EarliestDeadlineFunction

[Jun Rao (Jira)]

Jun Rao created KAFKA-15950:
---

 Summary: CommunicationEvent should be scheduled with 
EarliestDeadlineFunction
 Key: KAFKA-15950
 URL: https://issues.apache.org/jira/browse/KAFKA-15950
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.7.0
Reporter: Jun Rao


Currently, CommunicationEvent is scheduled with DeadlineFunction, which ignores 
the schedule time for an existing event. This wasn't an issue when 
CommunicationEvent is always periodic. However, with KAFKA-15360,  a 
CommunicationEvent could be scheduled immediately for offline dirs. If a 
periodic CommunicationEvent is scheduled after the immediate CommunicationEvent 
in KafkaEventQueue, the former will cancel the latter, but leaves the schedule 
time to be periodic. This will unnecessarily delay the communication of the 
failed dir to the controller. 
 
Using EarliestDeadlineFunction will fix this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-996: Pre-Vote

[Jun Rao]

Hi, Alyssa,

Thanks for the KIP. A few comments below.

10. "If a server happens to receive multiple VoteResponses from another
server for a particular VoteRequest, it can take the first and ignore the
rest.": Could you explain why a server would receive multiple responses for
the same request?

11. "e.g. S1 in the below diagram pg. 41)": What is pg. 41?

12. "if a server attempts to send out a Pre-Vote request while any other
server in the quorum does not understand it, it will get back an
UnsupportedVersionException from the network client and knows to default
back to the old behavior."
12.1 Based on ApiVersion, a server knows whether a peer supports PreVote or
not. If it doesn't, there is no need for the server to send a PreVote
request only to be rejected, right?
12.2 What happens when some servers understand PreVote while some others
don't?

Thanks,

Jun



On Wed, Nov 29, 2023 at 2:11 AM Luke Chen  wrote:

> Hi Alyssa,
>
> Thanks for the KIP!
> This is an important improvement for KRaft quorum.
>
> Some comments:
> 1. Follower transitions to: Prospective: After expiration of the election
> timeout
> -> Is this the fetch timeout, not election timeout?
>
> 2. I also agree we don't bump the epoch in prospective state.
>  A candidate will now send a VoteRequest with the PreVote field set to true
> and CandidateEpoch set to its [epoch + 1] when its election timeout
> expires.
> -> What is "CandidateEpoch"? And I thought you've agreed to not set [epoch
> + 1] ?
>
> Thanks.
> Luke
>
> On Wed, Nov 29, 2023 at 2:06 AM Alyssa Huang 
> wrote:
>
> > Thanks Jose, I've updated the KIP to reflect your and Jason's
> suggestions!
> >
> > On Tue, Nov 28, 2023 at 9:54 AM José Armando García Sancio
> >  wrote:
> >
> > > Hi Alyssa
> > >
> > > On Mon, Nov 27, 2023 at 1:40 PM Jason Gustafson
> > >  wrote:
> > > > 2. Do you think the pretend epoch bump is necessary? Would it be
> > simpler
> > > to
> > > > change the prevote acceptance check to assert a greater than or equal
> > > epoch?
> > >
> > > I agree with Jason it would be better if all of the requests always
> > > sent the current epoch. For the VoterRequest, it should be correct for
> > > the prospective node to not increase the epoch and send the current
> > > epoch and id. Since there are two states (prospective and candidate)
> > > that can send a VoteRequest, maybe we can change the field name to
> > > just ReplicaEpoch and ReplicaId.
> > >
> > > Thanks,
> > > --
> > > -José
> > >
> >
>

Re: [DISCUSS] KIP-996: Pre-Vote

[José Armando García Sancio]

Hi Alyssa,

1. In the schema for VoteRequest and VoteResponse, you are using
"boolean" as the type keyword. The correct keyword should be "bool"
instead.

2. In the states and state transaction table you have the following entry:
>  * Candidate transitions to:
> *...
> *Prospective: After expiration of the election timeout

Can you explain the reason a candidate would transition back to
prospective? If a voter transitions to the candidate state it is
because the voters don't support KIP-996 or the replica was able to
win the majority of the votes at some point in the past. Are we
concerned that the network partition might have occurred after the
replica has become a candidate? If so, I think we should state this
explicitly in the KIP.

3. In the proposed section and state transition section, I think it
would be helpful to explicitly state that we have an invariant that
only the prospective state can transition to the candidate state. This
transition to the candidate state from the prospective state can only
happen because the replica won the majority of the votes or there is
at least one remote voter that doesn't support pre-vote.

4. I am a bit confused by this paragraph
> A candidate will now send a VoteRequest with the PreVote field set to true 
> and CandidateEpoch set to its [epoch + 1] when its election timeout expires. 
> If [majority - 1] of VoteResponse grant the vote, the candidate will then 
> bump its epoch up and send a VoteRequest with PreVote set to false which is 
> our standard vote that will cause state changes for servers receiving the 
> request.

I am assuming that "candidate" refers to the states enumerated on the
table above this quote. If so, I think you mean "prospective" for the
first candidate.

CandidateEpoch should be ReplicaEpoch.

[epoch + 1] should just be epoch. I thought we agreed that replicas
will always send their current epoch to the remote replicas.

5. I am a bit confused by this bullet section
> true if the server receives less than [majority] VoteResponse with 
> VoteGranted set to false within [election.timeout.ms + a little randomness] 
> and the first bullet point does not apply
 Explanation for why we don't send a standard vote at this point
is explained in rejected alternatives.

Can we explain this case in plain english? I assume that this case is
trying to cover the scenario where the election timer expired but the
prospective candidate hasn't received enough votes (granted or
rejected) to make a decision if it could win an election.

6.
> Yes. If a leader is unable to receive fetch responses from a majority of 
> servers, it can impede followers that are able to communicate with it from 
> voting in an eligible leader that can communicate with a majority of the 
> cluster.

In general, leaders don't receive fetch responses. They receive FETCH
requests. Did you mean "if a leader is able to send FETCH responses to
the majority - 1 of the voters, it can impede fetching voters
(followers) from granting their vote to prospective candidates. This
should stop prospective candidates from getting enough votes to
transition to the candidate state and increase their epoch".

7.
> Check Quorum ensures a leader steps down if it is unable to receive fetch 
> responses from a majority of servers.

I think you mean "... if it is unable to receive FETCH requests from
the majority - 1 of the voters".

8. At the end of the Proposed changes section you have the following:
> The logic now looks like the following for servers receiving VoteRequests 
> with PreVote set to true:
>
> When servers receive VoteRequests with the PreVote field set to true, they 
> will respond with VoteGranted set to
>
> * true if they are not a Follower and the epoch and offsets in the Pre-Vote 
> request satisfy the same requirements as a standard vote
> * false if they are a Follower or the epoch and end offsets in the Pre-Vote 
> request do not satisfy the requirements

This seems to duplicate the same algorithm that was stated earlier in
the section.

9. I don't understand this rejected idea: Sending Standard Votes after
failure to win Pre-Vote

In your example in the "Disruptive server scenarios" voters 4 and 5
are partitioned from the majority of the voters. We don't want voters
4 and 5 increasing their epoch and transitioning to the candidate
state else they would disrupt the quorum established by voters 1, 2
and 3.


Thanks,
-- 
-José

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Justine Olshan]

Hey Raman,
Thanks for the KIP! I think this will be super useful.

Given https://issues.apache.org/jira/browse/KAFKA-15546 -- do you think it
would be useful to specify the duration of the completed transaction rather
than the time since the start in the describe output?
We would probably want to specify the transaction is completed as well to
differentiate from the response that does not have the tagged field. This
would more clearly resolve the ticket.
Maybe this was the plan but it wasn't clear from the KIP.

Thanks,
Justine

On Tue, Nov 28, 2023 at 12:38 PM Jason Gustafson 
wrote:

> Hey Raman,
>
> Thanks for the KIP! I think it makes sense. I agree that this becomes
> especially useful in the context of KIP-939 because transactions can last
> an indefinite amount of time, but it is useful even today. A large cluster
> may have a very large number of ongoing transactions at any time, so
> providing a better way to filter by time will make the tools much more
> efficient for this common use case.
>
> I have just a couple small comments.
>
> 1. In `ListTransactionsOptions`, we use a long in the setter for the
> duration filter. Can we use `Duration` instead?
> 2. I think we need to expose `TransactionLastUpdateTimeMs` on
> `TransactionDescription` as well, right?
>
> Thanks,
> Jason
>
> On Tue, Nov 28, 2023 at 8:34 AM Kirk True  wrote:
>
> > Hi Raman,
> >
> > Thanks for the KIP!
> >
> > Questions/comments:
> >
> >  1. Is there a Jira created for this? The Jira link still points to
> > KAFKA-1.
> >  2. There's a minor typo in the ListTransactionsRequest documentation for
> > the DurationFilter: trsanactions.
> >  3. Is the TransactionStartTimeMs return value in the
> > DescribeTransactionsResponse nullable?
> >  4. The API uses the general terminology of a "duration filter" but the
> > CLI uses the specific phrase "running longer than ms." These are both
> > referring to the same value, right? Can the naming of these two be more
> > consistent?
> >  5. What happens when a user runs the updated kafka-transactions.sh
> script
> > (using the new argument) against an older broker that doesn't support the
> > new filter? Does the user get an error, a warning, or a silent ignoring
> of
> > the filter?
> >
> > Thanks,
> > Kirk
> >
> > On Wed, Nov 15, 2023, at 3:03 PM, Raman Verma wrote:
> > > Thanks Artem,
> > >
> > > I have made changes to the `Public Interfaces` and `Compatibility...`
> > > sections to incorporate your comment.
> > >
> > > On Mon, Nov 6, 2023 at 3:44 PM Raman Verma 
> > wrote:
> > >
> > > > I would like to start a discussion on KIP-994
> > > >
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
> > > >
> > > >
> > >
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2430

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #121

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 413481 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldRemovePausedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldRemovePausedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldRemoveTasksFromAndClearInputQueueOnShutdown() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldRemoveTasksFromAndClearInputQueueOnShutdown() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldThrowIfAddingActiveTasksWithSameId() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldThrowIfAddingActiveTasksWithSameId() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > 
shouldRestoreActiveStatefulTasksAndUpdateStandbyTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > 
shouldRestoreActiveStatefulTasksAndUpdateStandbyTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldPauseStandbyTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldPauseStandbyTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldThrowIfStatefulTaskNotInStateRestoring() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultStateUpdaterTest > shouldThrowIfStatefulTaskNotInStateRestoring() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetUncaughtStreamsException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetUncaughtStreamsException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenRequired() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldProcessTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldProcessTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED

Exception: java.lang.AssertionError thrown from the UncaughtExceptionHandler in 
thread "TaskExecutor"

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> More than one query result throws IllegalArgumentException PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Zero query results shouldn't error STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Zero query results shouldn't error PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> Valid query results still works STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > StateQueryResultTest 
> V

Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery

[Hanyu (Peter) Zheng]

Thank you Bruno,
1. Thank you for the notification. I have updated the ticket link
accordingly.
2. Certainly, I'll update the KIP name. Should I initiate a new discussion
for it, because if I change the name, the link will change.
3. Understood, I will add that to the KIP.
4. I propose we accept both
`WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2)` and
`WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2)`,
while also reusing the existing `withKey` method.
5. Following a discussion with Matthias, we've decided to defer the
implementation of order guarantees to a future KIP.

Sincerely,
Hanyu

On Wed, Nov 29, 2023 at 6:22 AM Bruno Cadonna  wrote:

> Hi,
>
> Thanks for the updates!
>
>
> 1.
> Could you please link the correct ticket in the KIP?
>
> 2.
> Could you please adapt the motivation section and the title to the
> updated goal of the KIP? There is no fetch() or fetchAll() method in the
> query class.
>
> 3.
> Could you please add the "// newly added" comment to all parts that were
> newly added? That is methods lowerKeyBound() and upperKeyBound().
>
> 4.
> We should use a more fluent API as I proposed in my last e-mail:
>
> Here again
>
> WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2);
> WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2);
> WindowRangeQuery.withKeyRange(key1, key2).fromTime(time1).toTime(time2);
>
> 5.
> We should also consider the order of the results similar as we did in
> KIP-968. Alternatively, we do not guarantee any order and postpone the
> order guarantees to a future KIP.
>
>
> Best,
> Bruno
>
>
>
> On 11/17/23 3:11 AM, Matthias J. Sax wrote:
> > Thanks for the KIP.
> >
> > Given how `WindowRangeQuery` works right now, it's really time to
> > improve it.
> >
> >
> > 1) Agree. It's not clear what will be added right now. I think we should
> > deprecate existing `getKey()` w/o an actually replacement? For
> > `getFromKey` and `getToKey` we should actually be `lowerKeyBound()` and
> > `upperKeyBound()` to align to KIP-969?
> >
> > Also wondering if we should deprecate existing `withKey()` and
> > `withWindowStartRange`? `withKey` only works for SessionStores and
> > implements a single-key full-time-range query. Similarly,
> > `withWindowStartRange` only works for WindowedStore and implements an
> > all-key time-range query. Thus, both are rather special and it seems the
> > aim of this KIP is to generalize `WindowRangeQuery` to arbitrary
> > key-range/time-range queries?
> >
> > What raises one question about time-range semantics, given that we query
> > windows with different semantics.
> >
> >   - The current `WindowStore` semantics used for
> > `WindowRangeQuery#withWindowStartRange` is considering only the window
> > start time, ie, the window-start time must fall into the query
> > time-range to be returned.
> >
> >   - In contrast, `SessionStore` time ranges base on `findSession` use
> > earliest-session-end-time and latest-session-end-time and thus implement
> > an "window-bounds / search-time-range overlap query".
> >
> > Is there any concern about semantic differences? I would also be
> > possible to use the same semantics for both query types, and maybe even
> > let the user pick with semantics they want (let users chose might
> > actually be the best thing to do)? -- We can also do this incrementally,
> > and limit the scope of this KIP (or keep the full KIP scope but
> > implement it incrementally only)
> >
> > Btw: I think we should not add any ordering at this point, and
> > explicitly state that no ordering is guarantee whatsoever at this point.
> >
> >
> >
> > 2) Agreed. We should deprecate `getFromTime` and `getToTime` and add new
> > method `fromTime` and `toTime`.
> >
> >
> >
> > 3) About the API. If we move forward with general key-range/time-range I
> > agree that a more modular approach might be nice. Not sure right now,
> > what the best approach would be for this? Looking into KIP-969, we might
> > want to have:
> >
> >   - static withKeyRange
> >   - static withLowerKeyBound
> >   - static withUpperKeyBound
> >   - static withAllKeys (KIP-969 actually uses `allKeys` ?)
> >   - fromTime
> >   - toTime
> >
> > with default-time range would be "all / unbounded" ?
> >
> >
> >
> > 10: you mentioned that `WindowKeyQuery` functionality can be covered by
> > `WindowRangeQuery`. I agree. For this case, it seems we want to
> > deprecate `WindowKeyQuery` entirely?
> >
> >
> >
> > -Matthias
> >
> > On 11/16/23 1:19 AM, Bruno Cadonna wrote:
> >> Hi Hanyu,
> >>
> >> Thanks for the KIP!
> >>
> >> 1)
> >> Could you please mark the pieces that you want to add to the API in
> >> the code listing in the KIP? You can add a comment like "// newly
> >> added" or similar. That would make reading the KIP a bit easier
> >> because one does not need to compare your code with the code in the
> >> current codebase.
> >>
> >> 2)
> >> Could you -- as a side cleanup -- also change the getters to not use
> >> the get-prefix anymo

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

[Qichao Chu]

Hi Vahid,

Thank you so much for the review and voting!

Best,
Qichao

On Wed, Nov 29, 2023 at 2:55 PM Vahid Hashemian  wrote:

> Hi Qichao,
>
> Thanks for answering my questions and updating the KIP accordingly.
>
> It looks good to me.
>
> --Vahid
>
>
> On Tue, Nov 28, 2023, 7:22 PM Qichao Chu  wrote:
>
> > Hi Vahid,
> >
> > Thank you for taking the time to review the KIP and asking great
> questions.
> >
> > The execution path mentioned is exactly how this KIP is going to
> function.
> > We believe this config, compared with many other configurations like
> quota,
> > will not pose significant alteration to the broker's functionality. Thus
> > overwriting the permanent config dynamically is acceptable. After
> thinking
> > about this topic again, I think your question makes a lot of sense and we
> > shouldn't override the permanent config. Permanent config usually
> > represents the normal operation condition, so the temporal config should
> be
> > removed after the cluster is restarted/re-provisioned to be 'clean'. This
> > also helps to prevent undesirable behavior and to reduce the risk
> involved
> > in operation. I have updated the KIP to mention this behavior.
> >
> > `Medium` level is not mentioned as we don't have any medium-level metrics
> > for now. This is for future extension. I have also updated the KIP to
> > reflect this information.
> >
> > Best,
> > Qichao
> >
> >
> >
> > On Tue, Nov 28, 2023 at 9:22 AM Vahid Hashemian 
> wrote:
> >
> > > Hi Qichao,
> > >
> > > Thanks for proposing this KIP. It'd be super valuable to have the
> ability
> > > to have those partition level metrics for Kafka topics.
> > >
> > > Sorry I'm late to the discussion. I just wanted to bring up a point
> > > for clarification and one question:
> > >
> > > Let's assume that a production cluster cannot afford to enable high
> > > verbosity on a permanent basis (at least not for all topics) due to
> > > performance concerns.
> > >
> > > Since this new config can be set dynamically, in case of an issue or
> > > investigation that warrants obtaining partition level metrics, one can
> > > simply enable high verbosity for select topic(s), temporarily collect
> > > metrics at partition level, and then change the config back to the
> > previous
> > > setting. Since the config values are not set incrementally, the
> operator
> > > would need to run a `describe` to get the existing config first, and
> then
> > > amend it to enable high verbosity for the topic(s) of interest.
> Finally,
> > > when the investigation concludes, the config has to be reverted to its
> > > permanent setting.
> > >
> > > If the above execution path makes sense, in case the operator forgets
> to
> > > take an inventory of the existing (permanent) config and simply
> > overwrites
> > > it, then that permanent config will be gone and not retrievable. Is
> this
> > > correct?
> > >
> > > We usually don't need to temporarily change broker configs and I see
> this
> > > config as one that can be temporarily changed. So keeping track of what
> > the
> > > value was before the change is rather important.
> > >
> > > Aside from this point, my question is: What's the impact of `medium`
> > > setting for `level`? I couldn't find it described in the KIP.
> > >
> > > Thanks!
> > > --Vahid
> > >
> > >
> > >
> > > On Mon, Nov 13, 2023 at 5:34 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Thank you for updating the KIP Qichao.
> > > >
> > > > I don't have any more questions or suggestions. Looks good to move
> > > forward
> > > > from my perspective.
> > > >
> > > >
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Fri, Nov 10, 2023 at 2:25 PM Qichao Chu 
> > > > wrote:
> > > >
> > > > > Thank you again for the nice suggestions, Jorge!
> > > > > I will wait for Divij's response and move it to the vote stage once
> > the
> > > > > generic filter part reached concensus.
> > > > >
> > > > > Qichao Chu
> > > > > Software Engineer | Data - Kafka
> > > > > [image: Uber] 
> > > > >
> > > > >
> > > > > On Fri, Nov 10, 2023 at 6:49 AM Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jo...@gmail.com> wrote:
> > > > >
> > > > > > Hi Qichao,
> > > > > >
> > > > > > Thanks for updating the KIP, all updates look good to me.
> > > > > >
> > > > > > Looking forward to see this KIP moving forward!
> > > > > >
> > > > > > Cheers,
> > > > > > Jorge.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 8 Nov 2023 at 08:55, Qichao Chu  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Divij,
> > > > > > >
> > > > > > > Thank you for the feedback. I updated the KIP to make it a
> little
> > > bit
> > > > > > more
> > > > > > > generic: filters will stay in an array instead of different
> > > top-level
> > > > > > > objects. In this way, if we need language filters in the
> future.
> > > The
> > > > > > logic
> > > > > > > relationship of filters is also added.
> > > > > > >
> > > > > > > Hi Jorge,
> > > > > > >
> > > > > > > Thank you for the r

[DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

[Ashwin]

Hi all,

I'd like to begin discussion on KIP-995 which proposes to allow users
to specify initial offset as part of the request to create a connector

https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors

During the discussion for KIP-980
,
which proposed the creation of connectors in STOPPED state, there was
a suggestion to also allow setting the initial offset for a connector
in the connector creation API. The proposal was deemed valid

(point no.4) but was deferred to a future KIP. This KIP proposes to
implement that change and builds upon the changes introduced in
KIP-875 

and KIP-980 
.

Thanks,
Ashwin