[jira] [Created] (KAFKA-7776) Kafka Connect values converter parsing of ISO8601 not working properly
Andrew Schofield created KAFKA-7776: --- Summary: Kafka Connect values converter parsing of ISO8601 not working properly Key: KAFKA-7776 URL: https://issues.apache.org/jira/browse/KAFKA-7776 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.0.0, 1.1.0 Reporter: Andrew Schofield In org.apache.kafka.connect.data.Values, the Values.convertToDate/Time/Timestamp methods are intended to be able to accept Strings in ISO8601 format and convert into the Kafka Connect formats. However, the parser for strings incorrectly tokenizes the strings (having real trouble with colons) which means that the correct ISO8601 format is never actual presented as a single piece to the code that converts it into java.util.Date. The parser needs to be enhanced to accept an ISO8601 string as a single token, probably only when it knows that the intended use is as one of the date-based logical types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7841) KIP-419 Safely notify Kafka Connect SourceTask is stopped
Andrew Schofield created KAFKA-7841: --- Summary: KIP-419 Safely notify Kafka Connect SourceTask is stopped Key: KAFKA-7841 URL: https://issues.apache.org/jira/browse/KAFKA-7841 Project: Kafka Issue Type: New Feature Components: KafkaConnect Affects Versions: 2.2.0 Reporter: Andrew Schofield Implements KIP 419. https://cwiki.apache.org/confluence/display/KAFKA/KIP-419%3A+Safely+notify+Kafka+Connect+SourceTask+is+stopped -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-15831) List Client Metrics Configuration Resources
Andrew Schofield created KAFKA-15831: Summary: List Client Metrics Configuration Resources Key: KAFKA-15831 URL: https://issues.apache.org/jira/browse/KAFKA-15831 Project: Kafka Issue Type: Improvement Components: admin Reporter: Andrew Schofield Assignee: Andrew Schofield This JIRA tracks the development of KIP-1000 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15840) Correct initialization of ConsumerGroupHeartbeat by client
Andrew Schofield created KAFKA-15840: Summary: Correct initialization of ConsumerGroupHeartbeat by client Key: KAFKA-15840 URL: https://issues.apache.org/jira/browse/KAFKA-15840 Project: Kafka Issue Type: Sub-task Components: clients Reporter: Andrew Schofield Assignee: Andrew Schofield The new consumer using the KIP-848 protocol currently leaves the TopicPartitions set to null for the ConsumerGroupHeartbeat request, even when the MemberEpoch is zero. This violates the KIP which expects the list to be empty (but not null). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15842) Correct handling of KafkaConsumer.committed for new consumer
Andrew Schofield created KAFKA-15842: Summary: Correct handling of KafkaConsumer.committed for new consumer Key: KAFKA-15842 URL: https://issues.apache.org/jira/browse/KAFKA-15842 Project: Kafka Issue Type: Sub-task Components: clients Reporter: Andrew Schofield Assignee: Andrew Schofield KafkaConsumer.committed throws TimeOutException when there is no response. The new consumer currently returns a null. Changing the new consumer to behave like the old consumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15837) Throw error on use of Consumer.poll(long timeout)
[ https://issues.apache.org/jira/browse/KAFKA-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15837. -- Resolution: Fixed > Throw error on use of Consumer.poll(long timeout) > - > > Key: KAFKA-15837 > URL: https://issues.apache.org/jira/browse/KAFKA-15837 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer >Reporter: Kirk True >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.7.0 > > > Per [KIP-266|https://cwiki.apache.org/confluence/x/5kiHB], the > Consumer.poll(long timeout) method was deprecated back in 2.0.0. The method > will now throw a KafkaException. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15871) Implement kafka-client-metrics.sh tool
Andrew Schofield created KAFKA-15871: Summary: Implement kafka-client-metrics.sh tool Key: KAFKA-15871 URL: https://issues.apache.org/jira/browse/KAFKA-15871 Project: Kafka Issue Type: Sub-task Components: admin Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.7.0 Implement the `kafka-client-metrics.sh` tool which is introduced in KIP-714 and enhanced in KIP-1000. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15890) kafka-console-consumer.sh problems with new consumer
Andrew Schofield created KAFKA-15890: Summary: kafka-console-consumer.sh problems with new consumer Key: KAFKA-15890 URL: https://issues.apache.org/jira/browse/KAFKA-15890 Project: Kafka Issue Type: Bug Components: clients, consumer Reporter: Andrew Schofield Assignee: Andrew Schofield Various problems found testing `kafka-console-consumer.sh` with the new consumer, including NPEs, never-ending reconcilation states and failure to fetch records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15840) Correct initialization of ConsumerGroupHeartbeat by client
[ https://issues.apache.org/jira/browse/KAFKA-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15840. -- Resolution: Duplicate > Correct initialization of ConsumerGroupHeartbeat by client > -- > > Key: KAFKA-15840 > URL: https://issues.apache.org/jira/browse/KAFKA-15840 > Project: Kafka > Issue Type: Sub-task > Components: clients, consumer >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Labels: kip-848-client-support, kip-848-e2e, kip-848-preview > > The new consumer using the KIP-848 protocol currently leaves the > TopicPartitions set to null for the ConsumerGroupHeartbeat request, even when > the MemberEpoch is zero. This violates the KIP which expects the list to be > empty (but not null). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
Andrew Schofield created KAFKA-15932:
Summary: Flaky test -
PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
Key: KAFKA-15932
URL: https://issues.apache.org/jira/browse/KAFKA-15932
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Intermittently failing test for the new consumer.
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14859/1/tests/
```Error
org.opentest4j.AssertionFailedError: Timed out before consuming expected 1
records. The number consumed was 0.
Stacktrace
org.opentest4j.AssertionFailedError: Timed out before consuming expected 1
records. The number consumed was 0.
at
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
at
app//kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:161)
at
app//kafka.api.AbstractConsumerTest.consumeAndVerifyRecords(AbstractConsumerTest.scala:128)
at
app//kafka.api.PlaintextConsumerTest.testSeek(PlaintextConsumerTest.scala:616)
at
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
at
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
at
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
at
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
at
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
at
app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at
app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at
app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at
app//org.junit.platform.engine.suppor
[jira] [Created] (KAFKA-15943) Flaky test - testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest
Andrew Schofield created KAFKA-15943: Summary: Flaky test - testHighWaterMarkAfterPartitionReassignment(String).quorum=kraft – org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest Key: KAFKA-15943 URL: https://issues.apache.org/jira/browse/KAFKA-15943 Project: Kafka Issue Type: Bug Components: tools Affects Versions: 3.7.0 Reporter: Andrew Schofield Error org.opentest4j.AssertionFailedError: Timed out waiting for verifyAssignment result org.apache.kafka.tools.reassign.VerifyAssignmentResult@9c4dd302. The latest result was org.apache.kafka.tools.reassign.VerifyAssignmentResult@cc845dc Stacktrace org.opentest4j.AssertionFailedError: Timed out waiting for verifyAssignment result org.apache.kafka.tools.reassign.VerifyAssignmentResult@9c4dd302. The latest result was org.apache.kafka.tools.reassign.VerifyAssignmentResult@cc845dc at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134) at app//kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:1141) at app//kafka.utils.TestUtils.waitUntilTrue(TestUtils.scala) at app//org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest.waitForVerifyAssignment(ReassignPartitionsIntegrationTest.java:685) at app//org.apache.kafka.tools.reassign.ReassignPartitionsIntegrationTest.testHighWaterMarkAfterPartitionReassignment(ReassignPartitionsIntegrationTest.java:216) at java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566) at app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) at app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) at app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at app//org.junit.platform.e
[jira] [Created] (KAFKA-15944) Flaky test - verifyStore[cache=false, log=true, supplier=ROCKS_KV, kind=DSL] – org.apache.kafka.streams.integration.PositionRestartIntegrationTest
Andrew Schofield created KAFKA-15944:
Summary: Flaky test - verifyStore[cache=false, log=true,
supplier=ROCKS_KV, kind=DSL] –
org.apache.kafka.streams.integration.PositionRestartIntegrationTest
Key: KAFKA-15944
URL: https://issues.apache.org/jira/browse/KAFKA-15944
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Error
org.apache.kafka.common.errors.TimeoutException: The query never returned
within the bound. Last result:
StateQueryResult{partitionResults={0=SucceededQueryResult{result=org.apache.kafka.streams.state.internals.MeteredKeyValueStore$MeteredKeyValueTimestampedIterator@61b360a4,
executionInfo=[], position=Position{position={input-topic={0=1,
1=FailedQueryResult{failureReason=NOT_PRESENT, failure='The requested partition
was not present at the time of the query.', executionInfo=[], position=null}},
globalResult=null}
Stacktrace
org.apache.kafka.common.errors.TimeoutException: The query never returned
within the bound. Last result:
StateQueryResult{partitionResults={0=SucceededQueryResult{result=org.apache.kafka.streams.state.internals.MeteredKeyValueStore$MeteredKeyValueTimestampedIterator@61b360a4,
executionInfo=[], position=Position{position={input-topic={0=1,
1=FailedQueryResult{failureReason=NOT_PRESENT, failure='The requested partition
was not present at the time of the query.', executionInfo=[], position=null}},
globalResult=null}
Standard Output
[2023-11-28 22:52:47,353] INFO [Producer clientId=producer-129] Instantiated an
idempotent producer. (org.apache.kafka.clients.producer.KafkaProducer:587)
[2023-11-28 22:52:47,466] INFO [Producer clientId=producer-129] ProducerId set
to 0 with epoch 0
(org.apache.kafka.clients.producer.internals.TransactionManager:502)
[2023-11-28 22:52:47,473] INFO [Producer clientId=producer-129] Closing the
Kafka producer with timeoutMillis = 9223372036854775807 ms.
(org.apache.kafka.clients.producer.KafkaProducer:1332)
[2023-11-28 22:52:47,531] INFO stream-client
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
Kafka Streams version: test-version (org.apache.kafka.streams.KafkaStreams:914)
[2023-11-28 22:52:47,531] INFO stream-client
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
Kafka Streams commit ID: test-commit-ID
(org.apache.kafka.streams.KafkaStreams:915)
[2023-11-28 22:52:47,532] INFO stream-thread
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
Creating restore consumer client
(org.apache.kafka.streams.processor.internals.StreamThread:365)
[2023-11-28 22:52:47,537] INFO stream-thread
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
Creating thread producer client
(org.apache.kafka.streams.processor.internals.StreamThread:105)
[2023-11-28 22:52:47,538] INFO [Producer
clientId=app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-producer]
Instantiated an idempotent producer.
(org.apache.kafka.clients.producer.KafkaProducer:587)
[2023-11-28 22:52:47,545] INFO stream-thread
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1]
Creating consumer client
(org.apache.kafka.streams.processor.internals.StreamThread:432)
[2023-11-28 22:52:47,545] INFO state-updater
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StateUpdater-1]
State updater thread started
(org.apache.kafka.streams.processor.internals.DefaultStateUpdater:135)
[2023-11-28 22:52:47,547] INFO [Producer
clientId=app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-producer]
ProducerId set to 1 with epoch 0
(org.apache.kafka.clients.producer.internals.TransactionManager:502)
[2023-11-28 22:52:47,550] INFO stream-thread
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57-StreamThread-1-consumer]
Cooperative rebalancing protocol is enabled now
(org.apache.kafka.streams.processor.internals.assignment.AssignorConfiguration:141)
[2023-11-28 22:52:47,552] INFO stream-client
[app-org.apache.kafka.streams.integration.PositionRestartIntegrationTest-true-true-IN_MEMORY_KV-DSL-456895e1-b230-4a83-9164-8e6d65d1fc57]
State transi
[jira] [Created] (KAFKA-16676) Security docs missing RPCs from KIP-714 and KIP-1000
Andrew Schofield created KAFKA-16676: Summary: Security docs missing RPCs from KIP-714 and KIP-1000 Key: KAFKA-16676 URL: https://issues.apache.org/jira/browse/KAFKA-16676 Project: Kafka Issue Type: Improvement Components: docs Affects Versions: 3.7.0, 3.8.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.8.0 KIPs 714 and 1000 introduced 3 new RPCs to do with client metrics. None of them was added to the list of RPCs in the security documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16676) Security docs missing RPCs from KIP-714 and KIP-1000
[ https://issues.apache.org/jira/browse/KAFKA-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16676. -- Resolution: Fixed > Security docs missing RPCs from KIP-714 and KIP-1000 > > > Key: KAFKA-16676 > URL: https://issues.apache.org/jira/browse/KAFKA-16676 > Project: Kafka > Issue Type: Improvement > Components: docs >Affects Versions: 3.7.0, 3.8.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Minor > Fix For: 3.8.0 > > > KIPs 714 and 1000 introduced 3 new RPCs to do with client metrics. None of > them was added to the list of RPCs in the security documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16713) Add new RPC definitions
Andrew Schofield created KAFKA-16713: Summary: Add new RPC definitions Key: KAFKA-16713 URL: https://issues.apache.org/jira/browse/KAFKA-16713 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16714) kafka-share-groups.sh supporting list and describe
Andrew Schofield created KAFKA-16714: Summary: kafka-share-groups.sh supporting list and describe Key: KAFKA-16714 URL: https://issues.apache.org/jira/browse/KAFKA-16714 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16715) Create KafkaShareConsumer interface
Andrew Schofield created KAFKA-16715: Summary: Create KafkaShareConsumer interface Key: KAFKA-16715 URL: https://issues.apache.org/jira/browse/KAFKA-16715 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16716) Add AdminClient.describeShareGroups and AdminClient.listShareGroups
Andrew Schofield created KAFKA-16716: Summary: Add AdminClient.describeShareGroups and AdminClient.listShareGroups Key: KAFKA-16716 URL: https://issues.apache.org/jira/browse/KAFKA-16716 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16720) Add AdminClient.listShareGroupOffsets
Andrew Schofield created KAFKA-16720: Summary: Add AdminClient.listShareGroupOffsets Key: KAFKA-16720 URL: https://issues.apache.org/jira/browse/KAFKA-16720 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16717) Add AdminClient.alterShareGroupOffsets
Andrew Schofield created KAFKA-16717: Summary: Add AdminClient.alterShareGroupOffsets Key: KAFKA-16717 URL: https://issues.apache.org/jira/browse/KAFKA-16717 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16719) Add AdminClient.deleteShareGroups
Andrew Schofield created KAFKA-16719: Summary: Add AdminClient.deleteShareGroups Key: KAFKA-16719 URL: https://issues.apache.org/jira/browse/KAFKA-16719 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16718) Add AdminClient.deleteShareGroupOffsets
Andrew Schofield created KAFKA-16718: Summary: Add AdminClient.deleteShareGroupOffsets Key: KAFKA-16718 URL: https://issues.apache.org/jira/browse/KAFKA-16718 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16721) Add exceptions for the new error codes
Andrew Schofield created KAFKA-16721: Summary: Add exceptions for the new error codes Key: KAFKA-16721 URL: https://issues.apache.org/jira/browse/KAFKA-16721 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16722) Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor
Andrew Schofield created KAFKA-16722: Summary: Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor Key: KAFKA-16722 URL: https://issues.apache.org/jira/browse/KAFKA-16722 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16723) Add kafka-console-share-consumer.sh
Andrew Schofield created KAFKA-16723: Summary: Add kafka-console-share-consumer.sh Key: KAFKA-16723 URL: https://issues.apache.org/jira/browse/KAFKA-16723 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
Andrew Schofield created KAFKA-16724: Summary: Add new options for kafka-producer-perf-test.sh Key: KAFKA-16724 URL: https://issues.apache.org/jira/browse/KAFKA-16724 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16727) Add dynamic group configuration for record lock duration
Andrew Schofield created KAFKA-16727: Summary: Add dynamic group configuration for record lock duration Key: KAFKA-16727 URL: https://issues.apache.org/jira/browse/KAFKA-16727 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16725) Add broker configurations
Andrew Schofield created KAFKA-16725: Summary: Add broker configurations Key: KAFKA-16725 URL: https://issues.apache.org/jira/browse/KAFKA-16725 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16726) Add dynamic group configuration for offset reset
Andrew Schofield created KAFKA-16726: Summary: Add dynamic group configuration for offset reset Key: KAFKA-16726 URL: https://issues.apache.org/jira/browse/KAFKA-16726 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16729) Support for read-committed isolation level
Andrew Schofield created KAFKA-16729: Summary: Support for read-committed isolation level Key: KAFKA-16729 URL: https://issues.apache.org/jira/browse/KAFKA-16729 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16728) Add dynamic group configuration for heartbeat interval and session timeout
Andrew Schofield created KAFKA-16728: Summary: Add dynamic group configuration for heartbeat interval and session timeout Key: KAFKA-16728 URL: https://issues.apache.org/jira/browse/KAFKA-16728 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16731) Support for share-group-metrics in the broker
Andrew Schofield created KAFKA-16731: Summary: Support for share-group-metrics in the broker Key: KAFKA-16731 URL: https://issues.apache.org/jira/browse/KAFKA-16731 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16732) Support for share-coordinator-metrics in the broker
Andrew Schofield created KAFKA-16732: Summary: Support for share-coordinator-metrics in the broker Key: KAFKA-16732 URL: https://issues.apache.org/jira/browse/KAFKA-16732 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16733) Add support for formatting new records written to offsets topic
Andrew Schofield created KAFKA-16733: Summary: Add support for formatting new records written to offsets topic Key: KAFKA-16733 URL: https://issues.apache.org/jira/browse/KAFKA-16733 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16730) Initial code for share-group consumer
Andrew Schofield created KAFKA-16730: Summary: Initial code for share-group consumer Key: KAFKA-16730 URL: https://issues.apache.org/jira/browse/KAFKA-16730 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16734) Add support for formatting records written to share-group state topic
Andrew Schofield created KAFKA-16734: Summary: Add support for formatting records written to share-group state topic Key: KAFKA-16734 URL: https://issues.apache.org/jira/browse/KAFKA-16734 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16759) Invalid client telemetry transition on consumer close
Andrew Schofield created KAFKA-16759:
Summary: Invalid client telemetry transition on consumer close
Key: KAFKA-16759
URL: https://issues.apache.org/jira/browse/KAFKA-16759
Project: Kafka
Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
Fix For: 3.8.0
Using the console consumer with client telemetry enabled, I hit an invalid
state transition when closing the consumer with CTRL-C. The consumer sends a
final "terminating" telemetry push which puts the client telemetry reporter
into TERMINATING_PUSH_IN_PROGRESS state. When it receives a response in this
state, it attempts an invalid state transition.
{noformat}
[2024-05-13 19:19:35,804] WARN Error updating client telemetry state, disabled
telemetry (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
java.lang.IllegalStateException: Invalid telemetry state transition from
TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
at
org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:827)
at
org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
at
org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
at
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
at
org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.sendUnsentRequests(ConsumerNetworkThread.java:262)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:275)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:95)
[2024-05-13 19:19:35,805] WARN Unable to transition state after successful push
telemetry from state TERMINATING_PUSH_IN_PROGRESS
(org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16759) Invalid client telemetry transition on consumer close
[
https://issues.apache.org/jira/browse/KAFKA-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-16759.
--
Resolution: Fixed
> Invalid client telemetry transition on consumer close
> -
>
> Key: KAFKA-16759
> URL: https://issues.apache.org/jira/browse/KAFKA-16759
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Minor
> Fix For: 3.8.0
>
>
> Using the console consumer with client telemetry enabled, I hit an invalid
> state transition when closing the consumer with CTRL-C. The consumer sends a
> final "terminating" telemetry push which puts the client telemetry reporter
> into TERMINATING_PUSH_IN_PROGRESS state. When it receives a response in this
> state, it attempts an invalid state transition.
>
> {noformat}
> [2024-05-13 19:19:35,804] WARN Error updating client telemetry state,
> disabled telemetry
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter)
> java.lang.IllegalStateException: Invalid telemetry state transition from
> TERMINATING_PUSH_IN_PROGRESS to PUSH_NEEDED; the valid telemetry state
> transitions from TERMINATING_PUSH_IN_PROGRESS are: TERMINATED
> at
> org.apache.kafka.common.telemetry.ClientTelemetryState.validateTransition(ClientTelemetryState.java:163)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.maybeSetState(ClientTelemetryReporter.java:827)
> at
> org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter$DefaultClientTelemetrySender.handleResponse(ClientTelemetryReporter.java:520)
> at
> org.apache.kafka.clients.NetworkClient$TelemetrySender.handleResponse(NetworkClient.java:1321)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:948)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:594)
> at
> org.apache.kafka.clients.consumer.internals.NetworkClientDelegate.poll(NetworkClientDelegate.java:130)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.sendUnsentRequests(ConsumerNetworkThread.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.cleanup(ConsumerNetworkThread.java:275)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkThread.run(ConsumerNetworkThread.java:95)
> [2024-05-13 19:19:35,805] WARN Unable to transition state after successful
> push telemetry from state TERMINATING_PUSH_IN_PROGRESS
> (org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16713) Add new RPC definitions
[ https://issues.apache.org/jira/browse/KAFKA-16713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16713. -- Resolution: Fixed ShareGroupHeartbeat, ShareGroupDescribe, ShareFetch and ShareAcknowledge RPCs have been delivered. > Add new RPC definitions > --- > > Key: KAFKA-16713 > URL: https://issues.apache.org/jira/browse/KAFKA-16713 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16721) Add exceptions for the new error codes
[ https://issues.apache.org/jira/browse/KAFKA-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16721. -- Resolution: Fixed Delivered as part of https://github.com/apache/kafka/pull/16022. > Add exceptions for the new error codes > -- > > Key: KAFKA-16721 > URL: https://issues.apache.org/jira/browse/KAFKA-16721 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16715) Create KafkaShareConsumer interface
[ https://issues.apache.org/jira/browse/KAFKA-16715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16715. -- Resolution: Fixed > Create KafkaShareConsumer interface > --- > > Key: KAFKA-16715 > URL: https://issues.apache.org/jira/browse/KAFKA-16715 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16891) KIP-1043: Administration of groups
Andrew Schofield created KAFKA-16891: Summary: KIP-1043: Administration of groups Key: KAFKA-16891 URL: https://issues.apache.org/jira/browse/KAFKA-16891 Project: Kafka Issue Type: New Feature Reporter: Andrew Schofield This issue tracks the development of KIP-1043: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16740) Define skeleton for SharePartitionManager and SharePartition
[ https://issues.apache.org/jira/browse/KAFKA-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16740. -- Resolution: Fixed > Define skeleton for SharePartitionManager and SharePartition > > > Key: KAFKA-16740 > URL: https://issues.apache.org/jira/browse/KAFKA-16740 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > > Add high level design for broker side implementation for fetching and > acknowledging messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16894) Define group.version=2
Andrew Schofield created KAFKA-16894: Summary: Define group.version=2 Key: KAFKA-16894 URL: https://issues.apache.org/jira/browse/KAFKA-16894 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Create group.version=2 as the switch to enable share groups in the broker. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh
[ https://issues.apache.org/jira/browse/KAFKA-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16724. -- Resolution: Fixed > Add new options for kafka-producer-perf-test.sh > --- > > Key: KAFKA-16724 > URL: https://issues.apache.org/jira/browse/KAFKA-16724 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Shivsundar R >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16950) Define Persister and Share Coordinator RPCs
[
https://issues.apache.org/jira/browse/KAFKA-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-16950.
--
Resolution: Fixed
> Define Persister and Share Coordinator RPCs
> ---
>
> Key: KAFKA-16950
> URL: https://issues.apache.org/jira/browse/KAFKA-16950
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Andrew Schofield
>Priority: Major
>
> Add Persister interface with schemas for RPCs. The classes which are needed
> by SharePartition to integrate are below, note some of them results from the
> generated json schema classes.
>
>
> {code:java}
> import org.apache.kafka.server.group.share.GroupTopicPartitionData;
> import org.apache.kafka.server.group.share.PartitionAllData;
> import org.apache.kafka.server.group.share.PartitionErrorData;
> import org.apache.kafka.server.group.share.PartitionFactory;
> import org.apache.kafka.server.group.share.PartitionIdLeaderEpochData;
> import org.apache.kafka.server.group.share.PartitionStateBatchData;
> import org.apache.kafka.server.group.share.Persister;
> import org.apache.kafka.server.group.share.PersisterStateBatch;
> import org.apache.kafka.server.group.share.ReadShareGroupStateParameters;
> import org.apache.kafka.server.group.share.ReadShareGroupStateResult;
> import org.apache.kafka.server.group.share.TopicData;
> import org.apache.kafka.server.group.share.WriteShareGroupStateParameters;
> import org.apache.kafka.server.group.share.WriteShareGroupStateResult; {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-16725) Add broker configurations
[ https://issues.apache.org/jira/browse/KAFKA-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16725. -- Resolution: Fixed > Add broker configurations > - > > Key: KAFKA-16725 > URL: https://issues.apache.org/jira/browse/KAFKA-16725 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17028) FindCoordinator v6 initial implementation
Andrew Schofield created KAFKA-17028: Summary: FindCoordinator v6 initial implementation Key: KAFKA-17028 URL: https://issues.apache.org/jira/browse/KAFKA-17028 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17028) FindCoordinator v6 initial implementation
[ https://issues.apache.org/jira/browse/KAFKA-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17028. -- Fix Version/s: 3.9.0 Resolution: Fixed > FindCoordinator v6 initial implementation > - > > Key: KAFKA-17028 > URL: https://issues.apache.org/jira/browse/KAFKA-17028 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.9.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16731) Support for share-group-metrics in the broker
[ https://issues.apache.org/jira/browse/KAFKA-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16731. -- Resolution: Fixed > Support for share-group-metrics in the broker > - > > Key: KAFKA-16731 > URL: https://issues.apache.org/jira/browse/KAFKA-16731 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Sushant Mahajan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16741) Add ShareGroupHeartbeat API support in GroupCoordinator
[ https://issues.apache.org/jira/browse/KAFKA-16741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16741. -- Resolution: Fixed > Add ShareGroupHeartbeat API support in GroupCoordinator > --- > > Key: KAFKA-16741 > URL: https://issues.apache.org/jira/browse/KAFKA-16741 > Project: Kafka > Issue Type: Sub-task >Reporter: Apoorv Mittal >Assignee: Apoorv Mittal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17093) KafkaConsumer.seekToEnd should return LSO
[ https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17093. -- Resolution: Not A Problem > KafkaConsumer.seekToEnd should return LSO > -- > > Key: KAFKA-17093 > URL: https://issues.apache.org/jira/browse/KAFKA-17093 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Affects Versions: 3.6.1 > Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" % > "kafka-clients" % "3.6.1" >Reporter: Tom Kalmijn >Assignee: Andrew Schofield >Priority: Major > Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java > > > > Expected > When using a transactional producer then the method > KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level > "read_committed" should return the LSO. > Observed > The offset returned is always the actual last offset of the partition, which > is not the LSO if the latest offsets are occupied by transaction markers. > Also see this Slack thread: > https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16730) Initial code for share-group consumer
[ https://issues.apache.org/jira/browse/KAFKA-16730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16730. -- Resolution: Fixed > Initial code for share-group consumer > - > > Key: KAFKA-16730 > URL: https://issues.apache.org/jira/browse/KAFKA-16730 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17144) Retract v6 of ListGroups RPC added in error
Andrew Schofield created KAFKA-17144: Summary: Retract v6 of ListGroups RPC added in error Key: KAFKA-17144 URL: https://issues.apache.org/jira/browse/KAFKA-17144 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield As agreed in the latter stages of the review for KIP-932, the AdminClient.listGroups() and ListGroups v6 RPC were to be removed. However, I forgot to remove the RPC definition from the KIP and then merged the v6 RPC in AK. We need to retract this unused RPC version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17225) Refactor consumer membership managers
Andrew Schofield created KAFKA-17225: Summary: Refactor consumer membership managers Key: KAFKA-17225 URL: https://issues.apache.org/jira/browse/KAFKA-17225 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 The initial drop of ShareMembershipManager contained a lot of code duplicated from MembershipManagerImpl. The plan was always to share as much code as possible between the membership managers for consumer groups and share groups. This issue refactors the membership managers so that almost all of the code is in common. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17231) Sn
Andrew Schofield created KAFKA-17231: Summary: Sn Key: KAFKA-17231 URL: https://issues.apache.org/jira/browse/KAFKA-17231 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17247) Revised share group record schemas
Andrew Schofield created KAFKA-17247: Summary: Revised share group record schemas Key: KAFKA-17247 URL: https://issues.apache.org/jira/browse/KAFKA-17247 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 In KIP-932, the group coordinator does not persist assignments for share groups. While this sounds like a good idea in terms of minimising overhead for data which doesn't strictly need to be recoverable, it significantly adds to the complexity of working with the coordinator framework. This issue revised the definitions of the share group record schemas following more closely the schemas used for consumer groups, and eliminating the need to maintain soft state alongside the group coordinator's timeline structure. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17262) kafka-topics.sh usage message is confusing
Andrew Schofield created KAFKA-17262: Summary: kafka-topics.sh usage message is confusing Key: KAFKA-17262 URL: https://issues.apache.org/jira/browse/KAFKA-17262 Project: Kafka Issue Type: Bug Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 There is a lot of historical cruft in the usage message for kafka-topics.sh. For example, the --bootstrap-server option is required nowadays, but the usage message is written in some cases as if it was still optional. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17287) Add integration test for KafkaShareConsumer
Andrew Schofield created KAFKA-17287: Summary: Add integration test for KafkaShareConsumer Key: KAFKA-17287 URL: https://issues.apache.org/jira/browse/KAFKA-17287 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Shivsundar R Add an integration test suite for testing KafkaShareConsumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17289) Add integration test for ShareGroupDescribe requests
Andrew Schofield created KAFKA-17289: Summary: Add integration test for ShareGroupDescribe requests Key: KAFKA-17289 URL: https://issues.apache.org/jira/browse/KAFKA-17289 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17291) Add integration test for share group list and describe admin calls
Andrew Schofield created KAFKA-17291: Summary: Add integration test for share group list and describe admin calls Key: KAFKA-17291 URL: https://issues.apache.org/jira/browse/KAFKA-17291 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17290) Add integration test for ShareGroupFetch/Acknowledge requests
Andrew Schofield created KAFKA-17290: Summary: Add integration test for ShareGroupFetch/Acknowledge requests Key: KAFKA-17290 URL: https://issues.apache.org/jira/browse/KAFKA-17290 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17292) Add SHARE to group.coordinator.rebalance.protocols
Andrew Schofield created KAFKA-17292: Summary: Add SHARE to group.coordinator.rebalance.protocols Key: KAFKA-17292 URL: https://issues.apache.org/jira/browse/KAFKA-17292 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16716) Add AdminClient.describeShareGroups and AdminClient.listShareGroups
[ https://issues.apache.org/jira/browse/KAFKA-16716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16716. -- Resolution: Fixed > Add AdminClient.describeShareGroups and AdminClient.listShareGroups > --- > > Key: KAFKA-16716 > URL: https://issues.apache.org/jira/browse/KAFKA-16716 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17217) Clients : Optimise batching of requests per node in ShareConsumeRequestManager
[ https://issues.apache.org/jira/browse/KAFKA-17217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17217. -- Resolution: Fixed > Clients : Optimise batching of requests per node in ShareConsumeRequestManager > -- > > Key: KAFKA-17217 > URL: https://issues.apache.org/jira/browse/KAFKA-17217 > Project: Kafka > Issue Type: Sub-task >Reporter: Shivsundar R >Assignee: Shivsundar R >Priority: Major > > In ShareConsumeRequestManager, currently every time we perform a commitSync > or commitAsync, we create one ShareAcknowledge RPC for the same. Here, we can > optimise the number of RPC calls by batching the acknowledgements before the > next poll is invoked per node. > This will ensure that between 2 calls, the acknowledgements are accumulated > in one request per node and then sent during poll, resulting in lesser RPC > calls. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17289) Add integration test for ShareGroupDescribe requests
[ https://issues.apache.org/jira/browse/KAFKA-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17289. -- Fix Version/s: 4.0.0 Resolution: Fixed > Add integration test for ShareGroupDescribe requests > > > Key: KAFKA-17289 > URL: https://issues.apache.org/jira/browse/KAFKA-17289 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17318) Introduce ConsumerRecord.deliveryCount()
Andrew Schofield created KAFKA-17318: Summary: Introduce ConsumerRecord.deliveryCount() Key: KAFKA-17318 URL: https://issues.apache.org/jira/browse/KAFKA-17318 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17292) Add SHARE to group.coordinator.rebalance.protocols
[ https://issues.apache.org/jira/browse/KAFKA-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17292. -- Resolution: Fixed > Add SHARE to group.coordinator.rebalance.protocols > -- > > Key: KAFKA-17292 > URL: https://issues.apache.org/jira/browse/KAFKA-17292 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17225) Refactor consumer membership managers
[ https://issues.apache.org/jira/browse/KAFKA-17225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17225. -- Resolution: Fixed > Refactor consumer membership managers > - > > Key: KAFKA-17225 > URL: https://issues.apache.org/jira/browse/KAFKA-17225 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > The initial drop of ShareMembershipManager contained a lot of code duplicated > from MembershipManagerImpl. The plan was always to share as much code as > possible between the membership managers for consumer groups and share > groups. This issue refactors the membership managers so that almost all of > the code is in common. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16714) kafka-share-groups.sh supporting list and describe
[ https://issues.apache.org/jira/browse/KAFKA-16714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16714. -- Resolution: Fixed > kafka-share-groups.sh supporting list and describe > -- > > Key: KAFKA-16714 > URL: https://issues.apache.org/jira/browse/KAFKA-16714 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17341) Refactor consumer heartbeat managers
Andrew Schofield created KAFKA-17341: Summary: Refactor consumer heartbeat managers Key: KAFKA-17341 URL: https://issues.apache.org/jira/browse/KAFKA-17341 Project: Kafka Issue Type: Improvement Components: clients Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 HeartbeatRequestManager and ShareHeartbeatRequestManager are very closely related and there's a lot of code duplication. They can be refactored to share the majority of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17347) Add omitted --client-metrics option to kafka-configs.sh
Andrew Schofield created KAFKA-17347: Summary: Add omitted --client-metrics option to kafka-configs.sh Key: KAFKA-17347 URL: https://issues.apache.org/jira/browse/KAFKA-17347 Project: Kafka Issue Type: Bug Components: tools Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 KIP-714 introduced client metrics resources to kafka-configs.sh. The option `--entity-type client-metrics` was added, and a shorthand of `--client-metrics` was also included in the comments. However, the `--client-metrics` option whose syntax matches all of the other entity types was omitted. This corrects that omission. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17350) Improve kafka-share-groups.sh --describe output for empty groups
Andrew Schofield created KAFKA-17350: Summary: Improve kafka-share-groups.sh --describe output for empty groups Key: KAFKA-17350 URL: https://issues.apache.org/jira/browse/KAFKA-17350 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 When you use `kafka-share-groups.sh --describe` for an empty group, it prints an empty table without any indication that this is expected. `kafka-consumer-groups.sh` summarises the group status to make the output more informative. This is a simple improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17368) Add delivery count to kafka-console-share-consumer.sh
Andrew Schofield created KAFKA-17368: Summary: Add delivery count to kafka-console-share-consumer.sh Key: KAFKA-17368 URL: https://issues.apache.org/jira/browse/KAFKA-17368 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Andrew Schofield Assignee: Andrew Schofield Now that ConsumerRecord.deliveryCount() exists, enhance kafka-console-share-consumer.sh to exploit it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-17350) Improve kafka-share-groups.sh --describe output for empty groups
[ https://issues.apache.org/jira/browse/KAFKA-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-17350. -- Resolution: Fixed > Improve kafka-share-groups.sh --describe output for empty groups > > > Key: KAFKA-17350 > URL: https://issues.apache.org/jira/browse/KAFKA-17350 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 4.0.0 > > > When you use `kafka-share-groups.sh --describe` for an empty group, it prints > an empty table without any indication that this is expected. > `kafka-consumer-groups.sh` summarises the group status to make the output > more informative. This is a simple improvement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-17378) Initial performance testing fixes
Andrew Schofield created KAFKA-17378: Summary: Initial performance testing fixes Key: KAFKA-17378 URL: https://issues.apache.org/jira/browse/KAFKA-17378 Project: Kafka Issue Type: Sub-task Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 4.0.0 Collect together some fixes for problems identified during initial performance testing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15945) Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest
Andrew Schofield created KAFKA-15945: Summary: Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest Key: KAFKA-15945 URL: https://issues.apache.org/jira/browse/KAFKA-15945 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 3.7.0 Reporter: Andrew Schofield Last seen: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14811/7/tests Error org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, because it's explicitly defined on the target topic! ==> expected: <2000> but was: <8640> Stacktrace org.opentest4j.AssertionFailedError: `delete.retention.ms` should be 2000, because it's explicitly defined on the target topic! ==> expected: <2000> but was: <8640> at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151) at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132) at app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197) at app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182) at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1152) at app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.lambda$testSyncTopicConfigs$8(MirrorConnectorsIntegrationBaseTest.java:780) at app//org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:331) at app//org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:379) at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:328) at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:312) at app//org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:302) at app//org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest.testSyncTopicConfigs(MirrorConnectorsIntegrationBaseTest.java:774) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) at app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) at app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.ex
[jira] [Resolved] (KAFKA-15842) Correct handling of KafkaConsumer.committed for new consumer
[ https://issues.apache.org/jira/browse/KAFKA-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15842. -- Resolution: Fixed > Correct handling of KafkaConsumer.committed for new consumer > > > Key: KAFKA-15842 > URL: https://issues.apache.org/jira/browse/KAFKA-15842 > Project: Kafka > Issue Type: Sub-task > Components: clients, consumer >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Minor > Labels: kip-848-client-support, kip-848-e2e, kip-848-preview > > KafkaConsumer.committed throws TimeOutException when there is no response. > The new consumer currently returns a null. Changing the new consumer to > behave like the old consumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15890) Consumer.poll with long timeout unaware of assigned partitions
[ https://issues.apache.org/jira/browse/KAFKA-15890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15890. -- Resolution: Fixed > Consumer.poll with long timeout unaware of assigned partitions > -- > > Key: KAFKA-15890 > URL: https://issues.apache.org/jira/browse/KAFKA-15890 > Project: Kafka > Issue Type: Bug > Components: clients, consumer >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Labels: consumer-threading-refactor, kip-848-client-support, > kip-848-preview > > Various problems found testing `kafka-console-consumer.sh` with the new > consumer, including NPEs, never-ending reconcilation states and failure to > fetch records. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15953) Refactor polling delays
Andrew Schofield created KAFKA-15953: Summary: Refactor polling delays Key: KAFKA-15953 URL: https://issues.apache.org/jira/browse/KAFKA-15953 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.7.0 This is a follow-on tasks for https://issues.apache.org/jira/browse/KAFKA-15890. The idea is to reduce the interaction between the application thread and the request managers which was introduced in that earlier JIRA's patch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15966) Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy
Andrew Schofield created KAFKA-15966: Summary: Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy Key: KAFKA-15966 URL: https://issues.apache.org/jira/browse/KAFKA-15966 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 3.7.0 Reporter: Andrew Schofield Gradle Test Run :streams:test > Gradle Test Executor 3 > ConsistencyVectorIntegrationTest > shouldHaveSamePositionBoundActiveAndStandBy FAILED java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.kafka.streams.state.ValueAndTimestamp at org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.checkPosition(ConsistencyVectorIntegrationTest.java:169) at org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy(ConsistencyVectorIntegrationTest.java:140) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15953) Refactor polling delays
[ https://issues.apache.org/jira/browse/KAFKA-15953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15953. -- Resolution: Fixed > Refactor polling delays > --- > > Key: KAFKA-15953 > URL: https://issues.apache.org/jira/browse/KAFKA-15953 > Project: Kafka > Issue Type: Sub-task > Components: clients >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Fix For: 3.7.0 > > > This is a follow-on tasks for > https://issues.apache.org/jira/browse/KAFKA-15890. The idea is to reduce the > interaction between the application thread and the request managers which was > introduced in that earlier JIRA's patch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15966) Test failure - ClassCastException in ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy
[ https://issues.apache.org/jira/browse/KAFKA-15966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15966. -- Resolution: Duplicate Just fixed by https://github.com/apache/kafka/pull/14895 > Test failure - ClassCastException in > ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy > - > > Key: KAFKA-15966 > URL: https://issues.apache.org/jira/browse/KAFKA-15966 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Priority: Major > Labels: test-failure > > Gradle Test Run :streams:test > Gradle Test Executor 3 > > ConsistencyVectorIntegrationTest > > shouldHaveSamePositionBoundActiveAndStandBy FAILED > java.lang.ClassCastException: java.lang.Integer cannot be cast to > org.apache.kafka.streams.state.ValueAndTimestamp > at > org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.checkPosition(ConsistencyVectorIntegrationTest.java:169) > at > org.apache.kafka.streams.integration.ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy(ConsistencyVectorIntegrationTest.java:140) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15971) Re-enable consumer integration tests for new consumer
Andrew Schofield created KAFKA-15971: Summary: Re-enable consumer integration tests for new consumer Key: KAFKA-15971 URL: https://issues.apache.org/jira/browse/KAFKA-15971 Project: Kafka Issue Type: Improvement Components: clients Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.7.0 Re-enable the consumer integration tests for the new consumer making sure that build stability is not impacted. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15533) Ensure HeartbeatRequestManager only send out some fields once
[
https://issues.apache.org/jira/browse/KAFKA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-15533.
--
Resolution: Resolved
Already resolved
> Ensure HeartbeatRequestManager only send out some fields once
> -
>
> Key: KAFKA-15533
> URL: https://issues.apache.org/jira/browse/KAFKA-15533
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Minor
> Labels: kip-848, kip-848-client-support
>
> We want to ensure ConsumerGroupHeartbeatRequest is as lightweight as
> possible, so a lot of fields in it don't need to be resend. An example would
> be the rebalanceTimeoutMs, currently we have the following code:
>
>
> {code:java}
> ConsumerGroupHeartbeatRequestData data = new
> ConsumerGroupHeartbeatRequestData()
> .setGroupId(membershipManager.groupId())
> .setMemberEpoch(membershipManager.memberEpoch())
> .setMemberId(membershipManager.memberId())
> .setRebalanceTimeoutMs(rebalanceTimeoutMs); {code}
>
>
> We should encapsulate these once-used fields into a class such as
> HeartbeatMetdataBuilder, and it should maintain a state of whether a certain
> field needs to be sent or not.
>
> Note that, currently only 3 fields are mandatory in the request:
> * groupId
> * memberEpoch
> * memberId
> Note that on retriable errors and network errors (ex. timeout) a full request
> should be sent to the broker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15978) New consumer sends OffsetCommit with empty member ID
Andrew Schofield created KAFKA-15978: Summary: New consumer sends OffsetCommit with empty member ID Key: KAFKA-15978 URL: https://issues.apache.org/jira/browse/KAFKA-15978 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.7.0 Running the trogdor tests with the new consumer, it seemed that offsets were not being committed correctly, although the records were being fetched successfully. Upon investigation, it seems that the commit request manager uses a cached member ID which means that its OffsetCommit requests are rejected by the group coordinator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15932) Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
[
https://issues.apache.org/jira/browse/KAFKA-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-15932.
--
Resolution: Fixed
> Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
> ---
>
> Key: KAFKA-15932
> URL: https://issues.apache.org/jira/browse/KAFKA-15932
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.7.0
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
> Labels: flaky-test
>
> Intermittently failing test for the new consumer.
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14859/1/tests/
> ```Error
> org.opentest4j.AssertionFailedError: Timed out before consuming expected 1
> records. The number consumed was 0.
> Stacktrace
> org.opentest4j.AssertionFailedError: Timed out before consuming expected 1
> records. The number consumed was 0.
> at
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
> at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
> at
> app//kafka.api.AbstractConsumerTest.consumeRecords(AbstractConsumerTest.scala:161)
> at
> app//kafka.api.AbstractConsumerTest.consumeAndVerifyRecords(AbstractConsumerTest.scala:128)
> at
> app//kafka.api.PlaintextConsumerTest.testSeek(PlaintextConsumerTest.scala:616)
> at
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566)
> at
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
> at
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
> at
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> at
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
> at
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
> at
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
> at
> app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
> at
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
> at
> app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> at
> app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> at
> app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
> at
> app//org.
[jira] [Created] (KAFKA-15986) New consumer group protocol integration test failures
Andrew Schofield created KAFKA-15986: Summary: New consumer group protocol integration test failures Key: KAFKA-15986 URL: https://issues.apache.org/jira/browse/KAFKA-15986 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Fix For: 3.7.0 A recent change in `AsyncKafkaConsumer.updateFetchPositions` has made fetching fail without returning records in some situations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15986) New consumer group protocol integration test failures
[ https://issues.apache.org/jira/browse/KAFKA-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15986. -- Resolution: Fixed > New consumer group protocol integration test failures > - > > Key: KAFKA-15986 > URL: https://issues.apache.org/jira/browse/KAFKA-15986 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 3.7.0 >Reporter: Andrew Schofield >Assignee: Andrew Schofield >Priority: Major > Labels: CTR > Fix For: 3.7.0 > > > A recent change in `AsyncKafkaConsumer.updateFetchPositions` has made > fetching fail without returning records in some situations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15775) Implement listTopics() and partitionFor() for the AsyncKafkaConsumer
[
https://issues.apache.org/jira/browse/KAFKA-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Schofield resolved KAFKA-15775.
--
Resolution: Fixed
> Implement listTopics() and partitionFor() for the AsyncKafkaConsumer
>
>
> Key: KAFKA-15775
> URL: https://issues.apache.org/jira/browse/KAFKA-15775
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Andrew Schofield
>Priority: Major
> Labels: consumer-threading-refactor, kip-848,
> kip-848-client-support, kip-848-e2e, kip-848-preview
>
> {code:java}
> @Override
> public List partitionsFor(String topic) {
> return partitionsFor(topic, Duration.ofMillis(defaultApiTimeoutMs));
> }
> @Override
> public List partitionsFor(String topic, Duration timeout) {
> throw new KafkaException("method not implemented");
> }
> @Override
> public Map> listTopics() {
> return listTopics(Duration.ofMillis(defaultApiTimeoutMs));
> }
> @Override
> public Map> listTopics(Duration timeout) {
> throw new KafkaException("method not implemented");
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-16092) Queues for Kafka
Andrew Schofield created KAFKA-16092: Summary: Queues for Kafka Key: KAFKA-16092 URL: https://issues.apache.org/jira/browse/KAFKA-16092 Project: Kafka Issue Type: Improvement Reporter: Andrew Schofield Assignee: Andrew Schofield This Jira tracks the development of KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16099) Handle timeouts for AsyncKafkaConsumer.commitSync
Andrew Schofield created KAFKA-16099: Summary: Handle timeouts for AsyncKafkaConsumer.commitSync Key: KAFKA-16099 URL: https://issues.apache.org/jira/browse/KAFKA-16099 Project: Kafka Issue Type: Sub-task Components: clients Reporter: Andrew Schofield The handling of synchronous offset commits in the background thread does not observe the caller's timeout. In the situation that a commit request needs to be retried, the retries should not extend beyond the caller's timeout. The CommitApplicationEvent should contain the timeout and not continue beyond that time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16100) Consistent handling of timeouts and responses for new consumer ApplicationEvents
Andrew Schofield created KAFKA-16100: Summary: Consistent handling of timeouts and responses for new consumer ApplicationEvents Key: KAFKA-16100 URL: https://issues.apache.org/jira/browse/KAFKA-16100 Project: Kafka Issue Type: Sub-task Components: clients Reporter: Andrew Schofield The handling of timeouts and responses for the various kinds of ApplicationEvents in the new consumer is not consistent. A small amount of refactoring would make the code more maintainable and give consistent behaviour for the different requests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16104) Enable additional PlaintextConsumerTest tests for new consumer
Andrew Schofield created KAFKA-16104: Summary: Enable additional PlaintextConsumerTest tests for new consumer Key: KAFKA-16104 URL: https://issues.apache.org/jira/browse/KAFKA-16104 Project: Kafka Issue Type: Improvement Components: clients Reporter: Andrew Schofield It should be possible to enable: * testAutoCommitOnClose * testAutoCommitOnCloseAfterWakeup * testExpandingTopicSubscriptions * testShrinkingTopicSubscriptions * testMultiConsumerSessionTimeoutOnClose (KAFKA-16011 has been fixed) * testAutoCommitOnRebalance * testPerPartitionLeadMetricsCleanUpWithSubscribe * testPerPartitionLagMetricsCleanUpWithSubscribe * testStaticConsumerDetectsNewPartitionCreatedAfterRestart -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16137) ListClientMetricsResourcesResponse definition is missing field descriptions
Andrew Schofield created KAFKA-16137: Summary: ListClientMetricsResourcesResponse definition is missing field descriptions Key: KAFKA-16137 URL: https://issues.apache.org/jira/browse/KAFKA-16137 Project: Kafka Issue Type: Task Components: admin Affects Versions: 3.7.0 Reporter: Andrew Schofield This is purely improving the readability of the Kafka protocol documentation by adding missing description information for the fields of the `ListClientMetricsResources` response. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16182) Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest
Andrew Schofield created KAFKA-16182: Summary: Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest Key: KAFKA-16182 URL: https://issues.apache.org/jira/browse/KAFKA-16182 Project: Kafka Issue Type: Bug Components: admin Affects Versions: 3.7.0 Reporter: Andrew Schofield h3. Error org.apache.kafka.common.KafkaException: Error occurred while fetching client instance id Stacktrace org.apache.kafka.common.KafkaException: Error occurred while fetching client instance id at app//org.apache.kafka.clients.admin.KafkaAdminClient.clientInstanceId(KafkaAdminClient.java:4477) at app//org.apache.kafka.clients.admin.KafkaAdminClientTest.testClientInstanceId(KafkaAdminClientTest.java:7082) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568) at app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) at app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at app//org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) at app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base@17.0.7/java.util.ArrayList.forEach(ArrayList.java:1511) at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at app//org.junit.platform.engine.support
[jira] [Created] (KAFKA-16183) Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest
Andrew Schofield created KAFKA-16183: Summary: Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest Key: KAFKA-16183 URL: https://issues.apache.org/jira/browse/KAFKA-16183 Project: Kafka Issue Type: Bug Reporter: Andrew Schofield [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-15214/1/pipeline] h4. Error java.lang.AssertionError: assertion failed: Expect UnderReplicatedPartitionCount to be 0, but got: 1 h4. Stacktrace java.lang.AssertionError: assertion failed: Expect UnderReplicatedPartitionCount to be 0, but got: 1 at scala.Predef$.assert(Predef.scala:279) at kafka.integration.MetricsDuringTopicCreationDeletionTest.testMetricsDuringTopicCreateDelete(MetricsDuringTopicCreationDeletionTest.scala:123) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:226) at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:204) at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:142) at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:110) at java.base/java.util.str
[jira] [Created] (KAFKA-16184) Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest
Andrew Schofield created KAFKA-16184: Summary: Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest Key: KAFKA-16184 URL: https://issues.apache.org/jira/browse/KAFKA-16184 Project: Kafka Issue Type: Bug Reporter: Andrew Schofield [https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-15214/1/tests/] h4. Error org.opentest4j.AssertionFailedError: timed out waiting for replica movement h4. Stacktrace org.opentest4j.AssertionFailedError: timed out waiting for replica movement at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at org.junit.jupiter.api.Assertions.fail(Assertions.java:134) at kafka.api.PlaintextAdminIntegrationTest.$anonfun$testAlterReplicaLogDirs$7(PlaintextAdminIntegrationTest.scala:317) at kafka.api.PlaintextAdminIntegrationTest.$anonfun$testAlterReplicaLogDirs$7$adapted(PlaintextAdminIntegrationTest.scala:316) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at kafka.api.PlaintextAdminIntegrationTest.testAlterReplicaLogDirs(PlaintextAdminIntegrationTest.scala:316) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:45) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.
[jira] [Resolved] (KAFKA-8606) Provide a method to fetch committed offsets for a collection of TopicPartition
[ https://issues.apache.org/jira/browse/KAFKA-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-8606. - Fix Version/s: 2.4.0 Resolution: Fixed Method added in AK 2.4. > Provide a method to fetch committed offsets for a collection of TopicPartition > -- > > Key: KAFKA-8606 > URL: https://issues.apache.org/jira/browse/KAFKA-8606 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer >Affects Versions: 2.3.0, 2.2.1 >Reporter: ov7a >Priority: Major > Fix For: 2.4.0 > > > Currently KafkaConsumer has methods for fetching begging offsets, end offsets > and offsets for times, all of them accepting a collection of TopicPartition. > There is a method to fetch committed offset for single TopicPartition, but > there is no public API to fetch commited offsets for a collection of > TopicPartition. So, If one wants to fetch all committed offsets for topic, a > request per partition is created. > Note that ConsumerCoordinator.fetchCommittedOffsets which called internally > in "committed" method does accept a collection of TopicPartition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16243) Idle kafka-console-consumer with new consumer group protocol preemptively leaves group
Andrew Schofield created KAFKA-16243: Summary: Idle kafka-console-consumer with new consumer group protocol preemptively leaves group Key: KAFKA-16243 URL: https://issues.apache.org/jira/browse/KAFKA-16243 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 3.7.0 Reporter: Andrew Schofield Assignee: Andrew Schofield Using the new consumer group protocol with kafka-console-consumer.sh, I find that if I leave the consumer with no records to process for 5 minutes (max.poll.interval.ms = 30ms), the tool logs the following warning message and leaves the group. "consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records." With the older consumer, this did not occur. The reason is that the consumer keeps a poll timer which is used to ensure liveness of the application thread. The poll timer automatically updates while the `Consumer.poll(Duration)` method is blocked, while the newer consumer only updates the poll timer when a new call to `Consumer.poll(Duration)` is issued. This means that the kafka-console-consumer.sh tools, which uses a very long timeout by default, works differently with the new consumer. -- This message was sent by Atlassian Jira (v8.20.10#820010)
