apoorvmittal10 commented on PR #19928:
URL: https://github.com/apache/kafka/pull/19928#issuecomment-2956678794
## Test results for performance:
Setup share groups to read from earliest:
```
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups
--entity-name share-group-tier3
--alter --add-config 'share.auto.offset.reset=earliest'
Completed updating config for group share-group-tier3.
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type groups
--entity-name share-group-tier6
--alter --add-config 'share.auto.offset.reset=earliest'
Completed updating config for group share-group-tier6.
```
### Local tiered storage - message size 1024 Bytes
```
bin/kafka-producer-perf-test.sh --topic tieredTopic1 --num-records 1200000
--record-size 1024
--throughput -1 --producer-props bootstrap.servers=localhost:9092
```
#### On trunk - OOM
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic tieredTopic1
--messages 1000000 --timeout 300000 --group share-group-tier3
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "RLMMConsumerTask"
[2025-06-09 19:14:15,035] ERROR [ExpirationReaper-1-Fetch]: Error due to
(org.apache.kafka.server.purgatory.DelayedOperationPurgatory$ExpiredOperationReaper)
java.lang.OutOfMemoryError: Java heap space
[2025-06-09 19:14:15,035] ERROR Uncaught exception in thread
'quorum-controller-1-event-handler': (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
[2025-06-09 19:14:15,035] ERROR Uncaught exception in scheduled task
'kafka-log-retention' (org.apache.kafka.server.util.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2025-06-09 19:14:15,036] INFO [ExpirationReaper-1-Fetch]: Stopped
(org.apache.kafka.server.purgatory.DelayedOperationPurgatory$ExpiredOperationReaper)
[2025-06-09 19:14:15,035] ERROR [ExpirationReaper-1-ShareFetch]: Error due
to
(org.apache.kafka.server.purgatory.DelayedOperationPurgatory$ExpiredOperationReaper)
java.lang.OutOfMemoryError: Java heap space
at
java.base/java.lang.invoke.DirectMethodHandle.allocateInstance(DirectMethodHandle.java:500)
~[?:?]
at
java.base/java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(DirectMethodHandle$Holder)
~[?:?]
at
java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder)
~[?:?]
at
org.apache.kafka.server.util.timer.SystemTimer.advanceClock(SystemTimer.java:96)
~[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
at
org.apache.kafka.server.purgatory.DelayedOperationPurgatory.advanceClock(DelayedOperationPurgatory.java:387)
~[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
at
org.apache.kafka.server.purgatory.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperationPurgatory.java:417)
~[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
at
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
```
#### On fix branch - Successful Consumption
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic tieredTopic1
--messages 1000000 --timeout 300000 --group share-group-tier6
start.time, end.time, data.consumed.in.MB, MB.sec, nMsg.sec,
data.consumed.in.nMsg, fetch.time.ms
[2025-06-09 19:29:50,195] WARN Share groups and KafkaShareConsumer are part
of a preview feature introduced by KIP-932, and are not recommended for use in
production.
(org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator)
2025-06-09 19:29:50:224, 2025-06-09 19:30:12:655, 977.0361, 43.5574,
44602.7819, 1000485, 22431
```
### Local tiered storage - message size 10 Bytes
```
bin/kafka-producer-perf-test.sh --topic tieredTopic2 --num-records 1000000
--record-size 10
--throughput -1 --producer-props bootstrap.servers=localhost:9092
```
#### On trunk - OOM
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic tieredTopic2
--messages 1000000 --timeout 300000 --group share-group-tier3
java.lang.OutOfMemoryError: Java heap space
[2025-06-09 19:21:54,077] ERROR
[controller-1-to-controller-registration-channel-manager]: Stopped due to fatal
error with exit code 1 (kafka.server.NodeToControllerRequestThread)
org.apache.kafka.common.internals.FatalExitError: null
at
org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:131)
~[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
at
kafka.server.NodeToControllerRequestThread.doWork(NodeToControllerChannelManager.scala:322)
~[kafka_2.13-4.1.0-SNAPSHOT.jar:?]
at
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
[2025-06-09 19:21:54,077] ERROR
[broker-1-to-controller-heartbeat-channel-manager]: Stopped due to fatal error
with exit code 1 (kafka.server.NodeToControllerRequestThread)
org.apache.kafka.common.internals.FatalExitError: null
at
org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:131)
~[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
at
kafka.server.NodeToControllerRequestThread.doWork(NodeToControllerChannelManager.scala:322)
~[kafka_2.13-4.1.0-SNAPSHOT.jar:?]
at
org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
[kafka-server-common-4.1.0-SNAPSHOT.jar:?]
```
#### On fix branch - Successful Consumption
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic tieredTopic2 --messages 1000000 --timeout 300000 --group
share-group-tier6
start.time, end.time, data.consumed.in.MB, MB.sec, nMsg.sec,
data.consumed.in.nMsg, fetch.time.ms
[2025-06-09 19:31:08,148] WARN Share groups and KafkaShareConsumer are part
of a preview feature introduced by KIP-932, and are not recommended for use in
production.
(org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator)
2025-06-09 19:31:08:176, 2025-06-09 19:31:21:874, 9.5367, 0.6962,
73003.3582, 1000000, 13698
```
### Local log fetch - message size 1024 Bytes
```
bin/kafka-producer-perf-test.sh --topic T2 --num-records 1200000
--record-size 1024
--throughput -1 --producer-props bootstrap.servers=localhost:9092
```
#### On trunk - 21347 ms consumption time
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic T2
--messages 1000000 --timeout 300000 --group share-group-tier3
start.time, end.time, data.consumed.in.MB, MB.sec, nMsg.sec,
data.consumed.in.nMsg, fetch.time.ms
[2025-06-09 19:26:40,870] WARN Share groups and KafkaShareConsumer are part
of a preview feature introduced by KIP-932, and are not recommended for use in
production.
(org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator)
2025-06-09 19:26:40:902, 2025-06-09 19:27:02:249, 976.6699, 45.7521,
46850.1429, 1000110, 21347
```
#### On fix branch - 20254 ms consumption time
```
bin/kafka-share-consumer-perf-test.sh --bootstrap-server localhost:9092
--topic T2
--messages 1000000 --timeout 300000 --group share-group-tier6
start.time, end.time, data.consumed.in.MB, MB.sec, nMsg.sec,
data.consumed.in.nMsg, fetch.time.ms
[2025-06-09 19:34:04,444] WARN Share groups and KafkaShareConsumer are part
of a preview feature introduced by KIP-932, and are not recommended for use in
production.
(org.apache.kafka.clients.consumer.internals.ShareConsumerDelegateCreator)
2025-06-09 19:34:04:474, 2025-06-09 19:34:24:728, 976.6699, 48.2211,
49378.3944, 1000110, 20254
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
