[jira] [Commented] (KAFKA-8610) Don't use /bin/bash in scripts
[ https://issues.apache.org/jira/browse/KAFKA-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881828#comment-16881828 ] ASF GitHub Bot commented on KAFKA-8610: --- dongjinleekr commented on pull request #7063: KAFKA-8610: Don't use /bin/bash in scripts URL: https://github.com/apache/kafka/pull/7063 Currently, the interpreters (i.e., `/bin/sh`, `/bin/bash`, and `#!/usr/bin/env bash`) and indentations in the scripts are not unified. This PR improves portability by unifying them to use `/bin/sh`. It was validated with `shellcheck` and manual testing, but the more comprehensive test would be welcomed. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Don't use /bin/bash in scripts > -- > > Key: KAFKA-8610 > URL: https://issues.apache.org/jira/browse/KAFKA-8610 > Project: Kafka > Issue Type: Improvement >Reporter: Richard Lee >Priority: Minor > > On small container installations (such as alpine), /bin/bash is not > installed. It appears the scripts in the /bin directory would mostly work > with /bin/sh. Please use a simpler shell for shell scripts so that they are > more portable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8610) Don't use /bin/bash in scripts
[ https://issues.apache.org/jira/browse/KAFKA-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lee Dongjin reassigned KAFKA-8610: -- Assignee: Lee Dongjin > Don't use /bin/bash in scripts > -- > > Key: KAFKA-8610 > URL: https://issues.apache.org/jira/browse/KAFKA-8610 > Project: Kafka > Issue Type: Improvement >Reporter: Richard Lee >Assignee: Lee Dongjin >Priority: Minor > > On small container installations (such as alpine), /bin/bash is not > installed. It appears the scripts in the /bin directory would mostly work > with /bin/sh. Please use a simpler shell for shell scripts so that they are > more portable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8610) Don't use /bin/bash in scripts
[ https://issues.apache.org/jira/browse/KAFKA-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881834#comment-16881834 ] Lee Dongjin commented on KAFKA-8610: Hi [~llamahunter], Here is the draft fix. Please have a test and give some feedback. > Don't use /bin/bash in scripts > -- > > Key: KAFKA-8610 > URL: https://issues.apache.org/jira/browse/KAFKA-8610 > Project: Kafka > Issue Type: Improvement >Reporter: Richard Lee >Assignee: Lee Dongjin >Priority: Minor > > On small container installations (such as alpine), /bin/bash is not > installed. It appears the scripts in the /bin directory would mostly work > with /bin/sh. Please use a simpler shell for shell scripts so that they are > more portable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8636) Document behavior change for static members with `max.poll.interval.ms`
[ https://issues.apache.org/jira/browse/KAFKA-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881853#comment-16881853 ] ASF GitHub Bot commented on KAFKA-8636: --- hachikuji commented on pull request #7048: KAFKA-8636: add documentation change for max poll interval with static members URL: https://github.com/apache/kafka/pull/7048 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Document behavior change for static members with `max.poll.interval.ms` > --- > > Key: KAFKA-8636 > URL: https://issues.apache.org/jira/browse/KAFKA-8636 > Project: Kafka > Issue Type: Sub-task >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > The static member's behavior with max poll interval is potentially different > from current config documents. For example, if session timeout >> max poll > interval, static members will not leave the group until session timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8646) Materialized.withLoggingDisabled() does not disable the changelog topics creation
jmhostalet created KAFKA-8646:
-
Summary: Materialized.withLoggingDisabled() does not disable the
changelog topics creation
Key: KAFKA-8646
URL: https://issues.apache.org/jira/browse/KAFKA-8646
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.3.0
Reporter: jmhostalet
I have a cluster with 3 brokers running version 0.11
My kafka-streams app was using kafka-client 0.11.0.1 but recently I've migrated
to 2.3.0
I have no executed any migration as my data is disposable, therefore I have
deleted all intermediate topics, except input and output topics.
My streams config is:
{code:java}
application.id = consumer-id-v1.00
application.server =
bootstrap.servers = [foo1:9092, foo2:9092, foo3:9092]
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 524288000
client.id =
commit.interval.ms = 3
connections.max.idle.ms = 54
default.deserialization.exception.handler = class
org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.key.serde = class
org.apache.kafka.common.serialization.Serdes$StringSerde
default.production.exception.handler = class
org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class com.acme.stream.TimeExtractor
default.value.serde = class com.acme.serde.MyDtoSerde
max.task.idle.ms = 0
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
num.standby.replicas = 0
num.stream.threads = 25
partition.grouper = class
org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 4
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 60
state.dir = /tmp/kafka-streams
topology.optimization = none
upgrade.from = null
windowstore.changelog.additional.retention.ms = 8640
{code}
in my stream I am using withLoggingDisabled
{code:java}
stream.filter((key, val) -> val!=null)
.selectKey((key, val) -> getId(val))
.groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new
MyDtoSerde()))
.windowedBy(TimeWindows.of(aggregationWindowSizeDuration)
.grace(windowRetentionPeriodDuration))
.aggregate(MyDto::new,
new MyUpdater(),
Materialized.as("aggregation-updater")
.withLoggingDisabled()
.with(Serdes.String(), new MyDtoSerde()))
.toStream((k, v) -> k.key())
.mapValues(val -> { ...
{code}
but changelog topics are created (KSTREAM-AGGREGATE-STATE-STORE), no matter if
I delete them before running again the app or if I change the application.id
With a new application.id, topics are recreated with the new prefix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8646) Materialized.withLoggingDisabled() does not disable changelog topics creation
[
https://issues.apache.org/jira/browse/KAFKA-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jmhostalet updated KAFKA-8646:
--
Summary: Materialized.withLoggingDisabled() does not disable changelog
topics creation (was: Materialized.withLoggingDisabled() does not disable the
changelog topics creation)
> Materialized.withLoggingDisabled() does not disable changelog topics creation
> -
>
> Key: KAFKA-8646
> URL: https://issues.apache.org/jira/browse/KAFKA-8646
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.3.0
>Reporter: jmhostalet
>Priority: Minor
>
> I have a cluster with 3 brokers running version 0.11
> My kafka-streams app was using kafka-client 0.11.0.1 but recently I've
> migrated to 2.3.0
> I have no executed any migration as my data is disposable, therefore I have
> deleted all intermediate topics, except input and output topics.
> My streams config is:
>
> {code:java}
> application.id = consumer-id-v1.00
> application.server =
> bootstrap.servers = [foo1:9092, foo2:9092, foo3:9092]
> buffered.records.per.partition = 1000
> cache.max.bytes.buffering = 524288000
> client.id =
> commit.interval.ms = 3
> connections.max.idle.ms = 54
> default.deserialization.exception.handler = class
> org.apache.kafka.streams.errors.LogAndFailExceptionHandler
> default.key.serde = class
> org.apache.kafka.common.serialization.Serdes$StringSerde
> default.production.exception.handler = class
> org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
> default.timestamp.extractor = class com.acme.stream.TimeExtractor
> default.value.serde = class com.acme.serde.MyDtoSerde
> max.task.idle.ms = 0
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> num.standby.replicas = 0
> num.stream.threads = 25
> partition.grouper = class
> org.apache.kafka.streams.processor.DefaultPartitionGrouper
> poll.ms = 100
> processing.guarantee = at_least_once
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> replication.factor = 1
> request.timeout.ms = 4
> retries = 0
> retry.backoff.ms = 100
> rocksdb.config.setter = null
> security.protocol = PLAINTEXT
> send.buffer.bytes = 131072
> state.cleanup.delay.ms = 60
> state.dir = /tmp/kafka-streams
> topology.optimization = none
> upgrade.from = null
> windowstore.changelog.additional.retention.ms = 8640
> {code}
> in my stream I am using withLoggingDisabled
>
>
> {code:java}
> stream.filter((key, val) -> val!=null)
> .selectKey((key, val) -> getId(val))
> .groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new
> MyDtoSerde()))
> .windowedBy(TimeWindows.of(aggregationWindowSizeDuration)
>.grace(windowRetentionPeriodDuration))
> .aggregate(MyDto::new,
>new MyUpdater(),
>Materialized.as("aggregation-updater")
>.withLoggingDisabled()
>.with(Serdes.String(), new MyDtoSerde()))
> .toStream((k, v) -> k.key())
> .mapValues(val -> { ...
> {code}
> but changelog topics are created (KSTREAM-AGGREGATE-STATE-STORE), no matter
> if I delete them before running again the app or if I change the
> application.id
> With a new application.id, topics are recreated with the new prefix.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8646) Materialized.withLoggingDisabled() does not disable changelog topics creation
[
https://issues.apache.org/jira/browse/KAFKA-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jmhostalet updated KAFKA-8646:
--
Description:
I have a cluster with 3 brokers running version 0.11
My kafka-streams app was using kafka-client 0.11.0.1 but recently I've migrated
to 2.3.0
I have no executed any migration as my data is disposable, therefore I have
deleted all intermediate topics, except input and output topics.
My streams config is:
{code:java}
application.id = consumer-id-v1.00
application.server =
bootstrap.servers = [foo1:9092, foo2:9092, foo3:9092]
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 524288000
client.id =
commit.interval.ms = 3
connections.max.idle.ms = 54
default.deserialization.exception.handler = class
org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.key.serde = class
org.apache.kafka.common.serialization.Serdes$StringSerde
default.production.exception.handler = class
org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class com.acme.stream.TimeExtractor
default.value.serde = class com.acme.serde.MyDtoSerde
max.task.idle.ms = 0
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
num.standby.replicas = 0
num.stream.threads = 25
partition.grouper = class
org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 4
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 60
state.dir = /tmp/kafka-streams
topology.optimization = none
upgrade.from = null
windowstore.changelog.additional.retention.ms = 8640
{code}
in my stream I am using withLoggingDisabled
{code:java}
stream.filter((key, val) -> val!=null)
.selectKey((key, val) -> getId(val))
.groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new
MyDtoSerde()))
.windowedBy(TimeWindows.of(aggregationWindowSizeDuration)
.grace(windowRetentionPeriodDuration))
.aggregate(MyDto::new,
new MyUpdater(),
Materialized.as("aggregation-updater")
.withLoggingDisabled()
.with(Serdes.String(), new MyDtoSerde()))
.toStream((k, v) -> k.key())
.mapValues(val -> { ...
{code}
but changelog topics are created (KSTREAM-AGGREGATE-STATE-STORE), no matter if
I delete them before running again the app or if I change the application.id
With a new application.id, topics are recreated with the new prefix.
was:
I have a cluster with 3 brokers running version 0.11
My kafka-streams app was using kafka-client 0.11.0.1 but recently I've migrated
to 2.3.0
I have no executed any migration as my data is disposable, therefore I have
deleted all intermediate topics, except input and output topics.
My streams config is:
{code:java}
application.id = consumer-id-v1.00
application.server =
bootstrap.servers = [foo1:9092, foo2:9092, foo3:9092]
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 524288000
client.id =
commit.interval.ms = 3
connections.max.idle.ms = 54
default.deserialization.exception.handler = class
org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.key.serde = class
org.apache.kafka.common.serialization.Serdes$StringSerde
default.production.exception.handler = class
org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class com.acme.stream.TimeExtractor
default.value.serde = class com.acme.serde.MyDtoSerde
max.task.idle.ms = 0
metadata.max.age.ms = 30
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
num.standby.replicas = 0
num.stream.threads = 25
partition.grouper = class
org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 4
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 60
state.dir = /tmp/kafka-streams
topology.optimization = none
upgrade.from = null
windowstore.changelog.additional.retention.ms = 8640
{code}
in my stream I am using withLoggingDisabled
{code:java}
stream.filter((key, val) -> val!=null)
.selectKey((key, val) -> getId(val))
.groupByKey(Grouped.as("key-grouper").with(Serdes.String(), new
MyDtoSerde()))
.windowedBy(TimeWindows.of(aggregationWindowSizeDuration)
.grace(windo
[jira] [Created] (KAFKA-8647) Add consumer-props option to match producer-props for console consumer
Antony Stubbs created KAFKA-8647: Summary: Add consumer-props option to match producer-props for console consumer Key: KAFKA-8647 URL: https://issues.apache.org/jira/browse/KAFKA-8647 Project: Kafka Issue Type: Improvement Components: consumer, producer Affects Versions: 2.3.0 Reporter: Antony Stubbs Console consumer is missing a consumer-props option like the producer has. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8648) Console tools should fail fast if an unrecognised option is passed in
Antony Stubbs created KAFKA-8648: Summary: Console tools should fail fast if an unrecognised option is passed in Key: KAFKA-8648 URL: https://issues.apache.org/jira/browse/KAFKA-8648 Project: Kafka Issue Type: Improvement Components: clients, consumer, producer Affects Versions: 2.3.0 Reporter: Antony Stubbs It's confusing atm when you pass in incorrect cli options, but the tool silently accepts anything you give it. IMO it should ideally fail fast and tell you which option you passed in was not recognised. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job139.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job23145.txt, log-job23215.txt, > log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881907#comment-16881907
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/139
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=1146
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
{code}
log attached: log-job139.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job139.txt, log-job23145.txt, log-job23215.txt,
> log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Cadonna updated KAFKA-7937:
-
Attachment: log-job6122.txt
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.1.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: log-job6122.txt
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881924#comment-16881924
]
Bruno Cadonna commented on KAFKA-7937:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6122
{code:java}
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
coordinator is not available.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.$anonfun$describeConsumerGroups$1(ConsumerGroupCommand.scala:402)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeConsumerGroups(ConsumerGroupCommand.scala:401)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupsState(ConsumerGroupCommand.scala:478)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService.collectGroupState(ConsumerGroupCommand.scala:474)
at
kafka.admin.ResetConsumerGroupOffsetTest.$anonfun$testResetOffsetsNotExistingGroup$1(ResetConsumerGroupOffsetTest.scala:113)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:817)
at
kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:112)
{code}
logs attached: log-job6122.txt
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.1.1, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.4.0
>
> Attachments: log-job6122.txt
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[
https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881926#comment-16881926
]
Bruno Cadonna commented on KAFKA-8555:
--
https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/141
{code:java}
org.apache.kafka.connect.errors.DataException: Insufficient records committed
by connector simple-conn in 15000 millis. Records expected=2000, actual=924
at
org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)
at
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
{code}
log attached: log-job141.txt
> Flaky test ExampleConnectIntegrationTest#testSourceConnector
>
>
> Key: KAFKA-8555
> URL: https://issues.apache.org/jira/browse/KAFKA-8555
> Project: Kafka
> Issue Type: Bug
>Reporter: Boyang Chen
>Priority: Major
> Attachments: log-job139.txt, log-job141.txt, log-job23145.txt,
> log-job23215.txt, log-job6046.txt
>
>
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console]
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21*
> *02:03:21*
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest >
> testSourceConnector FAILED*02:03:21*
> org.apache.kafka.connect.errors.DataException: Insufficient records committed
> by connector simple-conn in 15000 millis. Records expected=2000,
> actual=1013*02:03:21* at
> org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21*
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Cadonna updated KAFKA-8555: - Attachment: log-job141.txt > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8647) Add consumer-props option to match producer-props for kafka-consumer-perf-test
[ https://issues.apache.org/jira/browse/KAFKA-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-8647: - Summary: Add consumer-props option to match producer-props for kafka-consumer-perf-test (was: Add consumer-props option to match producer-props for console consumer) > Add consumer-props option to match producer-props for kafka-consumer-perf-test > -- > > Key: KAFKA-8647 > URL: https://issues.apache.org/jira/browse/KAFKA-8647 > Project: Kafka > Issue Type: Improvement > Components: consumer, producer >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Minor > Labels: console > > Console consumer is missing a consumer-props option like the producer has. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8648) Console tools should fail fast if an unrecognised --property is passed in
[ https://issues.apache.org/jira/browse/KAFKA-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-8648: - Summary: Console tools should fail fast if an unrecognised --property is passed in (was: Console tools should fail fast if an unrecognised option is passed in) > Console tools should fail fast if an unrecognised --property is passed in > - > > Key: KAFKA-8648 > URL: https://issues.apache.org/jira/browse/KAFKA-8648 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer, producer >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Major > Labels: cli, console > > It's confusing atm when you pass in incorrect cli options, but the tool > silently accepts anything you give it. IMO it should ideally fail fast and > tell you which option you passed in was not recognised. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8647) Add consumer-props option to match producer-props for kafka-consumer-perf-test
[ https://issues.apache.org/jira/browse/KAFKA-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-8647: - Description: kafka-consumer-perf-test is missing a consumer-props option like the producer perf test has. was:Console consumer is missing a consumer-props option like the producer has. > Add consumer-props option to match producer-props for kafka-consumer-perf-test > -- > > Key: KAFKA-8647 > URL: https://issues.apache.org/jira/browse/KAFKA-8647 > Project: Kafka > Issue Type: Improvement > Components: consumer, producer >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Minor > Labels: console > > kafka-consumer-perf-test is missing a consumer-props option like the producer > perf test has. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8648) Console tools should fail fast if an unrecognised --property is passed in
[ https://issues.apache.org/jira/browse/KAFKA-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-8648: - Description: Some people don't realise that the --property for the console producer and consumer, is for passing options to the formatter, not the actual producer or consumer. If someone that doesn't know this, passes a property intending it to be a consumer or producer property, they are passed to the formatter, but if it's not a recognised option for the formatter, it fails silently and the user is wondering what happened to their intended setting. The console tools otherwise fail on unrecognised arguments, it would be great if the default formatter also failed with warning of the unrecognised option. was:It's confusing atm when you pass in incorrect cli options, but the tool silently accepts anything you give it. IMO it should ideally fail fast and tell you which option you passed in was not recognised. > Console tools should fail fast if an unrecognised --property is passed in > - > > Key: KAFKA-8648 > URL: https://issues.apache.org/jira/browse/KAFKA-8648 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer, producer >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Major > Labels: cli, console > > Some people don't realise that the --property for the console producer and > consumer, is for passing options to the formatter, not the actual producer or > consumer. > If someone that doesn't know this, passes a property intending it to be a > consumer or producer property, they are passed to the formatter, but if it's > not a recognised option for the formatter, it fails silently and the user is > wondering what happened to their intended setting. > The console tools otherwise fail on unrecognised arguments, it would be great > if the default formatter also failed with warning of the unrecognised option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8555) Flaky test ExampleConnectIntegrationTest#testSourceConnector
[ https://issues.apache.org/jira/browse/KAFKA-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881972#comment-16881972 ] Bruno Cadonna commented on KAFKA-8555: -- Related pull requests: * https://github.com/apache/kafka/pull/7061 * https://github.com/apache/kafka/pull/7019 > Flaky test ExampleConnectIntegrationTest#testSourceConnector > > > Key: KAFKA-8555 > URL: https://issues.apache.org/jira/browse/KAFKA-8555 > Project: Kafka > Issue Type: Bug >Reporter: Boyang Chen >Priority: Major > Attachments: log-job139.txt, log-job141.txt, log-job23145.txt, > log-job23215.txt, log-job6046.txt > > > [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/22798/console] > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector > failed, log available in > /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk8-scala2.11/connect/runtime/build/reports/testOutput/org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector.test.stdout*02:03:21* > *02:03:21* > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest > > testSourceConnector FAILED*02:03:21* > org.apache.kafka.connect.errors.DataException: Insufficient records committed > by connector simple-conn in 15000 millis. Records expected=2000, > actual=1013*02:03:21* at > org.apache.kafka.connect.integration.ConnectorHandle.awaitCommits(ConnectorHandle.java:188)*02:03:21* > at > org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector(ExampleConnectIntegrationTest.java:181)*02:03:21* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8637) WriteBatch objects leak off-heap memory
[ https://issues.apache.org/jira/browse/KAFKA-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman updated KAFKA-8637: --- Affects Version/s: 2.2.0 2.3.0 Priority: Blocker (was: Major) Fix Version/s: 2.3.1 2.2.2 2.1.2 > WriteBatch objects leak off-heap memory > --- > > Key: KAFKA-8637 > URL: https://issues.apache.org/jira/browse/KAFKA-8637 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0, 2.2.0, 2.3.0 >Reporter: Sophie Blee-Goldman >Priority: Blocker > Fix For: 2.1.2, 2.2.2, 2.3.1 > > > In 2.1 we did some refactoring that led to the WriteBatch objects in > RocksDBSegmentedBytesStore#restoreAllInternal being created in a separate > method, rather than in a try-with-resources statement as used elsewhere. This > causes a memory leak as the WriteBatches are no longer closed automatically -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8649) Error while rolling update from Kafka Streams 2.0.0 -> Kafka Streams 2.1.0
Suyash Garg created KAFKA-8649:
--
Summary: Error while rolling update from Kafka Streams 2.0.0 ->
Kafka Streams 2.1.0
Key: KAFKA-8649
URL: https://issues.apache.org/jira/browse/KAFKA-8649
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.0.0
Reporter: Suyash Garg
Fix For: 2.0.0
While doing a rolling update of a cluster of nodes running Kafka Streams
application, the stream threads in the nodes running the old version of the
library (2.0.0), fail with the following error:
{code:java}
[ERROR] [application-existing-StreamThread-336]
[o.a.k.s.p.internals.StreamThread] - stream-thread
[application-existing-StreamThread-336] Encountered the following error during
processing:
java.lang.IllegalArgumentException: version must be between 1 and 3; was: 4
#011at
org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfo.(SubscriptionInfo.java:67)
#011at
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.subscription(StreamsPartitionAssignor.java:312)
#011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.metadata(ConsumerCoordinator.java:176)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest(AbstractCoordinator.java:515)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.initiateJoinGroup(AbstractCoordinator.java:466)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:412)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
#011at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
#011at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333)
#011at
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
#011at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
#011at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:861)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:814)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
#011at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-8367) Non-heap memory leak in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880745#comment-16880745 ] Sophie Blee-Goldman edited comment on KAFKA-8367 at 7/10/19 4:26 PM: - Hey [~pavelsavov]! Sorry for the long silence. I believe we've tracked down another memory leak, which if you're able to test it out can be found here: [https://github.com/apache/kafka/pull/7049] This issue seems like it actually does affect 2.1 in addition to 2.2, but the patch was confirmed to fix the leak someone else was experiencing using a similar setup to yours on 2.2 so please give it a try. was (Author: ableegoldman): Hey [~pavelsavov]! Sorry for the long silence. I believe we've tracked down another memory leak, which if you're able to test it out can be found here: [https://github.com/apache/kafka/pull/7049] It may not be the only leak, as it affects 2.1 as well as 2.2, but we believe it's still an important fix. If this does not help we'll keep digging > Non-heap memory leak in Kafka Streams > - > > Key: KAFKA-8367 > URL: https://issues.apache.org/jira/browse/KAFKA-8367 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.2.0 >Reporter: Pavel Savov >Priority: Major > Attachments: memory-prod.png, memory-test.png > > > We have been observing a non-heap memory leak after upgrading to Kafka > Streams 2.2.0 from 2.0.1. We suspect the source to be around RocksDB as the > leak only happens when we enable stateful stream operations (utilizing > stores). We are aware of *KAFKA-8323* and have created our own fork of 2.2.0 > and ported the fix scheduled for release in 2.2.1 to our fork. It did not > stop the leak, however. > We are having this memory leak in our production environment where the > consumer group is auto-scaled in and out in response to changes in traffic > volume, and in our test environment where we have two consumers, no > autoscaling and relatively constant traffic. > Below is some information I'm hoping will be of help: > * RocksDB Config: > Block cache size: 4 MiB > Write buffer size: 2 MiB > Block size: 16 KiB > Cache index and filter blocks: true > Manifest preallocation size: 64 KiB > Max write buffer number: 3 > Max open files: 6144 > > * Memory usage in production > The attached graph (memory-prod.png) shows memory consumption for each > instance as a separate line. The horizontal red line at 6 GiB is the memory > limit. > As illustrated on the attached graph from production, memory consumption in > running instances goes up around autoscaling events (scaling the consumer > group either in or out) and associated rebalancing. It stabilizes until the > next autoscaling event but it never goes back down. > An example of scaling out can be seen from around 21:00 hrs where three new > instances are started in response to a traffic spike. > Just after midnight traffic drops and some instances are shut down. Memory > consumption in the remaining running instances goes up. > Memory consumption climbs again from around 6:00AM due to increased traffic > and new instances are being started until around 10:30AM. Memory consumption > never drops until the cluster is restarted around 12:30. > > * Memory usage in test > As illustrated by the attached graph (memory-test.png) we have a fixed number > of two instances in our test environment and no autoscaling. Memory > consumption rises linearly until it reaches the limit (around 2:00 AM on > 5/13) and Mesos restarts the offending instances, or we restart the cluster > manually. > > * No heap leaks observed > * Window retention: 2 or 11 minutes (depending on operation type) > * Issue not present in Kafka Streams 2.0.1 > * No memory leak for stateless stream operations (when no RocksDB stores are > used) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8650) Streams does not work as expected with auto.offset.reset=none
Raman Gupta created KAFKA-8650: -- Summary: Streams does not work as expected with auto.offset.reset=none Key: KAFKA-8650 URL: https://issues.apache.org/jira/browse/KAFKA-8650 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 2.3.0 Reporter: Raman Gupta The auto.offset.reset policy of none is useful as a safety measure, especially when * exactly-once processing is desired, or * at-least-once is desired, but it is expensive to reprocess from the beginning. In this case, using "none" forces the ops team to explicitly set the offset before the stream can re-start processing, in the (hopefully rare) situations in which the stream consumer offset has been lost for some reason, or in the case of a new stream that should not start processing from the beginning or the end, but somewhere in the middle (this scenario might occur during topic migrations). Kafka streams really only supports auto.offset.reset of earliest or latest (see the `Topology.AutoOffsetReset` enum). It is also possible to use the auto.offset.reset configuration value, but this works suboptimally because if the streams application reset tool is used (even with a specific offset specified), the offset is set for the input topic, but it is not, and cannot be, set for the internal topics, which won't exist yet. The internal topics are created by Kafka streams at startup time, but because the auto.offset.reset policy of "none" is passed to the consumer of those internal topics, the Kafka stream fails to start with a "NoOffsetForPartitionException". Proposals / options: 1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it affects the input topics, but not the internal topics. 2) Allow streams to be configured with auto.offset.reset=none, but explicitly set the offset to 0 for newly created internal topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8648) Console tools should fail fast if an unrecognised --property is passed in
[ https://issues.apache.org/jira/browse/KAFKA-8648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882248#comment-16882248 ] ASF GitHub Bot commented on KAFKA-8648: --- astubbs commented on pull request #7066: KAFKA-8648: Throw exception if bad --property sent to ConsoleConsumer URL: https://github.com/apache/kafka/pull/7066 Doesn’t check ConsoleProducer Doesn’t check properties sent to key/value deserialisers Includes unit test, also adds a test for the Custom Value Deserializer option. Also includes a more reliable way to test for system exit, if the code under test might throw an illegal argument exception. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Console tools should fail fast if an unrecognised --property is passed in > - > > Key: KAFKA-8648 > URL: https://issues.apache.org/jira/browse/KAFKA-8648 > Project: Kafka > Issue Type: Improvement > Components: clients, consumer, producer >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Major > Labels: cli, console > > Some people don't realise that the --property for the console producer and > consumer, is for passing options to the formatter, not the actual producer or > consumer. > If someone that doesn't know this, passes a property intending it to be a > consumer or producer property, they are passed to the formatter, but if it's > not a recognised option for the formatter, it fails silently and the user is > wondering what happened to their intended setting. > The console tools otherwise fail on unrecognised arguments, it would be great > if the default formatter also failed with warning of the unrecognised option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8634) Update ZooKeeper to 3.5.5
[ https://issues.apache.org/jira/browse/KAFKA-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882249#comment-16882249 ] ASF GitHub Bot commented on KAFKA-8634: --- ijuma commented on pull request #6802: KAFKA-8634: Update ZooKeeper to 3.5.5 URL: https://github.com/apache/kafka/pull/6802 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update ZooKeeper to 3.5.5 > - > > Key: KAFKA-8634 > URL: https://issues.apache.org/jira/browse/KAFKA-8634 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Ismael Juma >Priority: Major > Fix For: 2.4.0 > > > ZooKeeper 3.5.5 is the first stable release in the 3.5.x series. The key new > feature in ZK 3.5.x is TLS support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8650) Streams does not work as expected with auto.offset.reset=none
[ https://issues.apache.org/jira/browse/KAFKA-8650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-8650: --- Labels: needs-kip (was: ) > Streams does not work as expected with auto.offset.reset=none > - > > Key: KAFKA-8650 > URL: https://issues.apache.org/jira/browse/KAFKA-8650 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Raman Gupta >Priority: Major > Labels: needs-kip > > The auto.offset.reset policy of none is useful as a safety measure, > especially when > * exactly-once processing is desired, or > * at-least-once is desired, but it is expensive to reprocess from the > beginning. > In this case, using "none" forces the ops team to explicitly set the offset > before the stream can re-start processing, in the (hopefully rare) situations > in which the stream consumer offset has been lost for some reason, or in the > case of a new stream that should not start processing from the beginning or > the end, but somewhere in the middle (this scenario might occur during topic > migrations). > Kafka streams really only supports auto.offset.reset of earliest or latest > (see the `Topology.AutoOffsetReset` enum). It is also possible to use the > auto.offset.reset configuration value, but this works suboptimally because if > the streams application reset tool is used (even with a specific offset > specified), the offset is set for the input topic, but it is not, and cannot > be, set for the internal topics, which won't exist yet. > The internal topics are created by Kafka streams at startup time, but because > the auto.offset.reset policy of "none" is passed to the consumer of those > internal topics, the Kafka stream fails to start with a > "NoOffsetForPartitionException". > Proposals / options: > 1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it > affects the input topics, but not the internal topics. > 2) Allow streams to be configured with auto.offset.reset=none, but explicitly > set the offset to 0 for newly created internal topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8651) Add a #branch overload that takes a Map of predicate names to predicates
Antony Stubbs created KAFKA-8651: Summary: Add a #branch overload that takes a Map of predicate names to predicates Key: KAFKA-8651 URL: https://issues.apache.org/jira/browse/KAFKA-8651 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 2.3.0 Reporter: Antony Stubbs A map can be a more natural way to work with passing in predicates and extracting streams when working with the #branch method. KIP and PR on their way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8651) Add a #branch overload that takes a Map of predicate names to predicates
[ https://issues.apache.org/jira/browse/KAFKA-8651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882296#comment-16882296 ] ASF GitHub Bot commented on KAFKA-8651: --- astubbs commented on pull request #7068: KAFKA-8651: Add predicate map to #branch URL: https://github.com/apache/kafka/pull/7068 Using #branch with an array is error prone and not readable, due to referencing the results by magic index offsets. So, we suggest to add an API that allows naming different branches, which is more natural. See KIP (on it's way) ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a #branch overload that takes a Map of predicate names to predicates > > > Key: KAFKA-8651 > URL: https://issues.apache.org/jira/browse/KAFKA-8651 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Minor > > A map can be a more natural way to work with passing in predicates and > extracting streams when working with the #branch method. > > KIP and PR on their way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8651) Add a #branch overload that takes a Map of predicate names to predicates
[ https://issues.apache.org/jira/browse/KAFKA-8651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-8651: - Description: A map can be a more natural way to work with passing in predicates and extracting streams when working with the #branch method. https://github.com/apache/kafka/pull/7068 KIP and PR on their way. was: A map can be a more natural way to work with passing in predicates and extracting streams when working with the #branch method. KIP and PR on their way. > Add a #branch overload that takes a Map of predicate names to predicates > > > Key: KAFKA-8651 > URL: https://issues.apache.org/jira/browse/KAFKA-8651 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Minor > > A map can be a more natural way to work with passing in predicates and > extracting streams when working with the #branch method. > > https://github.com/apache/kafka/pull/7068 > > KIP and PR on their way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8650) Streams does not work as expected with auto.offset.reset=none
[ https://issues.apache.org/jira/browse/KAFKA-8650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882307#comment-16882307 ] Matthias J. Sax commented on KAFKA-8650: Thanks for creating this ticket. Sounds reasonable to me. I marked it as "need-kip" because if we add `none` as new options it's technically an API change. But maybe it's trivial enough to just skip the KIP process for it. For option (2), it's actually an interesting question and will be hard to get right. You only want to set offsets to zero iff those topics are newly created. Hence, topic creation and committing corresponding offsets for the application.id would need to be atomic. (We would need to change the create topic request what also requires a KIP; as an alternative, it might be possible to first commit offsets, and create those topics afterwards). For EOS, if you loose the repartition offsets (eg, because you application was offline for a long time), it could also be an error condition on which you want to stop. Hence, using rest policy earliest/latest might actually violate EOS, too. On the other hand, it might be sufficient to just use `earliest` for repartition topics, because if you loose offsets, you would loose them not just for the repartition topics but also for the input topic. For this case, I would assume that a clean reset of the application to new start offsets for input topics, would require to wipe out the repartition topics anyway? Another idea would be, to not change anything in KafkaStreams, but enhance the reset-tool to actually commit offsets zero for repartition topics after they were deleted (need to double check if it's possible to commit offsets for a non-existing topic). > Streams does not work as expected with auto.offset.reset=none > - > > Key: KAFKA-8650 > URL: https://issues.apache.org/jira/browse/KAFKA-8650 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Raman Gupta >Priority: Major > Labels: needs-kip > > The auto.offset.reset policy of none is useful as a safety measure, > especially when > * exactly-once processing is desired, or > * at-least-once is desired, but it is expensive to reprocess from the > beginning. > In this case, using "none" forces the ops team to explicitly set the offset > before the stream can re-start processing, in the (hopefully rare) situations > in which the stream consumer offset has been lost for some reason, or in the > case of a new stream that should not start processing from the beginning or > the end, but somewhere in the middle (this scenario might occur during topic > migrations). > Kafka streams really only supports auto.offset.reset of earliest or latest > (see the `Topology.AutoOffsetReset` enum). It is also possible to use the > auto.offset.reset configuration value, but this works suboptimally because if > the streams application reset tool is used (even with a specific offset > specified), the offset is set for the input topic, but it is not, and cannot > be, set for the internal topics, which won't exist yet. > The internal topics are created by Kafka streams at startup time, but because > the auto.offset.reset policy of "none" is passed to the consumer of those > internal topics, the Kafka stream fails to start with a > "NoOffsetForPartitionException". > Proposals / options: > 1) Allow auto.offset.reset=none to be specified in Consumed.with() so that it > affects the input topics, but not the internal topics. > 2) Allow streams to be configured with auto.offset.reset=none, but explicitly > set the offset to 0 for newly created internal topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8651) Add a #branch overload that takes a Map of predicate names to predicates
[ https://issues.apache.org/jira/browse/KAFKA-8651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882312#comment-16882312 ] Matthias J. Sax commented on KAFKA-8651: Seems to duplicate https://issues.apache.org/jira/browse/KAFKA-5488 ? > Add a #branch overload that takes a Map of predicate names to predicates > > > Key: KAFKA-8651 > URL: https://issues.apache.org/jira/browse/KAFKA-8651 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Antony Stubbs >Priority: Minor > > A map can be a more natural way to work with passing in predicates and > extracting streams when working with the #branch method. > > https://github.com/apache/kafka/pull/7068 > > KIP and PR on their way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8571) Not complete delayed produce requests when processing StopReplicaRequest causing high produce latency for acks=all
[
https://issues.apache.org/jira/browse/KAFKA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882357#comment-16882357
]
ASF GitHub Bot commented on KAFKA-8571:
---
kidkun commented on pull request #7069: KAFKA-8571: Clean up purgatory when
leader replica is kicked out of replica list.
URL: https://github.com/apache/kafka/pull/7069
Upon receiving `StopReplicaRequest`, broker should check its purgatory and
finish related pending produce/consume requests if any; otherwise client will
wait unnecessary long time before getting timeout response.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Not complete delayed produce requests when processing StopReplicaRequest
> causing high produce latency for acks=all
> --
>
> Key: KAFKA-8571
> URL: https://issues.apache.org/jira/browse/KAFKA-8571
> Project: Kafka
> Issue Type: Bug
>Reporter: Zhanxiang (Patrick) Huang
>Assignee: Zhanxiang (Patrick) Huang
>Priority: Major
>
> Currently a broker will only attempt to complete delayed requests upon
> highwater mark changes and receiving LeaderAndIsrRequest. When a broker
> receives StopReplicaRequest, it will not try to complete delayed operations
> including delayed produce for acks=all, which can cause the producer to
> timeout even though the producer should have attempted to talk to the new
> leader faster if a NotLeaderForPartition error is sent.
> This can happen during partition reassignment when controller is trying to
> kick the previous leader out of the replica set. It this case, controller
> will only send StopReplicaRequest (not LeaderAndIsrRequest) to the previous
> leader in the replica set shrink phase. Here is an example:
> {noformat}
> During Reassign the replica set of partition A from [B1, B2] to [B2, B3]:
> t0: Controller expands the replica set to [B1, B2, B3]
> t1: B1 receives produce request PR on partition A with acks=all and timetout
> T. B1 puts PR into the DelayedProducePurgatory with timeout T.
> t2: Controller elects B2 as the new leader and shrinks the replica set fo
> [B2, B3]. LeaderAndIsrRequests are sent to B2 and B3. StopReplicaRequest is
> sent to B!.
> t3: B1 receives StopReplicaRequest but doesn't try to comeplete PR.
> If PR cannot be fullfilled by t3, and t1 + T > t3, PR will eventually time
> out in the purgatory and producer will eventually time out the produce
> request.{noformat}
> Since it is possible for the leader to receive only a StopReplicaRequest
> (without receiving any LeaderAndIsrRequest) to leave the replica set, a fix
> for this issue is to also try to complete delay operations in processing
> StopReplicaRequest.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8652) Jenkins stop triggering tests for prs since 07/10
Boyang Chen created KAFKA-8652: -- Summary: Jenkins stop triggering tests for prs since 07/10 Key: KAFKA-8652 URL: https://issues.apache.org/jira/browse/KAFKA-8652 Project: Kafka Issue Type: Bug Reporter: Boyang Chen Jenkins stop triggering PR builds since 07/10, samples are: [https://github.com/apache/kafka/pull/6933] [https://github.com/apache/kafka/pull/6805] [https://github.com/apache/kafka/pull/7021] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882449#comment-16882449
]
James Ritt commented on KAFKA-4212:
---
Thanks [~ableegoldman] & [~mjsax]! So I took a look at
[`RocksDBConfigSetter`|https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/RocksDBConfigSetter.java#L45]
and my current understanding is that its `setConfig` method is used to mutate
the provided `org.rocksdb.Options` before it's subsequently passed into the
rocks DB constructor. Unfortunately, ttl doesn't seem to be configurable within
`org.rocksdb.Options`, and instead, as far as I can tell, the usage of the
TtlDB would instead need to be enacted by using the appropriate constructor
[here|https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L191].
Please let me know if I'm missing something, but given the above, the next
approach I was considering was adding a StreamsConfig rocksDbTtl setting
similar to [here;|https://github.com/apache/kafka/pull/2159/files] your
thoughts?
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7157) Connect TimestampConverter SMT doesn't handle null values
[
https://issues.apache.org/jira/browse/KAFKA-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882464#comment-16882464
]
ASF GitHub Bot commented on KAFKA-7157:
---
rayokota commented on pull request #7070: KAFKA-7157: Fix handling of nulls in
TimestampConverter
URL: https://github.com/apache/kafka/pull/7070
This is based on the PR from @Nimfadora
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Connect TimestampConverter SMT doesn't handle null values
> -
>
> Key: KAFKA-7157
> URL: https://issues.apache.org/jira/browse/KAFKA-7157
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
>
> TimestampConverter SMT is not able to handle null values (in any versions),
> so it's always trying to apply the transformation to the value. Instead, it
> needs to check for null and use the default value for the new schema's field.
> {noformat}
> [2018-07-03 02:31:52,490] ERROR Task MySourceConnector-2 threw an uncaught
> and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.TimestampConverter$2.toRaw(TimestampConverter.java:137)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.convertTimestamp(TimestampConverter.java:440)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyValueWithSchema(TimestampConverter.java:368)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.applyWithSchema(TimestampConverter.java:358)
>
> at
> org.apache.kafka.connect.transforms.TimestampConverter.apply(TimestampConverter.java:275)
>
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:435)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:264)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
>
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
>
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
> [2018-07-03 02:31:52,491] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8424) Replace ListGroups request/response with automated protocol
[ https://issues.apache.org/jira/browse/KAFKA-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882469#comment-16882469 ] ASF GitHub Bot commented on KAFKA-8424: --- cmccabe commented on pull request #6805: KAFKA-8424: replace ListGroups request/response with automated protocol URL: https://github.com/apache/kafka/pull/6805 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Replace ListGroups request/response with automated protocol > --- > > Key: KAFKA-8424 > URL: https://issues.apache.org/jira/browse/KAFKA-8424 > Project: Kafka > Issue Type: Sub-task >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8570) Downconversion could fail when log contains out of order message formats
[
https://issues.apache.org/jira/browse/KAFKA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882488#comment-16882488
]
ASF GitHub Bot commented on KAFKA-8570:
---
dhruvilshah3 commented on pull request #7071: KAFKA-8570: Grow buffer to hold
down converted records if it was insufficiently sized
URL: https://github.com/apache/kafka/pull/7071
Backport https://github.com/apache/kafka/pull/6974 to 1.1
When the log contains out of order message formats (for example v2 message
followed by v1 message) and consists of compressed batches typically greater
than 1kB in size, it is possible for down-conversion to fail. With compressed
batches, we estimate the size of down-converted batches using:
```
private static int estimateCompressedSizeInBytes(int size,
CompressionType compressionType) {
return compressionType == CompressionType.NONE ? size :
Math.min(Math.max(size / 2, 1024), 1 << 16);
}
```
This almost always underestimates size of down-converted records if the
batch is between 1kB-64kB in size. In general, this means we may under estimate
the total size required for compressed batches.
Because of an implicit assumption in the code that messages with a lower
message format appear before any with a higher message format, we do not grow
the buffer we copy the down converted records into when we see a message <= the
target message format. This assumption becomes incorrect when the log contains
out of order message formats, for example because of leaders flapping while
upgrading the message format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Downconversion could fail when log contains out of order message formats
>
>
> Key: KAFKA-8570
> URL: https://issues.apache.org/jira/browse/KAFKA-8570
> Project: Kafka
> Issue Type: Bug
>Reporter: Dhruvil Shah
>Assignee: Dhruvil Shah
>Priority: Major
> Fix For: 2.0.2, 2.1.2, 2.2.2, 2.3.1
>
>
> When the log contains out of order message formats (for example a v2 message
> followed by a v1 message), it is possible for down-conversion to fail in
> certain scenarios where batches compressed and greater than 1kB in size.
> Down-conversion fails with a stack like the following:
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.kafka.common.record.FileLogInputStream$FileChannelRecordBatch.writeTo(FileLogInputStream.java:176)
> at
> org.apache.kafka.common.record.AbstractRecords.downConvert(AbstractRecords.java:107)
> at
> org.apache.kafka.common.record.FileRecords.downConvert(FileRecords.java:242)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-8636) Document behavior change for static members with `max.poll.interval.ms`
[ https://issues.apache.org/jira/browse/KAFKA-8636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Chen resolved KAFKA-8636. Resolution: Fixed > Document behavior change for static members with `max.poll.interval.ms` > --- > > Key: KAFKA-8636 > URL: https://issues.apache.org/jira/browse/KAFKA-8636 > Project: Kafka > Issue Type: Sub-task >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > The static member's behavior with max poll interval is potentially different > from current config documents. For example, if session timeout >> max poll > interval, static members will not leave the group until session timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882516#comment-16882516
]
Sophie Blee-Goldman commented on KAFKA-4212:
Agreed, this wouldn't be possible within the current RocksDBConfigSetter. I
think the idea was to extend its functionality somehow to make ttl possible
while still keeping all the rocksdb related configs in the same place, and
making it clear that this is not "Streams retention" but "RocksDB's TTL"
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7614) Fix scalac warnings and enable warnings as errors
[ https://issues.apache.org/jira/browse/KAFKA-7614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tu Tran reassigned KAFKA-7614: -- Assignee: Tu Tran > Fix scalac warnings and enable warnings as errors > - > > Key: KAFKA-7614 > URL: https://issues.apache.org/jira/browse/KAFKA-7614 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Tu Tran >Priority: Major > > The only way to keep warnings out is to treat them as errors. Unfortunately, > some deprecation warnings cannot be disabled with scalac alone. The silencer > compiler plugin > ([https://github.com/ghik/silencer|https://github.com/ghik/silencer]) may > provide a way out. We should try it and see if it makes it possible to fix > all the warnings so that we can treat them as errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8653) Regression in JoinGroup v0 rebalance timeout handling
Jason Gustafson created KAFKA-8653: -- Summary: Regression in JoinGroup v0 rebalance timeout handling Key: KAFKA-8653 URL: https://issues.apache.org/jira/browse/KAFKA-8653 Project: Kafka Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jason Gustafson The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
[
https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882520#comment-16882520
]
Matthias J. Sax commented on KAFKA-4212:
I see. IMHO, we should not have a parameter for `StreamsConfig` because that
would be coarse grained. We should have a way to enable it on a per-store
basis. Maybe we can allow to set it via the provided `Map` of
`RocksDBConfigSetter#config(...)` ?
> Add a key-value store that is a TTL persistent cache
>
>
> Key: KAFKA-4212
> URL: https://issues.apache.org/jira/browse/KAFKA-4212
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Priority: Major
> Labels: api
>
> Some jobs needs to maintain as state a large set of key-values for some
> period of time. I.e. they need to maintain a TTL cache of values potentially
> larger than memory.
> Currently Kafka Streams provides non-windowed and windowed key-value stores.
> Neither is an exact fit to this use case.
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as
> required, but does not support expiration. The TTL option of RocksDB is
> explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment
> dropping, but it stores multiple items per key, based on their timestamp.
> But this store can be repurposed as a cache by fetching the items in reverse
> chronological order and returning the first item found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here
> we desire a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be
> useful to have an official and proper TTL cache API and implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-8653) Regression in JoinGroup v0 rebalance timeout handling
[ https://issues.apache.org/jira/browse/KAFKA-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reassigned KAFKA-8653: -- Assignee: Jason Gustafson > Regression in JoinGroup v0 rebalance timeout handling > - > > Key: KAFKA-8653 > URL: https://issues.apache.org/jira/browse/KAFKA-8653 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > > The rebalance timeout was added to the JoinGroup protocol in version 1. Prior > to 2.3, we handled version 0 JoinGroup requests by setting the rebalance > timeout to be equal to the session timeout. We lost this logic when we > converted the API to use the generated schema definition which uses the > default value of -1. The impact of this is that the group rebalance timeout > becomes 0, so rebalances finish immediately after we enter the > PrepareRebalance state and kick out all old members. This causes consumer > groups to enter an endless rebalance loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6520) When a Kafka Stream can't communicate with the server, it's Status stays RUNNING
[ https://issues.apache.org/jira/browse/KAFKA-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882573#comment-16882573 ] Brian Putt commented on KAFKA-6520: --- Any updates on this? > When a Kafka Stream can't communicate with the server, it's Status stays > RUNNING > > > Key: KAFKA-6520 > URL: https://issues.apache.org/jira/browse/KAFKA-6520 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Michael Kohout >Assignee: Milind Jain >Priority: Major > Labels: newbie, user-experience > > When you execute the following scenario the application is always in RUNNING > state > > 1)start kafka > 2)start app, app connects to kafka and starts processing > 3)kill kafka(stop docker container) > 4)the application doesn't give any indication that it's no longer > connected(Stream State is still RUNNING, and the uncaught exception handler > isn't invoked) > > > It would be useful if the Stream State had a DISCONNECTED status. > > See > [this|https://groups.google.com/forum/#!topic/confluent-platform/nQh2ohgdrIQ] > for a discussion from the google user forum. > [This|https://issues.apache.org/jira/browse/KAFKA-4564] is a link to a > related issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8653) Regression in JoinGroup v0 rebalance timeout handling
[ https://issues.apache.org/jira/browse/KAFKA-8653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882576#comment-16882576 ] ASF GitHub Bot commented on KAFKA-8653: --- hachikuji commented on pull request #7072: KAFKA-8653; Default rebalance timeout to session timeout for JoinGroup v0 URL: https://github.com/apache/kafka/pull/7072 The rebalance timeout was added to the JoinGroup protocol in version 1. Prior to 2.3, we handled version 0 JoinGroup requests by setting the rebalance timeout to be equal to the session timeout. We lost this logic when we converted the API to use the generated schema definition which uses the default value of -1. The impact of this is that the group rebalance timeout becomes 0, so rebalances finish immediately after we enter the PrepareRebalance state and kick out all old members. This causes consumer groups to enter an endless rebalance loop. This patch restores the old behavior. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Regression in JoinGroup v0 rebalance timeout handling > - > > Key: KAFKA-8653 > URL: https://issues.apache.org/jira/browse/KAFKA-8653 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > > The rebalance timeout was added to the JoinGroup protocol in version 1. Prior > to 2.3, we handled version 0 JoinGroup requests by setting the rebalance > timeout to be equal to the session timeout. We lost this logic when we > converted the API to use the generated schema definition which uses the > default value of -1. The impact of this is that the group rebalance timeout > becomes 0, so rebalances finish immediately after we enter the > PrepareRebalance state and kick out all old members. This causes consumer > groups to enter an endless rebalance loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8620) Race condition in StreamThread state change
[ https://issues.apache.org/jira/browse/KAFKA-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882624#comment-16882624 ] ASF GitHub Bot commented on KAFKA-8620: --- abbccdda commented on pull request #7021: KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing URL: https://github.com/apache/kafka/pull/7021 We have detected a race condition under system test failure. The problem was that the task manager internal active tasks should be guarded against state changes on the stream thread. Could definitely consider other fixes but this is currently the make-sense one. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Race condition in StreamThread state change > --- > > Key: KAFKA-8620 > URL: https://issues.apache.org/jira/browse/KAFKA-8620 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > In the call to `StreamThread.addRecordsToTasks` we don't have synchronization > when we attempt to extract active tasks. If after one long poll in runOnce > the application state changes to PENDING_SHUTDOWN, there is a potential close > on TaskManager which erases the active tasks map, thus triggering NPE and > bringing the thread state to a false shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8620) Race condition in StreamThread state change
[ https://issues.apache.org/jira/browse/KAFKA-8620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882623#comment-16882623 ] ASF GitHub Bot commented on KAFKA-8620: --- ijuma commented on pull request #7021: KAFKA-8620: fix NPE due to race condition during shutdown while rebalancing URL: https://github.com/apache/kafka/pull/7021 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Race condition in StreamThread state change > --- > > Key: KAFKA-8620 > URL: https://issues.apache.org/jira/browse/KAFKA-8620 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Boyang Chen >Assignee: Boyang Chen >Priority: Major > > In the call to `StreamThread.addRecordsToTasks` we don't have synchronization > when we attempt to extract active tasks. If after one long poll in runOnce > the application state changes to PENDING_SHUTDOWN, there is a potential close > on TaskManager which erases the active tasks map, thus triggering NPE and > bringing the thread state to a false shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
