[jira] [Commented] (KAFKA-8037) KTable restore may load bad data
[ https://issues.apache.org/jira/browse/KAFKA-8037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785420#comment-16785420 ] Patrik Kleindl commented on KAFKA-8037: --- [~guozhang] Does that work too if LogAndContinueExceptionHandler is used or will that "swallow" the error you mentioned above to know when to quit restoring? I am trying to work my way through the code and reproduce this with a test for the global state stores but I might need some help there. Just for my understanding, the TopologyDriver can't cover this scenario, or did I miss something there? Currently I am trying something similar to GlobalStateManagerImplTest.shouldSkipNullKeysWhenRestoring but was not really successful. > KTable restore may load bad data > > > Key: KAFKA-8037 > URL: https://issues.apache.org/jira/browse/KAFKA-8037 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Minor > > If an input topic contains bad data, users can specify a > `deserialization.exception.handler` to drop corrupted records on read. > However, this mechanism may be by-passed on restore. Assume a > `builder.table()` call reads and drops a corrupted record. If the table state > is lost and restored from the changelog topic, the corrupted record may be > copied into the store, because on restore plain bytes are copied. > If the KTable is used in a join, an internal `store.get()` call to lookup the > record would fail with a deserialization exception if the value part cannot > be deserialized. > GlobalKTables are affected, too (cf. KAFKA-7663 that may allow a fix for > GlobalKTable case). It's unclear to me atm, how this issue could be addressed > for KTables though. > Note, that user state stores are not affected, because they always have a > dedicated changelog topic (and don't reuse an input topic) and thus the > corrupted record would not be written into the changelog. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7976) Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[
https://issues.apache.org/jira/browse/KAFKA-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785456#comment-16785456
]
ASF GitHub Bot commented on KAFKA-7976:
---
rajinisivaram commented on pull request #6374: KAFKA-7976 - Fix
DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable
URL: https://github.com/apache/kafka/pull/6374
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
> ---
>
> Key: KAFKA-7976
> URL: https://issues.apache.org/jira/browse/KAFKA-7976
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/28/]
> {quote}java.lang.AssertionError: Unclean leader not elected
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable(DynamicBrokerReconfigurationTest.scala:488){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-8038) Flaky Test SslTransportLayerTest#testCloseSsl
[
https://issues.apache.org/jira/browse/KAFKA-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram reassigned KAFKA-8038:
-
Assignee: Rajini Sivaram
> Flaky Test SslTransportLayerTest#testCloseSsl
> -
>
> Key: KAFKA-8038
> URL: https://issues.apache.org/jira/browse/KAFKA-8038
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.network/SslTransportLayerTest/testCloseSsl/]
> {quote}java.lang.IllegalStateException: handshake is not completed at
> org.apache.kafka.common.network.SslTransportLayer.removeInterestOps(SslTransportLayer.java:770)
> at org.apache.kafka.common.network.KafkaChannel.mute(KafkaChannel.java:246)
> at org.apache.kafka.common.network.Selector.mute(Selector.java:685) at
> org.apache.kafka.common.network.Selector.muteAll(Selector.java:705) at
> org.apache.kafka.common.network.SslTransportLayerTest.testClose(SslTransportLayerTest.java:866)
> at
> org.apache.kafka.common.network.SslTransportLayerTest.testCloseSsl(SslTransportLayerTest.java:846){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8038) Flaky Test SslTransportLayerTest#testCloseSsl
[
https://issues.apache.org/jira/browse/KAFKA-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785463#comment-16785463
]
ASF GitHub Bot commented on KAFKA-8038:
---
rajinisivaram commented on pull request #6377: KAFKA-8038 - Fix timing issue in
SslTransportLayerTest.testCloseSsl
URL: https://github.com/apache/kafka/pull/6377
There is tiny timing window in the test where server has not marked the
channel ready, but client has. Wait for server to mark the channel ready.
I wasn't able to recreate the failure with or without the fix. The fix is
based on the stack trace in the JIRA.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test SslTransportLayerTest#testCloseSsl
> -
>
> Key: KAFKA-8038
> URL: https://issues.apache.org/jira/browse/KAFKA-8038
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.network/SslTransportLayerTest/testCloseSsl/]
> {quote}java.lang.IllegalStateException: handshake is not completed at
> org.apache.kafka.common.network.SslTransportLayer.removeInterestOps(SslTransportLayer.java:770)
> at org.apache.kafka.common.network.KafkaChannel.mute(KafkaChannel.java:246)
> at org.apache.kafka.common.network.Selector.mute(Selector.java:685) at
> org.apache.kafka.common.network.Selector.muteAll(Selector.java:705) at
> org.apache.kafka.common.network.SslTransportLayerTest.testClose(SslTransportLayerTest.java:866)
> at
> org.apache.kafka.common.network.SslTransportLayerTest.testCloseSsl(SslTransportLayerTest.java:846){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram reassigned KAFKA-7979: - Assignee: Rajini Sivaram > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785467#comment-16785467 ] Rajini Sivaram commented on KAFKA-7979: --- The stack trace shows that the threads are not deadlocked, but making progress, just slow. Will increase timeout in the test. > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785488#comment-16785488 ] ASF GitHub Bot commented on KAFKA-7979: --- rajinisivaram commented on pull request #6378: KAFKA-7979 - Clean up threads and increase timeout in PartitionTest URL: https://github.com/apache/kafka/pull/6378 Stack trace generated from the test failure shows that the test failed even though threads were runnable and making progress, indicating that the timeout may be too small when test machine is slow. Increasing timeout from 10 to 15 seconds, consistent with the default wait in other tests. Thread dump also showed a lot of left over threads from other tests, so added clean up of those as well. I wasn't able to recreate the failure with or without the fix. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8046) Shutdown broker because all log dirs in /tmp/kafka-logs have failed
[ https://issues.apache.org/jira/browse/KAFKA-8046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785495#comment-16785495 ] Edoardo Comar commented on KAFKA-8046: -- Looks like you're using /tmp as the disk store for Kafka. Are you sure there isn't some other process that cleans that location from files, i.e. something external to kafka deleted the checkpoint files ? > Shutdown broker because all log dirs in /tmp/kafka-logs have failed > --- > > Key: KAFKA-8046 > URL: https://issues.apache.org/jira/browse/KAFKA-8046 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 > Environment: centos 7 >Reporter: jaren >Priority: Major > > kafka stop working every few days.Here are some of logs. > ERROR Error while reading checkpoint file > /tmp/kafka-logs/cleaner-offset-checkpoint (kafka.server.LogDirFailureChannel) > java.io.FileNotFoundException: /tmp/kafka-logs/cleaner-offset-checkpoint (No > such file or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.(FileInputStream.java:138) > at > kafka.server.checkpoints.CheckpointFile.liftedTree2$1(CheckpointFile.scala:87) > at kafka.server.checkpoints.CheckpointFile.read(CheckpointFile.scala:86) > at > kafka.server.checkpoints.OffsetCheckpointFile.read(OffsetCheckpointFile.scala:61) > at > kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1$$anonfun$apply$1.apply(LogCleanerManager.scala:89) > at > kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1$$anonfun$apply$1.apply(LogCleanerManager.scala:87) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206) > at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104) > at > kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1.apply(LogCleanerManager.scala:87) > at > kafka.log.LogCleanerManager$$anonfun$allCleanerCheckpoints$1.apply(LogCleanerManager.scala:95) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at > kafka.log.LogCleanerManager.allCleanerCheckpoints(LogCleanerManager.scala:86) > at > kafka.log.LogCleanerManager$$anonfun$grabFilthiestCompactedLog$1.apply(LogCleanerManager.scala:126) > at > kafka.log.LogCleanerManager$$anonfun$grabFilthiestCompactedLog$1.apply(LogCleanerManager.scala:123) > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) > at > kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:123) > at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:296) > at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:289) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) > [2019-03-04 16:44:13,154] INFO [ReplicaManager broker=1] Stopping serving > replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager) > [2019-03-04 16:44:13,189] INFO [ReplicaFetcherManager on broker 1] Removed > fetcher for partitions > __consumer_offsets-22,FOTA_PLAIN_FORCESTOP-0,__consumer_offsets-30,OBSERVE_DEVICE- > > 0,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,FOTA_DOWNLOAD_ERROR-0,__consumer_offsets- > > 25,DEVICE_DE_REGISTER-0,__consumer_offsets-35,DEVICE_REG_UPDATE-0,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets- > > 16,__consumer_offsets-28,FOTA_IMEI_MONITOR-0,__consumer_offsets-31,__consumer_offsets-36,__consumer_offsets-42,FOTA_IMEI_MONITOR-1-0,__consumer_offsets-3,__consumer_offsets-18,DATA_TO_DEVICE- > > 0,__consumer_offsets-37,emq_notify-0,__consumer_offsets-15,__consumer_offsets-24,FOTA_PLAIN_MONITOR_FORCE-0,DEVICE_REGISTER-0,springCloudBus-0,__consumer_offsets-38,__consumer_offsets- > > 17,DEVICE_REP-0,__consumer_offsets-48,__consumer_offsets-19,__consumer_offsets-11,__consumer_offsets-13,__consumer_offsets-2,__consumer_offsets-43,__consumer_offsets-6,FOTA_STATICS_MONITOR-1- > > 0,__consumer_offsets-14,FOTA_STATICS_MONITOR-0,__consumer_offsets-20,__consumer_offsets-0,__consumer_offsets-44,__consumer_offsets-39,FOTA_STATE_CHANGE-0,__consumer_offsets-12,FOTA_UPGRADE_NOTIFY- > > 0,__consumer_offsets-45,__consumer_offsets-1,emq_message_down-0,__consumer_offsets-5,__consumer_offsets-26,__consumer_offsets-29,emq_me
[jira] [Commented] (KAFKA-7925) Constant 100% cpu usage by all kafka brokers
[
https://issues.apache.org/jira/browse/KAFKA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785591#comment-16785591
]
Rajini Sivaram commented on KAFKA-7925:
---
[~xabhi] Thanks for the logs, they were very useful. The logs show that
authentication succeeded, so the changes in the PR look ok. But there seems to
an issue for the clients working with the new broker. What version of
producers/consumers are you using? If you are able to use clients with the same
version as the broker, it will be good in terms of testing the PR. But we
should be able to use older clients as long as they are not using a very old
version of Kafka. Since there are no server-side errors logged at ERROR level,
it suggests that the server is failing client requests with error codes that
are not known to the client. The simplest way to figure out what the error was
would be turn on request logging on the broker. And then you can see the
request and responses in detail. In config/log4j.properties, change this line
to TRACE instead of WARN, and you should find the trace in kafka-request.log in
the log directory.
{code}
log4j.logger.kafka.request.logger=WARN, requestAppender
{code}
> Constant 100% cpu usage by all kafka brokers
>
>
> Key: KAFKA-7925
> URL: https://issues.apache.org/jira/browse/KAFKA-7925
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0, 2.1.1
> Environment: Java 11, Kafka v2.1.0, Kafka v2.1.1
>Reporter: Abhi
>Priority: Critical
> Attachments: jira-server.log-1, jira-server.log-2, jira-server.log-3,
> jira-server.log-4, jira-server.log-5, jira-server.log-6,
> jira_prod.producer.log, threadump20190212.txt
>
>
> Hi,
> I am seeing constant 100% cpu usage on all brokers in our kafka cluster even
> without any clients connected to any broker.
> This is a bug that we have seen multiple times in our kafka setup that is not
> yet open to clients. It is becoming a blocker for our deployment now.
> I am seeing lot of connections to other brokers in CLOSE_WAIT state (see
> below). In thread usage, I am seeing these threads
> 'kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-0,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-1,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2'
> taking up more than 90% of the cpu time in a 60s interval.
> I have attached a thread dump of one of the brokers in the cluster.
> *Java version:*
> openjdk 11.0.2 2019-01-15
> OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
> *Kafka verison:* v2.1.0
>
> *connections:*
> java 144319 kafkagod 88u IPv4 3063266 0t0 TCP *:35395 (LISTEN)
> java 144319 kafkagod 89u IPv4 3063267 0t0 TCP *:9144 (LISTEN)
> java 144319 kafkagod 104u IPv4 3064219 0t0 TCP
> mwkafka-prod-02.tbd:47292->mwkafka-zk-prod-05.tbd:2181 (ESTABLISHED)
> java 144319 kafkagod 2003u IPv4 3055115 0t0 TCP *:9092 (LISTEN)
> java 144319 kafkagod 2013u IPv4 7220110 0t0 TCP
> mwkafka-prod-02.tbd:60724->mwkafka-zk-prod-04.dr:2181 (ESTABLISHED)
> java 144319 kafkagod 2020u IPv4 30012904 0t0 TCP
> mwkafka-prod-02.tbd:38988->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2021u IPv4 30012961 0t0 TCP
> mwkafka-prod-02.tbd:58420->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2027u IPv4 30015723 0t0 TCP
> mwkafka-prod-02.tbd:58398->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2028u IPv4 30015630 0t0 TCP
> mwkafka-prod-02.tbd:36248->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2030u IPv4 30015726 0t0 TCP
> mwkafka-prod-02.tbd:39012->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2031u IPv4 30013619 0t0 TCP
> mwkafka-prod-02.tbd:38986->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2032u IPv4 30015604 0t0 TCP
> mwkafka-prod-02.tbd:36246->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2033u IPv4 30012981 0t0 TCP
> mwkafka-prod-02.tbd:36924->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2034u IPv4 30012967 0t0 TCP
> mwkafka-prod-02.tbd:39036->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2035u IPv4 30012898 0t0 TCP
> mwkafka-prod-02.tbd:36866->mwkafka-prod-01.dr:9092 (FIN_WAIT2)
> java 144319 kafkagod 2036u IPv4 30004729 0t0 TCP
> mwkafka-prod-02.tbd:36882->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2037u IPv4 30004914 0t0 TCP
> mwkafka-prod-02.tbd:58426->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2038u IPv4 30015651 0t0 TCP
> mwkafka-prod-02.tbd:36884->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2039u IPv4 30012966 0t0 TCP
> mwkafka-prod-02.tbd:58422->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319
[jira] [Resolved] (KAFKA-7976) Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[
https://issues.apache.org/jira/browse/KAFKA-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram resolved KAFKA-7976.
---
Resolution: Fixed
Reviewer: Manikumar
> Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
> ---
>
> Key: KAFKA-7976
> URL: https://issues.apache.org/jira/browse/KAFKA-7976
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/28/]
> {quote}java.lang.AssertionError: Unclean leader not elected
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable(DynamicBrokerReconfigurationTest.scala:488){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-8039) Flaky Test SaslAuthenticatorTest#testCannotReauthenticateAgainFasterThanOneSecond
[
https://issues.apache.org/jira/browse/KAFKA-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram reassigned KAFKA-8039:
-
Assignee: Rajini Sivaram
> Flaky Test
> SaslAuthenticatorTest#testCannotReauthenticateAgainFasterThanOneSecond
> -
>
> Key: KAFKA-8039
> URL: https://issues.apache.org/jira/browse/KAFKA-8039
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.security.authenticator/SaslAuthenticatorTest/testCannotReauthenticateAgainFasterThanOneSecond/]
> {quote}java.lang.AssertionError: Should have received the
> SaslHandshakeRequest bytes back since we re-authenticated too quickly, but
> instead we got our generated message echoed back, implying re-auth succeeded
> when it should not have at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testCannotReauthenticateAgainFasterThanOneSecond(SaslAuthenticatorTest.java:1503){quote}
> STDOUT
> {quote}[2019-03-04 19:33:46,222] ERROR Extensions provided in login context
> without a token
> (org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule:318)
> java.io.IOException: Extensions provided in login context without a token at
> org.apache.kafka.common.security.oauthbearer.internals.unsecured.OAuthBearerUnsecuredLoginCallbackHandler.handle(OAuthBearerUnsecuredLoginCallbackHandler.java:164)
> at
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.identifyToken(OAuthBearerLoginModule.java:316)
> at
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.login(OAuthBearerLoginModule.java:301)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
> javax.security.auth.login.LoginContext.login(LoginContext.java:587) at
> org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:61)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:104)
> at
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:149)
> at
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
> at
> org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85)
> at
> org.apache.kafka.common.network.NioEchoServer.(NioEchoServer.java:120)
> at
> org.apache.kafka.common.network.NioEchoServer.(NioEchoServer.java:96)
> at
> org.apache.kafka.common.network.NetworkTestUtils.createEchoServer(NetworkTestUtils.java:49)
> at
> org.apache.kafka.common.network.NetworkTestUtils.createEchoServer(NetworkTestUtils.java:43)
> at
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.createEchoServer(SaslAuthenticatorTest.java:1842)
> at
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.createEchoServer(SaslAuthenticatorTest.java:1838)
> at
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testValidSaslOauthBearerMechanismWithoutServerTokens(SaslAuthenticatorTest.java:1578)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners
[jira] [Assigned] (KAFKA-7980) Flaky Test SocketServerTest#testConnectionRateLimit
[
https://issues.apache.org/jira/browse/KAFKA-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram reassigned KAFKA-7980:
-
Assignee: Rajini Sivaram
> Flaky Test SocketServerTest#testConnectionRateLimit
> ---
>
> Key: KAFKA-7980
> URL: https://issues.apache.org/jira/browse/KAFKA-7980
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/]
> {quote}java.lang.AssertionError: Connections created too quickly: 4 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.network.SocketServerTest.testConnectionRateLimit(SocketServerTest.scala:1122){quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7925) Constant 100% cpu usage by all kafka brokers
[ https://issues.apache.org/jira/browse/KAFKA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785656#comment-16785656 ] Abhi commented on KAFKA-7925: - >> What version of producers/consumers are you using? I am using this client jar for producer and consumer - kafka_2.12-2.1.1/libs/kafka-clients-2.1.1.jar >>The simplest way to figure out what the error was would be turn on request >>logging on the broker. And then you can see the request and responses in >>detail. In config/log4j.properties, change this line to TRACE instead of >>WARN, and you should find the trace in kafka-request.log in the log directory. I will try this and update here soon. > Constant 100% cpu usage by all kafka brokers > > > Key: KAFKA-7925 > URL: https://issues.apache.org/jira/browse/KAFKA-7925 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.1.1 > Environment: Java 11, Kafka v2.1.0, Kafka v2.1.1 >Reporter: Abhi >Priority: Critical > Attachments: jira-server.log-1, jira-server.log-2, jira-server.log-3, > jira-server.log-4, jira-server.log-5, jira-server.log-6, > jira_prod.producer.log, threadump20190212.txt > > > Hi, > I am seeing constant 100% cpu usage on all brokers in our kafka cluster even > without any clients connected to any broker. > This is a bug that we have seen multiple times in our kafka setup that is not > yet open to clients. It is becoming a blocker for our deployment now. > I am seeing lot of connections to other brokers in CLOSE_WAIT state (see > below). In thread usage, I am seeing these threads > 'kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-0,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-1,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2' > taking up more than 90% of the cpu time in a 60s interval. > I have attached a thread dump of one of the brokers in the cluster. > *Java version:* > openjdk 11.0.2 2019-01-15 > OpenJDK Runtime Environment 18.9 (build 11.0.2+9) > OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode) > *Kafka verison:* v2.1.0 > > *connections:* > java 144319 kafkagod 88u IPv4 3063266 0t0 TCP *:35395 (LISTEN) > java 144319 kafkagod 89u IPv4 3063267 0t0 TCP *:9144 (LISTEN) > java 144319 kafkagod 104u IPv4 3064219 0t0 TCP > mwkafka-prod-02.tbd:47292->mwkafka-zk-prod-05.tbd:2181 (ESTABLISHED) > java 144319 kafkagod 2003u IPv4 3055115 0t0 TCP *:9092 (LISTEN) > java 144319 kafkagod 2013u IPv4 7220110 0t0 TCP > mwkafka-prod-02.tbd:60724->mwkafka-zk-prod-04.dr:2181 (ESTABLISHED) > java 144319 kafkagod 2020u IPv4 30012904 0t0 TCP > mwkafka-prod-02.tbd:38988->mwkafka-prod-02.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2021u IPv4 30012961 0t0 TCP > mwkafka-prod-02.tbd:58420->mwkafka-prod-01.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2027u IPv4 30015723 0t0 TCP > mwkafka-prod-02.tbd:58398->mwkafka-prod-01.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2028u IPv4 30015630 0t0 TCP > mwkafka-prod-02.tbd:36248->mwkafka-prod-02.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2030u IPv4 30015726 0t0 TCP > mwkafka-prod-02.tbd:39012->mwkafka-prod-02.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2031u IPv4 30013619 0t0 TCP > mwkafka-prod-02.tbd:38986->mwkafka-prod-02.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2032u IPv4 30015604 0t0 TCP > mwkafka-prod-02.tbd:36246->mwkafka-prod-02.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2033u IPv4 30012981 0t0 TCP > mwkafka-prod-02.tbd:36924->mwkafka-prod-01.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2034u IPv4 30012967 0t0 TCP > mwkafka-prod-02.tbd:39036->mwkafka-prod-02.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2035u IPv4 30012898 0t0 TCP > mwkafka-prod-02.tbd:36866->mwkafka-prod-01.dr:9092 (FIN_WAIT2) > java 144319 kafkagod 2036u IPv4 30004729 0t0 TCP > mwkafka-prod-02.tbd:36882->mwkafka-prod-01.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2037u IPv4 30004914 0t0 TCP > mwkafka-prod-02.tbd:58426->mwkafka-prod-01.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2038u IPv4 30015651 0t0 TCP > mwkafka-prod-02.tbd:36884->mwkafka-prod-01.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2039u IPv4 30012966 0t0 TCP > mwkafka-prod-02.tbd:58422->mwkafka-prod-01.nyc:9092 (ESTABLISHED) > java 144319 kafkagod 2040u IPv4 30005643 0t0 TCP > mwkafka-prod-02.tbd:36252->mwkafka-prod-02.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2041u IPv4 30012944 0t0 TCP > mwkafka-prod-02.tbd:36286->mwkafka-prod-02.dr:9092 (ESTABLISHED) > java 144319 kafkagod 2042u IPv4 30012973 0t0 TCP > mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51924 (ESTABLISHED) > java 144319 kafkagod 2043u sock 0,7 0t0 30012463 protocol: TCP > java 144319 kafkagod 2044u IPv4 30012979 0t0 TCP > mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39994 (ESTABLISHED) > j
[jira] [Commented] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785666#comment-16785666 ] ASF GitHub Bot commented on KAFKA-7979: --- rajinisivaram commented on pull request #6378: KAFKA-7979 - Clean up threads and increase timeout in PartitionTest URL: https://github.com/apache/kafka/pull/6378 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-7979. --- Resolution: Fixed Reviewer: Ismael Juma > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8052) Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH request
[
https://issues.apache.org/jira/browse/KAFKA-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartek Jakub updated KAFKA-8052:
Description:
I noticed in my logs some weird behavior. I see in logs intermittent log:
{noformat}
2019-03-06 14:02:13.024 INFO 1 --- [container-1-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-4,
groupId=service-main] Node 2 was unable to process the fetch request with
(sessionId=1321134604, epoch=125730): INVALID_FETCH_SESSION_EPOCH.{noformat}
which happens every ~1 hour.
I was wondering if it's my Kafka provider fault so I decided to investigate the
problem and I tried to reproduce the issue on my local - with success. My
configuration is:
* Kafka Clients version - 2.0.1
* Kafka - 2.12_2.1.0
I enabled trace logs for 'org.apache.kafka.clients' and that's what I get:
{noformat}
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=525)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.161 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=525,topics=[],forgotten_topics_data=[]}
with correlation id 629 to node 1001
2019-03-05 21:04:16.664 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Completed receive from node 1001 for FETCH with
correlation id 629, received
{throttle_time_ms=0,error_code=0,session_id=197970881,responses=[]}
2019-03-05 21:04:16.664 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Node 1001 sent an incremental fetch response for session
197970881 with response=(), implied=(itunes-command-19, itunes-command-18,
itunes-command-11, itunes-command-10, itunes-command-13, itunes-command-12,
itunes-command-15, itunes-command-14, itunes-command-17, itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=526)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.665 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - F630] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=526,topics=[],forgotten_topics_data=[]}
with correlation id 630 to node 1001
2019-03-05 21:04:17.152 DEBUG 3052 --- [ service-main]
o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3,
groupId=service-main - ???] Sending Heartbeat request to coordinator
localhost:9092 (id: 2147482646 rack: null)
2019-03-05 21:04:17.153 TRACE 3052 --- [ service-main]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - H631] Sending HEARTBEAT
{group_id=service-main,generation_id=108,member_id=consumer-3-7e3c718e-f472-4bd8-9404-458ae5ed9a26}
with correlation id 631 to node 2147482646
2019-03-05 21:04:17.254 TRACE 3052 --- [ service-main]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - ???] Completed receive from node 214
[jira] [Created] (KAFKA-8052) Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH request
Bartek Jakub created KAFKA-8052:
---
Summary: Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH
request
Key: KAFKA-8052
URL: https://issues.apache.org/jira/browse/KAFKA-8052
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 2.1.0
Reporter: Bartek Jakub
I noticed in my logs some weird symptom. I see in logs intermittent log:
{noformat}
2019-03-06 14:02:13.024 INFO 1 --- [container-1-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-4,
groupId=service-main] Node 2 was unable to process the fetch request with
(sessionId=1321134604, epoch=125730): INVALID_FETCH_SESSION_EPOCH.{noformat}
which happens every ~1 hour.
I was wondering if it's my Kafka provider fault so I decided to investigate the
problem and I tried to reproduce the issue on my local - with success. My
configuration is:
* Kafka Clients version - 2.0.1
* Kafka - 2.12_2.1.0
I enabled trace logs for 'org.apache.kafka.clients' and that's what I get:
{noformat}
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=525)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.161 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=525,topics=[],forgotten_topics_data=[]}
with correlation id 629 to node 1001
2019-03-05 21:04:16.664 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Completed receive from node 1001 for FETCH with
correlation id 629, received
{throttle_time_ms=0,error_code=0,session_id=197970881,responses=[]}
2019-03-05 21:04:16.664 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Node 1001 sent an incremental fetch response for session
197970881 with response=(), implied=(itunes-command-19, itunes-command-18,
itunes-command-11, itunes-command-10, itunes-command-13, itunes-command-12,
itunes-command-15, itunes-command-14, itunes-command-17, itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=526)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.665 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - F630] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=526,topics=[],forgotten_topics_data=[]}
with correlation id 630 to node 1001
2019-03-05 21:04:17.152 DEBUG 3052 --- [ service-main]
o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3,
groupId=service-main - ???] Sending Heartbeat request to coordinator
localhost:9092 (id: 2147482646 rack: null)
2019-03-05 21:04:17.153 TRACE 3052 --- [ service-main]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - H631] Sending HEARTBEAT
{group_id=service-main,generation_id=108,member_id=consumer-3-7e3c718e-f472-4bd8-9404-458ae5ed9a26}
with correlation id 631 to no
[jira] [Updated] (KAFKA-8052) Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH request
[
https://issues.apache.org/jira/browse/KAFKA-8052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bartek Jakub updated KAFKA-8052:
Description:
I noticed in my logs some weird behavior. I see in logs intermittent log:
{noformat}
2019-03-06 14:02:13.024 INFO 1 --- [container-1-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-4,
groupId=service-main] Node 2 was unable to process the fetch request with
(sessionId=1321134604, epoch=125730): INVALID_FETCH_SESSION_EPOCH.{noformat}
which happens every ~1 hour.
I was wondering if it's my Kafka provider fault so I decided to investigate the
problem and I tried to reproduce the issue on my local - with success. My
configuration is:
* Kafka Clients version - 2.0.1
* Kafka - 2.12_2.1.0
I enabled trace logs for 'org.apache.kafka.clients' and that's what I get:
{noformat}
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=525)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.161 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.161 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=525,topics=[],forgotten_topics_data=[]}
with correlation id 629 to node 1001
2019-03-05 21:04:16.664 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main] Completed receive from node 1001 for FETCH with
correlation id 629, received
{throttle_time_ms=0,error_code=0,session_id=197970881,responses=[]}
2019-03-05 21:04:16.664 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Node 1001 sent an incremental fetch response for session
197970881 with response=(), implied=(itunes-command-19, itunes-command-18,
itunes-command-11, itunes-command-10, itunes-command-13, itunes-command-12,
itunes-command-15, itunes-command-14, itunes-command-17, itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.kafka.clients.FetchSessionHandler : [Consumer clientId=consumer-3,
groupId=service-main] Built incremental fetch (sessionId=197970881, epoch=526)
for node 1001. Added (), altered (), removed () out of (itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)
2019-03-05 21:04:16.665 DEBUG 3052 --- [container-0-C-1]
o.a.k.c.consumer.internals.Fetcher : [Consumer clientId=consumer-3,
groupId=service-main] Sending READ_UNCOMMITTED
IncrementalFetchRequest(toSend=(), toForget=(), implied=(itunes-command-19,
itunes-command-18, itunes-command-11, itunes-command-10, itunes-command-13,
itunes-command-12, itunes-command-15, itunes-command-14, itunes-command-17,
itunes-command-16)) to broker localhost:9092 (id: 1001 rack: null)
2019-03-05 21:04:16.665 TRACE 3052 --- [container-0-C-1]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - F630] Sending FETCH
{replica_id=-1,max_wait_time=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=197970881,epoch=526,topics=[],forgotten_topics_data=[]}
with correlation id 630 to node 1001
2019-03-05 21:04:17.152 DEBUG 3052 --- [ service-main]
o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-3,
groupId=service-main - ???] Sending Heartbeat request to coordinator
localhost:9092 (id: 2147482646 rack: null)
2019-03-05 21:04:17.153 TRACE 3052 --- [ service-main]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - H631] Sending HEARTBEAT
{group_id=service-main,generation_id=108,member_id=consumer-3-7e3c718e-f472-4bd8-9404-458ae5ed9a26}
with correlation id 631 to node 2147482646
2019-03-05 21:04:17.254 TRACE 3052 --- [ service-main]
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-3,
groupId=service-main - ???] Completed receive from no
[jira] [Commented] (KAFKA-8044) System Test Failure: ReassignPartitionsTest.test_reassign_partitions
[
https://issues.apache.org/jira/browse/KAFKA-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785714#comment-16785714
]
ASF GitHub Bot commented on KAFKA-8044:
---
omkreddy commented on pull request #6380: KAFKA-8044: Increase memory for
VerifiableProducer in ReassignPartitionsTest
URL: https://github.com/apache/kafka/pull/6380
- Add support to configure delivery.timeout.ms verifiable_producer.py
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> System Test Failure: ReassignPartitionsTest.test_reassign_partitions
>
>
> Key: KAFKA-8044
> URL: https://issues.apache.org/jira/browse/KAFKA-8044
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 2.1.0
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Major
>
> {quote}
> Node ubuntu@worker10: did not stop within the specified timeout of 150
> seconds Traceback (most recent call last): File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
> line 132, in run data = self.run_test() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
> line 189, in run_test return self.test_context.function(self.test) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/mark/_mark.py",
> line 428, in wrapper return functools.partial(f, *args, **kwargs)(*w_args,
> **w_kwargs) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/tests/core/reassign_partitions_test.py",
> line 148, in test_reassign_partitions self.move_start_offset() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/tests/core/reassign_partitions_test.py",
> line 121, in move_start_offset producer.stop() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/services/background_thread.py",
> line 82, in stop super(BackgroundThreadService, self).stop() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/services/service.py",
> line 279, in stop self.stop_node(node) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/services/verifiable_producer.py",
> line 285, in stop_node (str(node.account), str(self.stop_timeout_sec))
> AssertionError: Node ubuntu@worker10: did not stop within the specified
> timeout of 150 seconds
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785719#comment-16785719
]
Randall Hauch commented on KAFKA-6755:
--
[~nimfadora], the KIP looks great, and I think you could start the discussion
thread. The only suggestion I would have at the moment is to make the
"Compatibility, Deprecation, and Migration Plan" section a bit easier to
understand by being more explicit; something like:
{quote}
The new configuration property in this proposal is backward compatible, so any
existing connector configurations that use the MaskField SMT will continue to
work as-is with no behavioral changes. To use the replacement feature, such
existing connector configurations will need to be modified to add a new
`replacement` property.
{quote}
Thanks for the contribution!
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7703) KafkaConsumer.position may return a wrong offset after "seekToEnd" is called
[ https://issues.apache.org/jira/browse/KAFKA-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785739#comment-16785739 ] Viktor Somogyi-Vass commented on KAFKA-7703: Hey [~nikita.glashenko], thanks for bumping this issue. I have a solution just need a day or two to polish it and create a PR. I'll let you know so you and folks on this issue and you can join the code review. Does that work for you? > KafkaConsumer.position may return a wrong offset after "seekToEnd" is called > > > Key: KAFKA-7703 > URL: https://issues.apache.org/jira/browse/KAFKA-7703 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0 >Reporter: Shixiong Zhu >Assignee: Viktor Somogyi-Vass >Priority: Major > > After "seekToEnd" is called, "KafkaConsumer.position" may return a wrong > offset set by another reset request. > Here is a reproducer: > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246 > In this reproducer, "poll(0)" will send an "earliest" request in background. > However, after "seekToEnd" is called, due to a race condition in > "Fetcher.resetOffsetIfNeeded" (It's not atomic, "seekToEnd" could happen > between the check > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R585 > and the seek > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R605), > "KafkaConsumer.position" may return an "earliest" offset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785761#comment-16785761
]
John Roesler commented on KAFKA-7965:
-
Saw this again on
https://github.com/apache/kafka/pull/6369#issuecomment-469885442
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7647) Flaky test LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic
[
https://issues.apache.org/jira/browse/KAFKA-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785766#comment-16785766
]
John Roesler commented on KAFKA-7647:
-
Saw this again on [https://github.com/apache/kafka/pull/6369]
{noformat}
kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic[0]
Error Message
java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), 1 -> (341,341),
6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> (342,342), 17 ->
(357,357), 12 -> (352,352), 7 -> (347,347), 3 -> (343,343), 18 -> (358,358), 16
-> (356,356), 11 -> (351,351), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344),
15 -> (355,355))> but was: (340,340), 88 -> (288,288), 5 -> (345,345),
10 -> (350,350), 14 -> (354,354), 93 -> (293,293), 84 -> (284,284), 89 ->
(289,289), 1 -> (341,341), 6 -> (346,346), 85 -> (285,285), 92 -> (292,292), 97
-> (297,297), 9 -> (349,349), 96 -> (296,296), 13 -> (353,353), 2 -> (342,342),
17 -> (357,357), 12 -> (352,352), 86 -> (286,286), 81 -> (281,281), 7 ->
(347,347), 98 -> (298,298), 91 -> (291,291), 3 -> (343,343), 80 -> (280,280),
18 -> (358,358), 95 -> (295,295), 16 -> (356,356), 11 -> (351,351), 99 ->
(299,299), 87 -> (287,287), 8 -> (348,348), 82 -> (282,282), 19 -> (359,359), 4
-> (344,344), 94 -> (294,294), 15 -> (355,355), 90 -> (290,290), 83 ->
(283,283))>
Stacktrace
java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), 1 -> (341,341),
6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> (342,342), 17 ->
(357,357), 12 -> (352,352), 7 -> (347,347), 3 -> (343,343), 18 -> (358,358), 16
-> (356,356), 11 -> (351,351), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344),
15 -> (355,355))> but was: (340,340), 88 -> (288,288), 5 -> (345,345),
10 -> (350,350), 14 -> (354,354), 93 -> (293,293), 84 -> (284,284), 89 ->
(289,289), 1 -> (341,341), 6 -> (346,346), 85 -> (285,285), 92 -> (292,292), 97
-> (297,297), 9 -> (349,349), 96 -> (296,296), 13 -> (353,353), 2 -> (342,342),
17 -> (357,357), 12 -> (352,352), 86 -> (286,286), 81 -> (281,281), 7 ->
(347,347), 98 -> (298,298), 91 -> (291,291), 3 -> (343,343), 80 -> (280,280),
18 -> (358,358), 95 -> (295,295), 16 -> (356,356), 11 -> (351,351), 99 ->
(299,299), 87 -> (287,287), 8 -> (348,348), 82 -> (282,282), 19 -> (359,359), 4
-> (344,344), 94 -> (294,294), 15 -> (355,355), 90 -> (290,290), 83 ->
(283,283))>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at
kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic(LogCleanerParameterizedIntegrationTest.scala:129)
Standard Output
[2019-03-05 23:11:05,309] ERROR [Log partition=log-0,
dir=/tmp/kafka-7516638606137040428] Could not find offset index file
corresponding to log file
/tmp/kafka-7516638606137040428/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:11:41,070] ERROR [Log partition=log-0,
dir=/tmp/kafka-2400954359557239394] Could not find offset index file
corresponding to log file
/tmp/kafka-2400954359557239394/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:12:19,134] ERROR [Log partition=log-0,
dir=/tmp/kafka-9465311811373934212] Could not find offset index file
corresponding to log file
/tmp/kafka-9465311811373934212/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:12:56,650] ERROR [Log partition=log-0,
dir=/tmp/kafka-6186993585168750935] Could not find offset index file
corresponding to log file
/tmp/kafka-6186993585168750935/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74){noformat}
> Flaky test
> LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic
> -
>
> Key: KAFKA-7647
> URL: https://issues.apache.org/jira/browse/KAFKA-7647
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Dong Lin
>Priority: Major
>
> {code}
> kafka.log.LogCleanerParameterizedIntegrationTest >
> testCleansCombinedCompactAndDeleteTopic[3] FAILED
> java.lang.AssertionError: Contents of the map shouldn't change
> expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 ->
> (354,354), 1 -> (341,341), 6 -> (346,346), 9 -> (349,349), 13 -> (353,353),
> 2 -> (342,342), 17 -> (357,357), 12 -> (352,352), 7 -> (347,347), 3 ->
> (343,343), 18 -> (358,358), 16 -> (356,356), 11 -> (351,351), 8 ->
> (348,348
[jira] [Commented] (KAFKA-7703) KafkaConsumer.position may return a wrong offset after "seekToEnd" is called
[ https://issues.apache.org/jira/browse/KAFKA-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785784#comment-16785784 ] Nikita Glashenko commented on KAFKA-7703: - [~viktorsomogyi], yes, thank you. > KafkaConsumer.position may return a wrong offset after "seekToEnd" is called > > > Key: KAFKA-7703 > URL: https://issues.apache.org/jira/browse/KAFKA-7703 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0 >Reporter: Shixiong Zhu >Assignee: Viktor Somogyi-Vass >Priority: Major > > After "seekToEnd" is called, "KafkaConsumer.position" may return a wrong > offset set by another reset request. > Here is a reproducer: > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246 > In this reproducer, "poll(0)" will send an "earliest" request in background. > However, after "seekToEnd" is called, due to a race condition in > "Fetcher.resetOffsetIfNeeded" (It's not atomic, "seekToEnd" could happen > between the check > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R585 > and the seek > https://github.com/zsxwing/kafka/commit/4e1aa11bfa99a38ac1e2cb0872c055db56b33246#diff-b45245913eaae46aa847d2615d62cde0R605), > "KafkaConsumer.position" may return an "earliest" offset. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7647) Flaky test LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic
[
https://issues.apache.org/jira/browse/KAFKA-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785766#comment-16785766
]
John Roesler edited comment on KAFKA-7647 at 3/6/19 3:42 PM:
-
Saw this again on [https://github.com/apache/kafka/pull/6369]
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2912/testReport/junit/kafka.log/LogCleanerParameterizedIntegrationTest/testCleansCombinedCompactAndDeleteTopic_0_/]
{noformat}
kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic[0]
Error Message
java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), 1 -> (341,341),
6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> (342,342), 17 ->
(357,357), 12 -> (352,352), 7 -> (347,347), 3 -> (343,343), 18 -> (358,358), 16
-> (356,356), 11 -> (351,351), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344),
15 -> (355,355))> but was: (340,340), 88 -> (288,288), 5 -> (345,345),
10 -> (350,350), 14 -> (354,354), 93 -> (293,293), 84 -> (284,284), 89 ->
(289,289), 1 -> (341,341), 6 -> (346,346), 85 -> (285,285), 92 -> (292,292), 97
-> (297,297), 9 -> (349,349), 96 -> (296,296), 13 -> (353,353), 2 -> (342,342),
17 -> (357,357), 12 -> (352,352), 86 -> (286,286), 81 -> (281,281), 7 ->
(347,347), 98 -> (298,298), 91 -> (291,291), 3 -> (343,343), 80 -> (280,280),
18 -> (358,358), 95 -> (295,295), 16 -> (356,356), 11 -> (351,351), 99 ->
(299,299), 87 -> (287,287), 8 -> (348,348), 82 -> (282,282), 19 -> (359,359), 4
-> (344,344), 94 -> (294,294), 15 -> (355,355), 90 -> (290,290), 83 ->
(283,283))>
Stacktrace
java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), 1 -> (341,341),
6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> (342,342), 17 ->
(357,357), 12 -> (352,352), 7 -> (347,347), 3 -> (343,343), 18 -> (358,358), 16
-> (356,356), 11 -> (351,351), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344),
15 -> (355,355))> but was: (340,340), 88 -> (288,288), 5 -> (345,345),
10 -> (350,350), 14 -> (354,354), 93 -> (293,293), 84 -> (284,284), 89 ->
(289,289), 1 -> (341,341), 6 -> (346,346), 85 -> (285,285), 92 -> (292,292), 97
-> (297,297), 9 -> (349,349), 96 -> (296,296), 13 -> (353,353), 2 -> (342,342),
17 -> (357,357), 12 -> (352,352), 86 -> (286,286), 81 -> (281,281), 7 ->
(347,347), 98 -> (298,298), 91 -> (291,291), 3 -> (343,343), 80 -> (280,280),
18 -> (358,358), 95 -> (295,295), 16 -> (356,356), 11 -> (351,351), 99 ->
(299,299), 87 -> (287,287), 8 -> (348,348), 82 -> (282,282), 19 -> (359,359), 4
-> (344,344), 94 -> (294,294), 15 -> (355,355), 90 -> (290,290), 83 ->
(283,283))>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:120)
at
kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic(LogCleanerParameterizedIntegrationTest.scala:129)
Standard Output
[2019-03-05 23:11:05,309] ERROR [Log partition=log-0,
dir=/tmp/kafka-7516638606137040428] Could not find offset index file
corresponding to log file
/tmp/kafka-7516638606137040428/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:11:41,070] ERROR [Log partition=log-0,
dir=/tmp/kafka-2400954359557239394] Could not find offset index file
corresponding to log file
/tmp/kafka-2400954359557239394/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:12:19,134] ERROR [Log partition=log-0,
dir=/tmp/kafka-9465311811373934212] Could not find offset index file
corresponding to log file
/tmp/kafka-9465311811373934212/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74)
[2019-03-05 23:12:56,650] ERROR [Log partition=log-0,
dir=/tmp/kafka-6186993585168750935] Could not find offset index file
corresponding to log file
/tmp/kafka-6186993585168750935/log-0/0300.log, recovering
segment and rebuilding index files... (kafka.log.Log:74){noformat}
was (Author: vvcephei):
Saw this again on [https://github.com/apache/kafka/pull/6369]
{noformat}
kafka.log.LogCleanerParameterizedIntegrationTest.testCleansCombinedCompactAndDeleteTopic[0]
Error Message
java.lang.AssertionError: Contents of the map shouldn't change expected: (340,340), 5 -> (345,345), 10 -> (350,350), 14 -> (354,354), 1 -> (341,341),
6 -> (346,346), 9 -> (349,349), 13 -> (353,353), 2 -> (342,342), 17 ->
(357,357), 12 -> (352,352), 7 -> (347,347), 3 -> (343,343), 18 -> (358,358), 16
-> (356,356), 11 -> (351,351), 8 -> (348,348), 19 -> (359,359), 4 -> (344,344),
15 -> (355,355))> but was: (340,340), 88 -> (288,288), 5 -> (345,345),
10 -> (35
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785771#comment-16785771
]
John Roesler commented on KAFKA-7965:
-
Saw this again on
[https://github.com/apache/kafka/pull/6369#issuecomment-470152793]
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/20024/testReport/junit/kafka.api/ConsumerBounceTest/testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup/]
{noformat}
kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
Error Message
java.lang.AssertionError: Received 0, expected at least 68
Stacktrace
java.lang.AssertionError: Received 0, expected at least 68
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at
kafka.api.ConsumerBounceTest.kafka$api$ConsumerBounceTest$$receiveAndCommit(ConsumerBounceTest.scala:562)
at
kafka.api.ConsumerBounceTest$$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$2.apply(ConsumerBounceTest.scala:325)
at
kafka.api.ConsumerBounceTest$$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$2.apply(ConsumerBounceTest.scala:324)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:324)
Standard Output
[2019-03-05 23:58:43,298] ERROR [ReplicaFetcher replicaId=1, leaderId=2,
fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,298] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition __consumer_offsets-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,526] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition topic-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,526] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
fetcherId=0] Error for partition topic-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition closetest-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
fetcherId=0] Error for partition closetest-0 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition closetest-3 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition closetest-7 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=2, leaderId=1,
fetcherId=0] Error for partition closetest-3 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,948] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition closetest-1 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,947] ERROR [ReplicaFetcher replicaId=0, leaderId=1,
fetcherId=0] Error for partition closetest-6 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition.
[2019-03-05 23:58:43,948] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=0] Error for partition closetest-4 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors
[jira] [Created] (KAFKA-8053) kafka-topics.sh gives confusing error message when the topic doesn't exist
Jakub Scholz created KAFKA-8053:
---
Summary: kafka-topics.sh gives confusing error message when the
topic doesn't exist
Key: KAFKA-8053
URL: https://issues.apache.org/jira/browse/KAFKA-8053
Project: Kafka
Issue Type: Bug
Reporter: Jakub Scholz
The kafka-topics.sh utility gives a confusing message when the topic it is
called with doesn't exist or when no topics exist at all:
{code}
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic xxx
Error while executing topic command : Topics in [] does not exist
[2019-03-06 13:26:33,982] ERROR java.lang.IllegalArgumentException: Topics in
[] does not exist
at
kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:416)
at
kafka.admin.TopicCommand$ZookeeperTopicService.describeTopic(TopicCommand.scala:332)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:66)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
(kafka.admin.TopicCommand$)
{code}
It tries to list the topics, but because list of topics is always empty, it
always prints just `[]`. The error message should be more useful and instead
list the topic passed by the user as the parameter or not try to list anything
at all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8053) kafka-topics.sh gives confusing error message when the topic doesn't exist
[
https://issues.apache.org/jira/browse/KAFKA-8053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785817#comment-16785817
]
ASF GitHub Bot commented on KAFKA-8053:
---
scholzj commented on pull request #6381: KAFKA-8053: Provide better error in
kafka-topics.sh message when topi…
URL: https://github.com/apache/kafka/pull/6381
…c doesn't exist
This PR tries to address
[KAFKA-8053](https://issues.apache.org/jira/browse/KAFKA-8053). It passes the
option with which the user originally called the `kafka-topics.sh` tool to the
`ensureTopicExists` method to try to produce better error message. It also
handles the case when the user doesn't specify any topics at all (which is for
example allowed for `--describe`).
If you prefer to address this differently, let me know and I can update the
PR.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> kafka-topics.sh gives confusing error message when the topic doesn't exist
> --
>
> Key: KAFKA-8053
> URL: https://issues.apache.org/jira/browse/KAFKA-8053
> Project: Kafka
> Issue Type: Bug
>Reporter: Jakub Scholz
>Priority: Minor
>
> The kafka-topics.sh utility gives a confusing message when the topic it is
> called with doesn't exist or when no topics exist at all:
> {code}
> bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic xxx
> Error while executing topic command : Topics in [] does not exist
> [2019-03-06 13:26:33,982] ERROR java.lang.IllegalArgumentException: Topics in
> [] does not exist
> at
> kafka.admin.TopicCommand$.kafka$admin$TopicCommand$$ensureTopicExists(TopicCommand.scala:416)
> at
> kafka.admin.TopicCommand$ZookeeperTopicService.describeTopic(TopicCommand.scala:332)
> at kafka.admin.TopicCommand$.main(TopicCommand.scala:66)
> at kafka.admin.TopicCommand.main(TopicCommand.scala)
> (kafka.admin.TopicCommand$)
> {code}
> It tries to list the topics, but because list of topics is always empty, it
> always prints just `[]`. The error message should be more useful and instead
> list the topic passed by the user as the parameter or not try to list
> anything at all.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7944) Add more natural Suppress test
[ https://issues.apache.org/jira/browse/KAFKA-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785852#comment-16785852 ] ASF GitHub Bot commented on KAFKA-7944: --- vvcephei commented on pull request #6382: KAFKA-7944: Improve Suppress test coverage URL: https://github.com/apache/kafka/pull/6382 * add a normal windowed suppress with short windows and a short grace period * improve the smoke test so that it actually verifies the intended conditions ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add more natural Suppress test > -- > > Key: KAFKA-7944 > URL: https://issues.apache.org/jira/browse/KAFKA-7944 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Priority: Major > > Can be integration or system test. > The idea is to test with tighter time bounds with windows of say 30 seconds > and use system time without adding any extra time for verification. > Currently, all the tests rely on artificially advancing system time, which > should be reliable, but you never know. So, we want to add a test that works > exactly like production code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7963) Extract hard-coded Streams metric name strings to centralized place
[ https://issues.apache.org/jira/browse/KAFKA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785885#comment-16785885 ] ASF GitHub Bot commented on KAFKA-7963: --- bbejeck commented on pull request #6355: KAFKA-7963: Extract hard-coded Streams metric name strings to centralized place URL: https://github.com/apache/kafka/pull/6355 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Extract hard-coded Streams metric name strings to centralized place > --- > > Key: KAFKA-7963 > URL: https://issues.apache.org/jira/browse/KAFKA-7963 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Sophie Blee-Goldman >Priority: Major > > Several string literals are hard-coded into the metrics, eg > "expired-window-record-drop" and "late-record-drop" in the window bytes > stores. These should be moved to a sensible central location, and widespread > string literals from these metrics may be causing memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7963) Extract hard-coded Streams metric name strings to centralized place
[ https://issues.apache.org/jira/browse/KAFKA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-7963: --- Affects Version/s: 2.3.0 > Extract hard-coded Streams metric name strings to centralized place > --- > > Key: KAFKA-7963 > URL: https://issues.apache.org/jira/browse/KAFKA-7963 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > Several string literals are hard-coded into the metrics, eg > "expired-window-record-drop" and "late-record-drop" in the window bytes > stores. These should be moved to a sensible central location, and widespread > string literals from these metrics may be causing memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7963) Extract hard-coded Streams metric name strings to centralized place
[ https://issues.apache.org/jira/browse/KAFKA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reassigned KAFKA-7963: -- Assignee: Sophie Blee-Goldman > Extract hard-coded Streams metric name strings to centralized place > --- > > Key: KAFKA-7963 > URL: https://issues.apache.org/jira/browse/KAFKA-7963 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > Several string literals are hard-coded into the metrics, eg > "expired-window-record-drop" and "late-record-drop" in the window bytes > stores. These should be moved to a sensible central location, and widespread > string literals from these metrics may be causing memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7963) Extract hard-coded Streams metric name strings to centralized place
[ https://issues.apache.org/jira/browse/KAFKA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7963. Resolution: Fixed > Extract hard-coded Streams metric name strings to centralized place > --- > > Key: KAFKA-7963 > URL: https://issues.apache.org/jira/browse/KAFKA-7963 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.3.0 > > > Several string literals are hard-coded into the metrics, eg > "expired-window-record-drop" and "late-record-drop" in the window bytes > stores. These should be moved to a sensible central location, and widespread > string literals from these metrics may be causing memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7963) Extract hard-coded Streams metric name strings to centralized place
[ https://issues.apache.org/jira/browse/KAFKA-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-7963: --- Fix Version/s: 2.3.0 > Extract hard-coded Streams metric name strings to centralized place > --- > > Key: KAFKA-7963 > URL: https://issues.apache.org/jira/browse/KAFKA-7963 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.3.0 > > > Several string literals are hard-coded into the metrics, eg > "expired-window-record-drop" and "late-record-drop" in the window bytes > stores. These should be moved to a sensible central location, and widespread > string literals from these metrics may be causing memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7944) Add more natural Suppress test
[ https://issues.apache.org/jira/browse/KAFKA-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler reassigned KAFKA-7944: --- Assignee: John Roesler > Add more natural Suppress test > -- > > Key: KAFKA-7944 > URL: https://issues.apache.org/jira/browse/KAFKA-7944 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > > Can be integration or system test. > The idea is to test with tighter time bounds with windows of say 30 seconds > and use system time without adding any extra time for verification. > Currently, all the tests rely on artificially advancing system time, which > should be reliable, but you never know. So, we want to add a test that works > exactly like production code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8054) KafkaProducer(Properties, Serializer, Serializer) fails when serializers are lambdas
Margaret Figura created KAFKA-8054:
--
Summary: KafkaProducer(Properties, Serializer, Serializer)
fails when serializers are lambdas
Key: KAFKA-8054
URL: https://issues.apache.org/jira/browse/KAFKA-8054
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 1.1.0
Reporter: Margaret Figura
I refactored some code that creates a KafkaProducer. Previously, it set
configuration parameters in a HashMap, but I changed it to use a Properties
instead. This calls a different constructor, and that constructor fails in some
cases.
In this example, we create a KafkaProducer with the Map constructor which
succeeds, and the Properties constructor which fails:
{code:java}
package test;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.Serializer;
public class KafkaProducerProperties
{
@FunctionalInterface
public interface KafkaSerializer extends Serializer
{
@Override
default public void configure(Map configs, boolean isKey){};
@Override
default public void close() {};
}
private final static KafkaSerializer stringSerializer = (topic,
key) -> {
return null; // TODO: do something :)
};
public static void main(String[] args)
{
HashMap hashConfig = new HashMap<>();
Properties propConfig = new Properties();
hashConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "1.2.3.4:1234");
propConfig.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "1.2.3.4:1234");
new KafkaProducer<>(hashConfig, stringSerializer,
stringSerializer).close(); // Works fine
new KafkaProducer<>(propConfig, stringSerializer,
stringSerializer).close(); // Throws ConfigException
}
}
{code}
The exception thrown from the Properties-based constructor is:
{noformat}
Exception in thread "main" org.apache.kafka.common.config.ConfigException:
Invalid value test.KafkaProducerProperties$$Lambda$1/1296064247 for
configuration key.serializer: Class
test.KafkaProducerProperties$$Lambda$1/1296064247 could not be found.
at
org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:724)
at
org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:469)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:462)
at
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
at
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75)
at
org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:365)
at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:318)
at test.KafkaProducerProperties.main(KafkaProducerProperties.java:36)
{noformat}
Note that the Properties constructor seems to work fine if I create a
Serializer instance in a 'normal way', but if it's implemented via a functional
interface it throws this error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8038) Flaky Test SslTransportLayerTest#testCloseSsl
[
https://issues.apache.org/jira/browse/KAFKA-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785899#comment-16785899
]
ASF GitHub Bot commented on KAFKA-8038:
---
rajinisivaram commented on pull request #6377: KAFKA-8038 - Fix timing issue in
SslTransportLayerTest.testCloseSsl
URL: https://github.com/apache/kafka/pull/6377
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test SslTransportLayerTest#testCloseSsl
> -
>
> Key: KAFKA-8038
> URL: https://issues.apache.org/jira/browse/KAFKA-8038
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.network/SslTransportLayerTest/testCloseSsl/]
> {quote}java.lang.IllegalStateException: handshake is not completed at
> org.apache.kafka.common.network.SslTransportLayer.removeInterestOps(SslTransportLayer.java:770)
> at org.apache.kafka.common.network.KafkaChannel.mute(KafkaChannel.java:246)
> at org.apache.kafka.common.network.Selector.mute(Selector.java:685) at
> org.apache.kafka.common.network.Selector.muteAll(Selector.java:705) at
> org.apache.kafka.common.network.SslTransportLayerTest.testClose(SslTransportLayerTest.java:866)
> at
> org.apache.kafka.common.network.SslTransportLayerTest.testCloseSsl(SslTransportLayerTest.java:846){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8003) Flaky Test TransactionsTest #testFencingOnTransactionExpiration
[
https://issues.apache.org/jira/browse/KAFKA-8003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785924#comment-16785924
]
Matthias J. Sax commented on KAFKA-8003:
Happend again:
https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/48/testReport/junit/kafka.api/TransactionsTest/testFencingOnTransactionExpiration/
> Flaky Test TransactionsTest #testFencingOnTransactionExpiration
> ---
>
> Key: KAFKA-8003
> URL: https://issues.apache.org/jira/browse/KAFKA-8003
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/34/]
> {quote}java.lang.AssertionError: expected:<1> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.TransactionsTest.testFencingOnTransactionExpiration(TransactionsTest.scala:510){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8055) Flaky Test LogCleanerParameterizedIntegrationTest#cleanerTest
Matthias J. Sax created KAFKA-8055:
--
Summary: Flaky Test
LogCleanerParameterizedIntegrationTest#cleanerTest
Key: KAFKA-8055
URL: https://issues.apache.org/jira/browse/KAFKA-8055
Project: Kafka
Issue Type: Bug
Components: core, unit tests
Affects Versions: 2.2.0
Reporter: Matthias J. Sax
Fix For: 2.2.1
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/49/]
{quote}java.lang.AssertionError: log cleaner should have processed up to offset
588, but lastCleaned=295 at org.junit.Assert.fail(Assert.java:88) at
org.junit.Assert.assertTrue(Assert.java:41) at
kafka.log.LogCleanerParameterizedIntegrationTest.checkLastCleaned(LogCleanerParameterizedIntegrationTest.scala:284)
at
kafka.log.LogCleanerParameterizedIntegrationTest.cleanerTest(LogCleanerParameterizedIntegrationTest.scala:77){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6824) Flaky Test DynamicBrokerReconfigurationTest#testAddRemoveSslListener
[
https://issues.apache.org/jira/browse/KAFKA-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785938#comment-16785938
]
Matthias J. Sax commented on KAFKA-6824:
One more on `2.2`:
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/49/testReport/junit/kafka.server/DynamicBrokerReconfigurationTest/testAddRemoveSaslListeners/]
{quote}java.util.concurrent.ExecutionException: java.lang.AssertionError:
expected:<0> but was:<1> at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:206) at
kafka.server.DynamicBrokerReconfigurationTest.verifyTimeout(DynamicBrokerReconfigurationTest.scala:1317)
at
kafka.server.DynamicBrokerReconfigurationTest.verifyConnectionFailure(DynamicBrokerReconfigurationTest.scala:1311)
at
kafka.server.DynamicBrokerReconfigurationTest.verifyRemoveListener(DynamicBrokerReconfigurationTest.scala:960)
at
kafka.server.DynamicBrokerReconfigurationTest.testAddRemoveSaslListeners(DynamicBrokerReconfigurationTest.scala:841){quote}
STDOUT
{quote}Completed Updating config for entity: brokers '0'. Completed Updating
config for entity: brokers '1'. Completed Updating config for entity: brokers
'2'. [2019-03-06 14:37:15,003] ERROR [ReplicaFetcher replicaId=0, leaderId=2,
fetcherId=1] Error for partition testtopic-2 at offset 0
(kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:15,003] ERROR
[ReplicaFetcher replicaId=0, leaderId=2, fetcherId=1] Error for partition
testtopic-8 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:15,041] ERROR
[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=1] Error for partition
testtopic-2 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:15,048] ERROR
[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=1] Error for partition
testtopic-8 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:15,074] ERROR
[ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error for partition
testtopic-5 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:15,074] ERROR
[ReplicaFetcher replicaId=1, leaderId=2, fetcherId=0] Error for partition
testtopic-5 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. Completed Updating config for entity:
brokers '0'. Completed Updating config for entity: brokers '1'. Completed
Updating config for entity: brokers '2'. [2019-03-06 14:37:28,893] ERROR
[ReplicaFetcher replicaId=2, leaderId=0, fetcherId=0] Error for partition
__consumer_offsets-3 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:28,899] ERROR
[ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Error for partition
__consumer_offsets-1 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:28,901] ERROR
[ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error for partition
__consumer_offsets-1 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:28,901] ERROR
[ReplicaFetcher replicaId=0, leaderId=1, fetcherId=1] Error for partition
__consumer_offsets-4 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:33,312] WARN Unable to
read additional data from client sessionid 0x10069a6e5f20004, likely client has
closed socket (org.apache.zookeeper.server.NIOServerCnxn:376) Completed
Updating config for entity: brokers '0'. Completed Updating config for entity:
brokers '1'. Completed Updating config for entity: brokers '2'. [2019-03-06
14:37:39,387] ERROR [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error
for partition testtopic-7 at offset 0 (kafka.server.ReplicaFetcherThread:76)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
does not host this topic-partition. [2019-03-06 14:37:39
[jira] [Commented] (KAFKA-7964) Flaky Test ConsumerBounceTest#testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
[
https://issues.apache.org/jira/browse/KAFKA-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785946#comment-16785946
]
Matthias J. Sax commented on KAFKA-7964:
One more:
[https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/44/]
> Flaky Test
> ConsumerBounceTest#testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
> --
>
> Key: KAFKA-7964
> URL: https://issues.apache.org/jira/browse/KAFKA-7964
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: expected:<100> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.ConsumerBounceTest.receiveExactRecords(ConsumerBounceTest.scala:551)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize$2(ConsumerBounceTest.scala:409)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize$2$adapted(ConsumerBounceTest.scala:408)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize(ConsumerBounceTest.scala:408){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7736) Consolidate Map usages in TransactionManager
[ https://issues.apache.org/jira/browse/KAFKA-7736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785945#comment-16785945 ] ASF GitHub Bot commented on KAFKA-7736: --- hachikuji commented on pull request #6270: KAFKA-7736: Consolidate Map usages in TransactionManager URL: https://github.com/apache/kafka/pull/6270 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Consolidate Map usages in TransactionManager > > > Key: KAFKA-7736 > URL: https://issues.apache.org/jira/browse/KAFKA-7736 > Project: Kafka > Issue Type: Task > Components: producer >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Minor > Labels: exactly-once > > There are a bunch of Map collections in TransactionManager > which could be consolidated into a single map to consolidate bookkeeping and > get rid of potential bugs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7939) Flaky Test KafkaAdminClientTest#testCreateTopicsRetryBackoff
[
https://issues.apache.org/jira/browse/KAFKA-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785922#comment-16785922
]
Matthias J. Sax commented on KAFKA-7939:
Happend again:
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/47/testReport/junit/org.apache.kafka.clients.admin/KafkaAdminClientTest/testCreateTopicsRetryBackoff/]
> Flaky Test KafkaAdminClientTest#testCreateTopicsRetryBackoff
>
>
> Key: KAFKA-7939
> URL: https://issues.apache.org/jira/browse/KAFKA-7939
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/12/]
> {quote}org.junit.runners.model.TestTimedOutException: test timed out after
> 12 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testCreateTopicsRetryBackoff(KafkaAdminClientTest.java:347){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785940#comment-16785940
]
Matthias J. Sax commented on KAFKA-7965:
Failed again (different error this time):
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/50/testReport/junit/kafka.api/ConsumerBounceTest/testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup/]
{quote}org.apache.kafka.common.KafkaException: Socket server failed to bind to
localhost:34632: Address already in use. at
kafka.network.Acceptor.openServerSocket(SocketServer.scala:573) at
kafka.network.Acceptor.(SocketServer.scala:451) at
kafka.network.SocketServer.createAcceptor(SocketServer.scala:245) at
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:215)
at
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:214)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:214)
at kafka.network.SocketServer.startup(SocketServer.scala:114) at
kafka.server.KafkaServer.startup(KafkaServer.scala:253) at
kafka.utils.TestUtils$.createServer(TestUtils.scala:139) at
kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$3(ConsumerBounceTest.scala:367)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at
kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$2(ConsumerBounceTest.scala:329)
at scala.util.control.Breaks.breakable(Breaks.scala:42) at
kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:329){quote}
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-8039) Flaky Test SaslAuthenticatorTest#testCannotReauthenticateAgainFasterThanOneSecond
[
https://issues.apache.org/jira/browse/KAFKA-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785972#comment-16785972
]
ASF GitHub Bot commented on KAFKA-8039:
---
rajinisivaram commented on pull request #6383: KAFKA-8039 - Use MockTime in
fast reauth test to avoid transient failures
URL: https://github.com/apache/kafka/pull/6383
Test uses 100ms as `connections.max.reauth.ms` and checks that a second
reauthentication doesn't occur within this interval. But since the interval is
small, we cannot guarantee that the time between the two checks is not higher
than the max interval. Change the test to use MockTime so that we can control
the time.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test
> SaslAuthenticatorTest#testCannotReauthenticateAgainFasterThanOneSecond
> -
>
> Key: KAFKA-8039
> URL: https://issues.apache.org/jira/browse/KAFKA-8039
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.security.authenticator/SaslAuthenticatorTest/testCannotReauthenticateAgainFasterThanOneSecond/]
> {quote}java.lang.AssertionError: Should have received the
> SaslHandshakeRequest bytes back since we re-authenticated too quickly, but
> instead we got our generated message echoed back, implying re-auth succeeded
> when it should not have at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testCannotReauthenticateAgainFasterThanOneSecond(SaslAuthenticatorTest.java:1503){quote}
> STDOUT
> {quote}[2019-03-04 19:33:46,222] ERROR Extensions provided in login context
> without a token
> (org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule:318)
> java.io.IOException: Extensions provided in login context without a token at
> org.apache.kafka.common.security.oauthbearer.internals.unsecured.OAuthBearerUnsecuredLoginCallbackHandler.handle(OAuthBearerUnsecuredLoginCallbackHandler.java:164)
> at
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.identifyToken(OAuthBearerLoginModule.java:316)
> at
> org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule.login(OAuthBearerLoginModule.java:301)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at
> javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at
> javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at
> javax.security.auth.login.LoginContext.login(LoginContext.java:587) at
> org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:61)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:104)
> at
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:149)
> at
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
> at
> org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:85)
> at
> org.apache.kafka.common.network.NioEchoServer.(NioEchoServer.java:120)
> at
> org.apache.kafka.common.network.NioEchoServer.(NioEchoServer.java:96)
> at
> org.apache.kafka.common.network.NetworkTestUtils.createEchoServer(NetworkTestUtils.java:49)
> at
> org.apache.kafka.common.network.NetworkTestUtils.createEchoServer(Ne
[jira] [Created] (KAFKA-8056) Replace FindCoordinator request/response with automated protocol
Mickael Maison created KAFKA-8056: - Summary: Replace FindCoordinator request/response with automated protocol Key: KAFKA-8056 URL: https://issues.apache.org/jira/browse/KAFKA-8056 Project: Kafka Issue Type: Sub-task Reporter: Mickael Maison Assignee: Mickael Maison -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8020) Consider changing design of ThreadCache
[
https://issues.apache.org/jira/browse/KAFKA-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786002#comment-16786002
]
Matthias J. Sax commented on KAFKA-8020:
[~Yohan123] Can you elaborate a little bit more? I don't see why we would need
to change anything atm – I guess, I don't understand the proposal yet. Also, do
you suggest to change the policy from LRU to time-base for all caches? Or would
the user be able to pick? In any case, even if this might not require a public
API change, I am wondering if we should consider it a major change and do a
KIP? However, I would like to understand the proposal better first, before we
can make a decision.
> Consider changing design of ThreadCache
>
>
> Key: KAFKA-8020
> URL: https://issues.apache.org/jira/browse/KAFKA-8020
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Richard Yu
>Priority: Major
>
> In distributed systems, time-aware LRU Caches offers a superior eviction
> policy better than traditional LRU models, having more cache hits than
> misses. In this new policy, if an item is stored beyond its useful lifespan,
> then it is removed. For example, in {{CachingWindowStore}}, a window usually
> is of limited size. After it expires, it would no longer be queried for, but
> it potentially could stay in the ThreadCache for an unnecessary amount of
> time if it is not evicted (i.e. the number of entries being inserted is few).
> For better allocation of memory, it would be better if we implement a
> time-aware LRU Cache which takes into account the lifespan of an entry and
> removes it once it has expired.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8057) Producer (and probably consumer) retains IP address of broker instead of resolving
Bolke de Bruin created KAFKA-8057: - Summary: Producer (and probably consumer) retains IP address of broker instead of resolving Key: KAFKA-8057 URL: https://issues.apache.org/jira/browse/KAFKA-8057 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 2.1.1 Reporter: Bolke de Bruin Producers, contrary to brokers, retain the IPs of brokers instead of resolving the Hostname again. This stops producers from connecting to the brokers. To reproduce have a producer connect to brokers, bring down the brokers and bring them up with different ip but with same hostname (DNS updated, eg. On kubernetes) Producers will start logging “Broker maybe down”, but will never reconnect as the IP address in the log message will keep on referencing the old IP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8006) Guard calls to init() and close() within user code
[ https://issues.apache.org/jira/browse/KAFKA-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786018#comment-16786018 ] ASF GitHub Bot commented on KAFKA-8006: --- mjsax commented on pull request #6353: KAFKA-8006: Guard calls to init and close from global processor URL: https://github.com/apache/kafka/pull/6353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Guard calls to init() and close() within user code > -- > > Key: KAFKA-8006 > URL: https://issues.apache.org/jira/browse/KAFKA-8006 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Major > Labels: beginner, newbie > > When using a global state store within the provided Processor, the user is > able to call init() and close() on the state stores. Those APIs should only > be called by Kafka Streams runtime but not by the user. > We should guard those APIs so that the user cannot call them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8006) Guard calls to init() and close() within user code
[ https://issues.apache.org/jira/browse/KAFKA-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-8006. Resolution: Fixed Assignee: Sophie Blee-Goldman Fix Version/s: 2.3.0 > Guard calls to init() and close() within user code > -- > > Key: KAFKA-8006 > URL: https://issues.apache.org/jira/browse/KAFKA-8006 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Assignee: Sophie Blee-Goldman >Priority: Major > Labels: beginner, newbie > Fix For: 2.3.0 > > > When using a global state store within the provided Processor, the user is > able to call init() and close() on the state stores. Those APIs should only > be called by Kafka Streams runtime but not by the user. > We should guard those APIs so that the user cannot call them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7928) Deprecate WindowStore.put(key, value)
[ https://issues.apache.org/jira/browse/KAFKA-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-7928: -- Assignee: (was: Lee Dongjin) > Deprecate WindowStore.put(key, value) > - > > Key: KAFKA-7928 > URL: https://issues.apache.org/jira/browse/KAFKA-7928 > Project: Kafka > Issue Type: Improvement >Reporter: John Roesler >Priority: Major > Labels: beginner, easy-fix, needs-kip, newbie > > Specifically, `org.apache.kafka.streams.state.WindowStore#put(K, V)` > This method is strange... A window store needs to have a timestamp associated > with the key, so if you do a put without a timestamp, it's up to the store to > just make one up. > Even the javadoc on the method recommends not to use it, due to this > confusing behavior. > We should just deprecate it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8007) In-memory window store copies on fetch
[ https://issues.apache.org/jira/browse/KAFKA-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786025#comment-16786025 ] ASF GitHub Bot commented on KAFKA-8007: --- bbejeck commented on pull request #6335: KAFKA-8007: Avoid copying on fetch in InMemoryWindowStore URL: https://github.com/apache/kafka/pull/6335 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > In-memory window store copies on fetch > -- > > Key: KAFKA-8007 > URL: https://issues.apache.org/jira/browse/KAFKA-8007 > Project: Kafka > Issue Type: Improvement >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > The current implementation of InMemoryWindowStore copies all record data into > a new list and returns an iterator over that list when fetching. That is > inefficient in terms of both memory and time. The fetching logic should be > moved to the iterators, which could be cleaned up as well (currently they > rely on ListIterator to provide peekNextKey() functionality, which is > unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8007) In-memory window store copies on fetch
[ https://issues.apache.org/jira/browse/KAFKA-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-8007: --- Fix Version/s: 2.3.0 > In-memory window store copies on fetch > -- > > Key: KAFKA-8007 > URL: https://issues.apache.org/jira/browse/KAFKA-8007 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > Fix For: 2.3.0 > > > The current implementation of InMemoryWindowStore copies all record data into > a new list and returns an iterator over that list when fetching. That is > inefficient in terms of both memory and time. The fetching logic should be > moved to the iterators, which could be cleaned up as well (currently they > rely on ListIterator to provide peekNextKey() functionality, which is > unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8007) In-memory window store copies on fetch
[ https://issues.apache.org/jira/browse/KAFKA-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-8007: --- Affects Version/s: 2.3.0 > In-memory window store copies on fetch > -- > > Key: KAFKA-8007 > URL: https://issues.apache.org/jira/browse/KAFKA-8007 > Project: Kafka > Issue Type: Improvement >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > The current implementation of InMemoryWindowStore copies all record data into > a new list and returns an iterator over that list when fetching. That is > inefficient in terms of both memory and time. The fetching logic should be > moved to the iterators, which could be cleaned up as well (currently they > rely on ListIterator to provide peekNextKey() functionality, which is > unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-8007) In-memory window store copies on fetch
[ https://issues.apache.org/jira/browse/KAFKA-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-8007: --- Component/s: streams > In-memory window store copies on fetch > -- > > Key: KAFKA-8007 > URL: https://issues.apache.org/jira/browse/KAFKA-8007 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > The current implementation of InMemoryWindowStore copies all record data into > a new list and returns an iterator over that list when fetching. That is > inefficient in terms of both memory and time. The fetching logic should be > moved to the iterators, which could be cleaned up as well (currently they > rely on ListIterator to provide peekNextKey() functionality, which is > unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8038) Flaky Test SslTransportLayerTest#testCloseSsl
[
https://issues.apache.org/jira/browse/KAFKA-8038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram resolved KAFKA-8038.
---
Resolution: Fixed
> Flaky Test SslTransportLayerTest#testCloseSsl
> -
>
> Key: KAFKA-8038
> URL: https://issues.apache.org/jira/browse/KAFKA-8038
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Rajini Sivaram
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/46/testReport/junit/org.apache.kafka.common.network/SslTransportLayerTest/testCloseSsl/]
> {quote}java.lang.IllegalStateException: handshake is not completed at
> org.apache.kafka.common.network.SslTransportLayer.removeInterestOps(SslTransportLayer.java:770)
> at org.apache.kafka.common.network.KafkaChannel.mute(KafkaChannel.java:246)
> at org.apache.kafka.common.network.Selector.mute(Selector.java:685) at
> org.apache.kafka.common.network.Selector.muteAll(Selector.java:705) at
> org.apache.kafka.common.network.SslTransportLayerTest.testClose(SslTransportLayerTest.java:866)
> at
> org.apache.kafka.common.network.SslTransportLayerTest.testCloseSsl(SslTransportLayerTest.java:846){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8057) Producer (and probably consumer) retains IP address of broker instead of resolving
[ https://issues.apache.org/jira/browse/KAFKA-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated KAFKA-8057: -- Affects Version/s: (was: 2.1.1) 2.1.0 > Producer (and probably consumer) retains IP address of broker instead of > resolving > -- > > Key: KAFKA-8057 > URL: https://issues.apache.org/jira/browse/KAFKA-8057 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 2.1.0 >Reporter: Bolke de Bruin >Priority: Critical > > Producers, contrary to brokers, retain the IPs of brokers instead of > resolving the Hostname again. This stops producers from connecting to the > brokers. > To reproduce have a producer connect to brokers, bring down the brokers and > bring them up with different ip but with same hostname (DNS updated, eg. On > kubernetes) Producers will start logging “Broker maybe down”, but will never > reconnect as the IP address in the log message will keep on referencing the > old IP. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8044) System Test Failure: ReassignPartitionsTest.test_reassign_partitions
[
https://issues.apache.org/jira/browse/KAFKA-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786107#comment-16786107
]
ASF GitHub Bot commented on KAFKA-8044:
---
omkreddy commented on pull request #6380: KAFKA-8044: Add timeout to
VerifiableProducer close to avoid indefinite blocking
URL: https://github.com/apache/kafka/pull/6380
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> System Test Failure: ReassignPartitionsTest.test_reassign_partitions
>
>
> Key: KAFKA-8044
> URL: https://issues.apache.org/jira/browse/KAFKA-8044
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 2.1.0
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Major
>
> {quote}
> Node ubuntu@worker10: did not stop within the specified timeout of 150
> seconds Traceback (most recent call last): File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
> line 132, in run data = self.run_test() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/tests/runner_client.py",
> line 189, in run_test return self.test_context.function(self.test) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/mark/_mark.py",
> line 428, in wrapper return functools.partial(f, *args, **kwargs)(*w_args,
> **w_kwargs) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/tests/core/reassign_partitions_test.py",
> line 148, in test_reassign_partitions self.move_start_offset() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/tests/core/reassign_partitions_test.py",
> line 121, in move_start_offset producer.stop() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/services/background_thread.py",
> line 82, in stop super(BackgroundThreadService, self).stop() File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/venv/lib/python2.7/site-packages/ducktape-0.7.5-py2.7.egg/ducktape/services/service.py",
> line 279, in stop self.stop_node(node) File
> "/home/jenkins/workspace/system-test-kafka_2.2-WCMO537TEDQFHHUF4SVK33SPUR5AG4KSITCCNJIYFPKDZQWBEJOA/kafka/tests/kafkatest/services/verifiable_producer.py",
> line 285, in stop_node (str(node.account), str(self.stop_timeout_sec))
> AssertionError: Node ubuntu@worker10: did not stop within the specified
> timeout of 150 seconds
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-8057) Producer (and probably consumer) retains IP address of broker instead of resolving
[ https://issues.apache.org/jira/browse/KAFKA-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved KAFKA-8057. --- Resolution: Fixed Fix Version/s: 2.1.1 > Producer (and probably consumer) retains IP address of broker instead of > resolving > -- > > Key: KAFKA-8057 > URL: https://issues.apache.org/jira/browse/KAFKA-8057 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 2.1.0 >Reporter: Bolke de Bruin >Priority: Critical > Fix For: 2.1.1 > > > Producers, contrary to brokers, retain the IPs of brokers instead of > resolving the Hostname again. This stops producers from connecting to the > brokers. > To reproduce have a producer connect to brokers, bring down the brokers and > bring them up with different ip but with same hostname (DNS updated, eg. On > kubernetes) Producers will start logging “Broker maybe down”, but will never > reconnect as the IP address in the log message will keep on referencing the > old IP. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786188#comment-16786188
]
Matthias J. Sax commented on KAFKA-7965:
One more with the "Boxed Error":
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/341/tests]
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format
[ https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786194#comment-16786194 ] ASF GitHub Bot commented on KAFKA-3522: --- mjsax commented on pull request #6175: KAFKA-3522: Add public interfaces for timestamped stores URL: https://github.com/apache/kafka/pull/6175 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Consider adding version information into rocksDB storage format > --- > > Key: KAFKA-3522 > URL: https://issues.apache.org/jira/browse/KAFKA-3522 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Guozhang Wang >Assignee: Matthias J. Sax >Priority: Major > Labels: architecture > > Kafka Streams does not introduce any modifications to the data format in the > underlying Kafka protocol, but it does use RocksDB for persistent state > storage, and currently its data format is fixed and hard-coded. We want to > consider the evolution path in the future we we change the data format, and > hence having some version info stored along with the storage file / directory > would be useful. > And this information could be even out of the storage file; for example, we > can just use a small "version indicator" file in the rocksdb directory for > this purposes. Thoughts? [~enothereska] [~jkreps] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8058) ConnectRestExtensionContext does not return list of connectors from its ConnectClusterState
Chris Egerton created KAFKA-8058: Summary: ConnectRestExtensionContext does not return list of connectors from its ConnectClusterState Key: KAFKA-8058 URL: https://issues.apache.org/jira/browse/KAFKA-8058 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 2.1.1, 2.1.0, 2.0.1, 2.0.0 Reporter: Chris Egerton Assignee: Chris Egerton Title is not entirely accurate, but describes the most common way this bug presents itself. The [ConnectClusterStateImpl|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/health/ConnectClusterStateImpl.java] used by Kafka Connect to provide cluster information to REST extensions makes an asynchronous call to [Herder.connectors(...)|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java#L70] in its [connectors()|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/health/ConnectClusterStateImpl.java#L46] method that populates a list of connector names, then immediately returns that list of connector names. If the list is read soon after it is returned, it's possible it will be empty or at least not completely populated. Since the method is implicitly synchronous (no callback logic is involved in its parameters or return values, and there is no documentation indicating asynchronous behavior for the [ConnectClusterState.connectors()|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/api/src/main/java/org/apache/kafka/connect/health/ConnectClusterState.java#L28-L34] method), this should be remedied by adding a wait for the Herder.connectors(...) method to complete in the ConnectClusterStateImpl.connectors() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6020) Broker side filtering
[ https://issues.apache.org/jira/browse/KAFKA-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786242#comment-16786242 ] Yiming Zang commented on KAFKA-6020: Any updates for this? We have smilier needs on our side, strongly support this idea on broker-side filtering. Our use case comes from N-DC replication. Basically imagine if you have 5 data centers and you need to replicate data to everywhere, typically you'll have to run N*(N-1) which is 20 mirror-maker jobs in order replicate messages in each local data center to all remote data centers. Each mirror maker will have to read the whole 5 copies of events, do some processing and only replicate one fifth of the events. This is a huge waste of network bandwidth and cpu resources. If we can have a way to pre filter the events on broker side, mirror maker doesn't need to read all 5 copies of events any more, which can be a huge amount of savings when we have even more data centers in the future. > Broker side filtering > - > > Key: KAFKA-6020 > URL: https://issues.apache.org/jira/browse/KAFKA-6020 > Project: Kafka > Issue Type: New Feature > Components: consumer >Reporter: Pavel Micka >Priority: Major > Labels: needs-kip > > Currently, it is not possible to filter messages on broker side. Filtering > messages on broker side is convenient for filter with very low selectivity > (one message in few thousands). In my case it means to transfer several GB of > data to consumer, throw it away, take one message and do it again... > While I understand that filtering by message body is not feasible (for > performance reasons), I propose to filter just by message key prefix. This > can be achieved even without any deserialization, as the prefix to be matched > can be passed as an array (hence the broker would do just array prefix > compare). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8058) ConnectRestExtensionContext does not return list of connectors from its ConnectClusterState
[ https://issues.apache.org/jira/browse/KAFKA-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786245#comment-16786245 ] ASF GitHub Bot commented on KAFKA-8058: --- C0urante commented on pull request #6384: KAFKA-8058: Fix ConnectClusterStateImpl.connectors() method URL: https://github.com/apache/kafka/pull/6384 This makes the `ConnectClusterStateImpl.connectors()` method synchronous, whereas before it was implicitly asynchronous with no way to tell whether it had completed or not. More detail can be found in the [Jira](https://issues.apache.org/jira/browse/KAFKA-8058). Tested manually. Seems like a light enough change that even unit tests would be overkill, but if reviewers feel differently tests can be added. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > ConnectRestExtensionContext does not return list of connectors from its > ConnectClusterState > --- > > Key: KAFKA-8058 > URL: https://issues.apache.org/jira/browse/KAFKA-8058 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Major > > Title is not entirely accurate, but describes the most common way this bug > presents itself. > The > [ConnectClusterStateImpl|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/health/ConnectClusterStateImpl.java] > used by Kafka Connect to provide cluster information to REST extensions > makes an asynchronous call to > [Herder.connectors(...)|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Herder.java#L70] > in its > [connectors()|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/health/ConnectClusterStateImpl.java#L46] > method that populates a list of connector names, then immediately returns > that list of connector names. If the list is read soon after it is returned, > it's possible it will be empty or at least not completely populated. > Since the method is implicitly synchronous (no callback logic is involved in > its parameters or return values, and there is no documentation indicating > asynchronous behavior for the > [ConnectClusterState.connectors()|https://github.com/apache/kafka/blob/ab00c51b6362b41071ae32611ebf698ba9c221ee/connect/api/src/main/java/org/apache/kafka/connect/health/ConnectClusterState.java#L28-L34] > method), this should be remedied by adding a wait for the > Herder.connectors(...) method to complete in the > ConnectClusterStateImpl.connectors() method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8059) Flaky Test DynamicConnectionQuotaTest #testDynamicConnectionQuota
Matthias J. Sax created KAFKA-8059:
--
Summary: Flaky Test DynamicConnectionQuotaTest
#testDynamicConnectionQuota
Key: KAFKA-8059
URL: https://issues.apache.org/jira/browse/KAFKA-8059
Project: Kafka
Issue Type: Bug
Components: core, unit tests
Affects Versions: 2.2.0
Reporter: Matthias J. Sax
Fix For: 2.2.1
[https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/46/tests]
{quote}org.scalatest.junit.JUnitTestFailedError: Expected exception
java.io.IOException to be thrown, but no exception was thrown
at
org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:100)
at
org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71)
at org.scalatest.Assertions$class.intercept(Assertions.scala:822)
at org.scalatest.junit.JUnitSuite.intercept(JUnitSuite.scala:71)
at
kafka.network.DynamicConnectionQuotaTest.testDynamicConnectionQuota(DynamicConnectionQuotaTest.scala:82){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format
[ https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786302#comment-16786302 ] ASF GitHub Bot commented on KAFKA-3522: --- mjsax commented on pull request #5422: [WIP: DO NOT MERGE] KAFKA-3522: Enable StateStore upgrade path URL: https://github.com/apache/kafka/pull/5422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Consider adding version information into rocksDB storage format > --- > > Key: KAFKA-3522 > URL: https://issues.apache.org/jira/browse/KAFKA-3522 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Guozhang Wang >Assignee: Matthias J. Sax >Priority: Major > Labels: architecture > > Kafka Streams does not introduce any modifications to the data format in the > underlying Kafka protocol, but it does use RocksDB for persistent state > storage, and currently its data format is fixed and hard-coded. We want to > consider the evolution path in the future we we change the data format, and > hence having some version info stored along with the storage file / directory > would be useful. > And this information could be even out of the storage file; for example, we > can just use a small "version indicator" file in the rocksdb directory for > this purposes. Thoughts? [~enothereska] [~jkreps] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format
[ https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786301#comment-16786301 ] ASF GitHub Bot commented on KAFKA-3522: --- mjsax commented on pull request #6044: [WIP: DO NOT MERGE] KAFKA-3522: Allow storing timestamps in RocksDB URL: https://github.com/apache/kafka/pull/6044 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Consider adding version information into rocksDB storage format > --- > > Key: KAFKA-3522 > URL: https://issues.apache.org/jira/browse/KAFKA-3522 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Guozhang Wang >Assignee: Matthias J. Sax >Priority: Major > Labels: architecture > > Kafka Streams does not introduce any modifications to the data format in the > underlying Kafka protocol, but it does use RocksDB for persistent state > storage, and currently its data format is fixed and hard-coded. We want to > consider the evolution path in the future we we change the data format, and > hence having some version info stored along with the storage file / directory > would be useful. > And this information could be even out of the storage file; for example, we > can just use a small "version indicator" file in the rocksdb directory for > this purposes. Thoughts? [~enothereska] [~jkreps] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
