[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14803215#comment-14803215 ]
Mayuresh Gharat edited comment on KAFKA-2120 at 9/17/15 5:01 PM: ----------------------------------------------------------------- I checked if the failures are related to this patch and I found that the failures occur with commits earlier to this patch. Here is what I did: 1) Checked out the commit : commit b658e25207174578c26ea94a75e5b0ea10f06ba5 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: Mon Sep 14 15:02:32 2015 -0700 2) This is one commit before this patch. I ran ./gradlew test. 3) Got following errors, in different runs : *RUN 1 :* kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED java.lang.AssertionError: Failed to update high watermark for follower after timeout at org.junit.Assert.fail(Assert.java:88) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:654) at kafka.server.LogRecoveryTest.testHWCheckpointNoFailuresMultipleLogSegments(LogRecoveryTest.scala:173) kafka.api.SSLConsumerTest > testGroupConsumption FAILED java.lang.IllegalStateException: Failed to consume the expected records after 301 iterations. at kafka.api.SSLConsumerTest.consumeRecords(SSLConsumerTest.scala:226) at kafka.api.SSLConsumerTest.testGroupConsumption(SSLConsumerTest.scala:165) kafka.integration.UncleanLeaderElectionTest > testCleanLeaderElectionDisabledByTopicOverride FAILED java.lang.AssertionError: expected:<List(first, second, third)> but was:<List()> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:253) at kafka.integration.UncleanLeaderElectionTest.testCleanLeaderElectionDisabledByTopicOverride(UncleanLeaderElectionTest.scala:155) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:132) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. kafka.api.ProducerBounceTest > testBrokerFailure FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:110) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. *RUN 2 :* kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED java.lang.AssertionError: Failed to update high watermark for follower after timeout at org.junit.Assert.fail(Assert.java:88) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:654) at kafka.server.LogRecoveryTest.testHWCheckpointNoFailuresMultipleLogSegments(LogRecoveryTest.scala:173) kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresMultipleLogSegments FAILED java.lang.AssertionError: expected:<4> but was:<3> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at kafka.server.LogRecoveryTest.testHWCheckpointWithFailuresMultipleLogSegments(LogRecoveryTest.scala:229) *RUN 3 :* kafka.server.PlaintextReplicaFetchTest > testReplicaFetcherThread FAILED kafka.common.FailedToSendMessageException: Failed to send messages after 5 tries. kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled FAILED java.lang.AssertionError: expected:<List(first)> but was:<List(first, first)> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionEnabled(UncleanLeaderElectionTest.scala:185) at kafka.integration.UncleanLeaderElectionTest.testUncleanLeaderElectionEnabled(UncleanLeaderElectionTest.scala:110) kafka.integration.UncleanLeaderElectionTest > testCleanLeaderElectionDisabledByTopicOverride FAILED java.lang.AssertionError: expected:<List(first, second, third)> but was:<List()> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:253) at kafka.integration.UncleanLeaderElectionTest.testCleanLeaderElectionDisabledByTopicOverride(UncleanLeaderElectionTest.scala:155) kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled FAILED java.lang.AssertionError: expected:<List(first)> but was:<List(first, first, first)> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:220) at kafka.integration.UncleanLeaderElectionTest.testUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:123) kafka.api.ProducerBounceTest > testBrokerFailure FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:110) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. 4) Than I ran ./gradlew test with commit : commit b7d4043d8da1d296519faf321e9f0f7188d7d67a Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: Sat Sep 12 21:58:34 2015 -0700 5) This is 4 commits behind this patch. All tests pass for this. was (Author: mgharat): I am not sure if the failures are related to this patch. Here is what I did: 1) Checked out the lcommit : commit b658e25207174578c26ea94a75e5b0ea10f06ba5 Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: Mon Sep 14 15:02:32 2015 -0700 2) This is one commit before this patch. I ran ./gradlew test. 3) Got following errors, in different runs : *RUN 1 :* kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED java.lang.AssertionError: Failed to update high watermark for follower after timeout at org.junit.Assert.fail(Assert.java:88) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:654) at kafka.server.LogRecoveryTest.testHWCheckpointNoFailuresMultipleLogSegments(LogRecoveryTest.scala:173) kafka.api.SSLConsumerTest > testGroupConsumption FAILED java.lang.IllegalStateException: Failed to consume the expected records after 301 iterations. at kafka.api.SSLConsumerTest.consumeRecords(SSLConsumerTest.scala:226) at kafka.api.SSLConsumerTest.testGroupConsumption(SSLConsumerTest.scala:165) kafka.integration.UncleanLeaderElectionTest > testCleanLeaderElectionDisabledByTopicOverride FAILED java.lang.AssertionError: expected:<List(first, second, third)> but was:<List()> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:253) at kafka.integration.UncleanLeaderElectionTest.testCleanLeaderElectionDisabledByTopicOverride(UncleanLeaderElectionTest.scala:155) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:132) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. kafka.api.ProducerBounceTest > testBrokerFailure FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:110) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. *RUN 2 :* kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments FAILED java.lang.AssertionError: Failed to update high watermark for follower after timeout at org.junit.Assert.fail(Assert.java:88) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:654) at kafka.server.LogRecoveryTest.testHWCheckpointNoFailuresMultipleLogSegments(LogRecoveryTest.scala:173) kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresMultipleLogSegments FAILED java.lang.AssertionError: expected:<4> but was:<3> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at kafka.server.LogRecoveryTest.testHWCheckpointWithFailuresMultipleLogSegments(LogRecoveryTest.scala:229) *RUN 3 :* kafka.server.PlaintextReplicaFetchTest > testReplicaFetcherThread FAILED kafka.common.FailedToSendMessageException: Failed to send messages after 5 tries. kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled FAILED java.lang.AssertionError: expected:<List(first)> but was:<List(first, first)> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionEnabled(UncleanLeaderElectionTest.scala:185) at kafka.integration.UncleanLeaderElectionTest.testUncleanLeaderElectionEnabled(UncleanLeaderElectionTest.scala:110) kafka.integration.UncleanLeaderElectionTest > testCleanLeaderElectionDisabledByTopicOverride FAILED java.lang.AssertionError: expected:<List(first, second, third)> but was:<List()> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:253) at kafka.integration.UncleanLeaderElectionTest.testCleanLeaderElectionDisabledByTopicOverride(UncleanLeaderElectionTest.scala:155) kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled FAILED java.lang.AssertionError: expected:<List(first)> but was:<List(first, first, first)> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:220) at kafka.integration.UncleanLeaderElectionTest.testUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:123) kafka.api.ProducerBounceTest > testBrokerFailure FAILED java.lang.AssertionError at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117) kafka.api.ConsumerTest > testPatternSubscription FAILED java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:56) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:43) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at kafka.api.ConsumerTest$$anonfun$sendRecords$1.apply(ConsumerTest.scala:449) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.ConsumerTest.sendRecords(ConsumerTest.scala:449) at kafka.api.ConsumerTest.testPatternSubscription(ConsumerTest.scala:110) Caused by: org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received. 4) Than I ran ./gradlew test with commit : commit b7d4043d8da1d296519faf321e9f0f7188d7d67a Author: Parth Brahmbhatt <brahmbhatt.pa...@gmail.com> Date: Sat Sep 12 21:58:34 2015 -0700 5) This is 4 commits behind this patch. All tests pass for this. > Add a request timeout to NetworkClient > -------------------------------------- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature > Reporter: Jiangjie Qin > Assignee: Mayuresh Gharat > Priority: Blocker > Fix For: 0.9.0.0 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, > KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, > KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, > KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)