[
https://issues.apache.org/jira/browse/KAFKA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16727992#comment-16727992
]
Raghu Baddam edited comment on KAFKA-6947 at 12/23/18 4:04 PM:
---------------------------------------------------------------
Even we are facing the same issue in Kafka 1.1.1 version.
{code}
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 978 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)
at java.lang.Thread.run(Thread.java:745)
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 1235 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)
at java.lang.Thread.run(Thread.java:745)
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 1175 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)
{code}
{code}
bash-4.2$ cat mirrormaker.properties
{code}
{code}
# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
whitelist=<list of topics>
num.streams=4
{code}
{code}
bash-4.2$ cat producer.properties
{code}
{code}
# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
bootstrap.servers=<list of kafka servers>
request.timeout.ms=60000
-bash-4.2$
-bash-4.2$ cat consumer.properties
# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
# mirror cluster's consumer group id
group.id=KafkaMirror-vulcan-sc
auto.offset.reset=earliest
bootstrap.servers=<list of kafka servers>
{code}
was (Author: raghu98...@gmail.com):
Even we are facing the same issue in Kafka 1.1.1 version.
{code}
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 978 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)
at java.lang.Thread.run(Thread.java:745)
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 1235 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)
at java.lang.Thread.run(Thread.java:745)
[2018-12-23 15:23:51,800] INFO Closing producer due to send failure.
(kafka.tools.MirrorMaker$)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Closing the Kafka
producer with timeoutMillis = 0 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] INFO [Producer clientId=producer-1] Proceeding to
force close the producer since pending requests could not be completed within
timeout 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2018-12-23 15:23:51,800] ERROR Error when sending message to topic
vulcan_carrier_tracking_updates_enriched_walmart_prod-az-uswest with key: 36
bytes, value: 1175 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
java.lang.IllegalStateException: Producer is closed forcefully.
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:696)
at
org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:683)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:185)\{code}
{code}bash-4.2$ cat mirrormaker.properties \{code}
{code}# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
whitelist=<list of topics>
num.streams=4
{code}
{code}bash-4.2$ cat producer.properties \{code}
{code}# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
bootstrap.servers=<list of kafka servers>
request.timeout.ms=60000
-bash-4.2$
-bash-4.2$ cat consumer.properties
# This file is generated by chef
# Any manual changes will be overwritten!
# To make changes, edit the template in the mirrormaker cookbook
#
# mirror cluster's consumer group id
group.id=KafkaMirror-vulcan-sc
auto.offset.reset=earliest
bootstrap.servers=<list of kafka servers>
{code}
> Mirrormaker Closing producer due to send failure
> ------------------------------------------------
>
> Key: KAFKA-6947
> URL: https://issues.apache.org/jira/browse/KAFKA-6947
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 0.11.0.0
> Reporter: Andrew Holford
> Priority: Major
>
> Hi
> On occasion our mirrormakers fail with the below error....
> [2018-05-25 05:10:31,695] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 355 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 38 record(s) for
> com_snapshot--demo-5: 91886 ms has passed since last append
> [2018-05-25 05:10:31,710] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:31,710] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:31,710] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and just after this:
> [2018-05-25 05:10:32,041] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 323 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for
> com_snapshot--demo-5: 89429 ms has passed since batch creation plus linger
> time
> [2018-05-25 05:10:32,041] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,041] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,041] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> and sometimes we get:
> [2018-05-25 08:24:29,249] ERROR Error when sending message to topic
> com_snapshot--demo with key: 13 bytes, value: 354 bytes with error:
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
> at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:183)
> at java.lang.Thread.run(Thread.java:745)
> [2018-05-25 08:24:29,249] INFO Closing producer due to send failure.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 08:24:29,249] INFO Closing the Kafka producer with timeoutMillis
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 08:24:29,249] INFO Proceeding to force close the producer since
> pending requests could not be completed within timeout 0 ms.
> (org.apache.kafka.clients.producer.KafkaProducer)
> then it completely shuts down:
> [2018-05-25 05:10:32,060] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,083] INFO [mirrormaker-thread-0] Flushing producer.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO [mirrormaker-thread-0] Committing consumer
> offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,094] INFO Exiting on send failure, skip committing
> offsets. (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,105] INFO [mirrormaker-thread-0] Shutting down consumer
> connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,138] INFO Publish thread interrupted!
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,144] INFO Publishing Monitoring Metrics stopped for
> clientID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,148] INFO Closing the Kafka producer with timeoutMillis
> = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:32,183] INFO Closed monitoring interceptor for client
> ID=peach-prod2prod3-mirrormaker-0
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
> [2018-05-25 05:10:32,197] INFO [mirrormaker-thread-0] Mirror maker thread
> stopped (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,202] FATAL [mirrormaker-thread-0] Mirror maker thread
> exited abnormally, stopping the whole mirror maker.
> (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,203] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,245] INFO Start clean shutdown.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,245] INFO Shutting down consumer threads.
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:32,261] INFO [mirrormaker-thread-0] mirrormaker-thread-0
> shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2018-05-25 05:10:32,265] INFO [mirrormaker-thread-0] Mirror maker thread
> shutdown complete (kafka.tools.MirrorMaker$MirrorMakerThread)
> On the broker side we see:
> WARN Attempting to send response via channel for which there is no open
> connection, connection id 10.82.6.105:9093-172.27.205.216:32796
> (kafka.network.Processor)
> Does anyone know what could cause this and what a possible solution could be?
> Im a little confused by the "timeoutMillis = 0 ms" as well mentioned, is this
> some setting which needs adjusting somewhere? We have
> request.timeout.ms=60000 on the producer config with most other settings left
> as the defaults.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
