[
https://issues.apache.org/jira/browse/KAFKA-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Bates updated KAFKA-13876:
------------------------------------
Description:
I am starting a mirrormaker connect cluster using ./bin/connect-mirror-maker.sh
and a properties file.
I followed the guide at
[https://github.com/apache/kafka/tree/trunk/connect/mirror] to attempt to
configure the consumer properties, however, no matter what I set, the timeout
for the consumer remains fixed at the default, confirmed both by kafka
outputting its config in the log and by timing how long between disconnect
messages I get. e.g. I set:
{{CLOUD_EU.consumer.request.timeout.ms=120000}}
In the properties that I start MM-2 with and it is ignored.
based on various guides I have found while looking at this, I have also tried
{{CLOUD_EU.request.timeout.ms=120000}}
{{CLOUD_EU.cluster.consumer.request.timeout.ms=120000}}
{{CLOUD_EU.consumer.override.request.timeout.ms=120000}}
{{CLOUD_EU.cluster.consumer.override.request.timeout.ms=120000}}
None of which have worked.
How can I change the consumer request.timeout setting?
Context:
I am attempting to use mirrormaker 2 to replicate data between AWS Managed
Kafkas (MSK) in 2 different AWS regions - one in eu-west-1 (CLOUD_EU) and the
other in us-west-2 (CLOUD_NA), both running Kafka 2.6.1. For testing I am
currently trying just to replicate topics 1 way, from EU -> NA.
This works fine for topics with small messages on them, but one of my topic has
binary messages up to 20MB in size. When I try to replicate that topic I get
an error every 30 seconds
{{[2022-04-21 13:47:05,268] INFO [Consumer clientId=consumer-29, groupId=null]
Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 2: {}.
(org.apache.kafka.clients.FetchSessionHandler:481)}}
{{org.apache.kafka.common.errors.DisconnectException}}
When logging in DEBUG to get more information we get
{{[2022-04-21 13:47:05,267] DEBUG [Consumer clientId=consumer-29, groupId=null]
Disconnecting from node 2 due to request timeout.
(org.apache.kafka.clients.NetworkClient:784)}}
{{[2022-04-21 13:47:05,268] DEBUG [Consumer clientId=consumer-29, groupId=null]
Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=11,
clientId=consumer-29, correlationId=35) due to node 2 being disconnected
(org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient:593)}}
It gets stuck in a loop constantly disconnecting with request timeout every 30s
and then trying again.
Looking at this, I suspected that the problem is the `request.timeout.ms` was
on the default (30s) and timing out trying to read the topic with many large
messages, hence I was trying to override the consumer settings to have a longer
timeout.
The 2 specified versions are the ones I have explicitly tested against.
Properties file I am using to start the cluster:
{{clusters = CLOUD_EU, CLOUD_NA}}
{{CLOUD_EU.bootstrap.servers = kafka.eu-west-1.amazonaws.com:9092}}
{{CLOUD_NA.bootstrap.servers = kafka.us-west-2.amazonaws.com:9092}}
{{CLOUD_EU->CLOUD_NA.enabled = true}}
{{CLOUD_EU->CLOUD_NA.topics = METRICS_ATTACHMENTS_OVERSIZE_EU}}
{{CLOUD_NA->CLOUD_EU.enabled = false}}
{{replication.factor=3}}
{{tasks.max = 1}}
{{checkpoints.topic.replication.factor=3}}
{{heartbeats.topic.replication.factor=3}}
{{offset-syncs.topic.replication.factor=3}}
{{offset.storage.replication.factor=3}}
{{status.storage.replication.factor=3}}
{{config.storage.replication.factor=3}}
{{CLOUD_EU.consumer.request.timeout.ms=120000}}
{{CLOUD_EU.consumer.session.timeout.ms=150000}}
was:
I am starting a mirrormaker connect cluster using ./bin/connect-mirror-maker.sh
and a properties file.
I followed the guide at
[https://github.com/apache/kafka/tree/trunk/connect/mirror] to attempt to
configure the consumer properties, however, no matter what I set, the timeout
for the consumer remains fixed at the default, confirmed both by kafka
outputting its config in the log and by timing how long between disconnect
messages I get. e.g. I set:
{{CLOUD_EU.consumer.request.timeout.ms=120000}}
In the properties that I start MM-2 with and it is ignored.
based on various guides I have found while looking at this, I have also tried
{{CLOUD_EU.request.timeout.ms=120000}}
{{CLOUD_EU.cluster.consumer.request.timeout.ms=120000}}
{{CLOUD_EU.consumer.override.request.timeout.ms=120000}}
{{CLOUD_EU.cluster.consumer.override.request.timeout.ms=120000}}
None of which have worked.
How can I change the consumer request.timeout setting?
Context:
I am attempting to use mirrormaker 2 to replicate data between AWS Managed
Kafkas (MSK) in 2 different AWS regions - one in eu-west-1 (CLOUD_EU) and the
other in us-west-2 (CLOUD_NA), both running Kafka 2.6.1. For testing I am
currently trying just to replicate topics 1 way, from EU -> NA.
This works fine for topics with small messages on them, but one of my topic has
binary messages up to 20MB in size. When I try to replicate that topic I get
an error every 30 seconds
{{[2022-04-21 13:47:05,268] INFO [Consumer clientId=consumer-29, groupId=null]
Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 2: {}.
(org.apache.kafka.clients.FetchSessionHandler:481)}}
{{org.apache.kafka.common.errors.DisconnectException}}
When logging in DEBUG to get more information we get
{{[2022-04-21 13:47:05,267] DEBUG [Consumer clientId=consumer-29, groupId=null]
Disconnecting from node 2 due to request timeout.
(org.apache.kafka.clients.NetworkClient:784)}}
{{[2022-04-21 13:47:05,268] DEBUG [Consumer clientId=consumer-29, groupId=null]
Cancelled request with header RequestHeader(apiKey=FETCH, apiVersion=11,
clientId=consumer-29, correlationId=35) due to node 2 being disconnected
(org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient:593)}}
It gets stuck in a loop constantly disconnecting with request timeout every 30s
and then trying again.
Looking at this, I suspected that the problem is the `request.timeout.ms` was
on the default (30s) and timing out trying to read the topic with many large
messages, hence I was trying to override the consumer settings to have a longer
timeout.
The 2 specified versions are the ones I have explicitly tested against.
Properties file I am using to start the cluster:
{{clusters = CLOUD_EU, CLOUD_NA}}
{{CLOUD_EU.bootstrap.servers = kafka.eu-west-1.amazonaws.com:9092}}
{{CLOUD_NA.bootstrap.servers = kafka.us-west-2.amazonaws.com:9092}}
{{CLOUD_EU->CLOUD_NA.enabled = true}}
{{CLOUD_EU->CLOUD_NA.topics = METRICS_ATTACHMENTS_OVERSIZE_EU}}
{{CLOUD_NA->CLOUD_EU.enabled = false}}
{{replication.factor=3}}
{{{}tasks.max = 1{}}}{{{}checkpoints.topic.replication.factor=3{}}}
{{heartbeats.topic.replication.factor=3}}
{{offset-syncs.topic.replication.factor=3}}
{{offset.storage.replication.factor=3}}
{{status.storage.replication.factor=3}}
{{config.storage.replication.factor=3}}
{{CLOUD_EU.consumer.request.timeout.ms=120000}}
{{CLOUD_EU.consumer.session.timeout.ms=150000}}
> Mirrormaker-2 consumer settings not working
> -------------------------------------------
>
> Key: KAFKA-13876
> URL: https://issues.apache.org/jira/browse/KAFKA-13876
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.5.0, 2.6.1
> Reporter: Christian Bates
> Priority: Minor
>
> I am starting a mirrormaker connect cluster using
> ./bin/connect-mirror-maker.sh and a properties file.
> I followed the guide at
> [https://github.com/apache/kafka/tree/trunk/connect/mirror] to attempt to
> configure the consumer properties, however, no matter what I set, the timeout
> for the consumer remains fixed at the default, confirmed both by kafka
> outputting its config in the log and by timing how long between disconnect
> messages I get. e.g. I set:
> {{CLOUD_EU.consumer.request.timeout.ms=120000}}
> In the properties that I start MM-2 with and it is ignored.
>
> based on various guides I have found while looking at this, I have also tried
> {{CLOUD_EU.request.timeout.ms=120000}}
> {{CLOUD_EU.cluster.consumer.request.timeout.ms=120000}}
> {{CLOUD_EU.consumer.override.request.timeout.ms=120000}}
> {{CLOUD_EU.cluster.consumer.override.request.timeout.ms=120000}}
> None of which have worked.
> How can I change the consumer request.timeout setting?
>
> Context:
> I am attempting to use mirrormaker 2 to replicate data between AWS Managed
> Kafkas (MSK) in 2 different AWS regions - one in eu-west-1 (CLOUD_EU) and the
> other in us-west-2 (CLOUD_NA), both running Kafka 2.6.1. For testing I am
> currently trying just to replicate topics 1 way, from EU -> NA.
> This works fine for topics with small messages on them, but one of my topic
> has binary messages up to 20MB in size. When I try to replicate that topic I
> get an error every 30 seconds
> {{[2022-04-21 13:47:05,268] INFO [Consumer clientId=consumer-29,
> groupId=null] Error sending fetch request (sessionId=INVALID, epoch=INITIAL)
> to node 2: {}. (org.apache.kafka.clients.FetchSessionHandler:481)}}
> {{org.apache.kafka.common.errors.DisconnectException}}
>
> When logging in DEBUG to get more information we get
> {{[2022-04-21 13:47:05,267] DEBUG [Consumer clientId=consumer-29,
> groupId=null] Disconnecting from node 2 due to request timeout.
> (org.apache.kafka.clients.NetworkClient:784)}}
> {{[2022-04-21 13:47:05,268] DEBUG [Consumer clientId=consumer-29,
> groupId=null] Cancelled request with header RequestHeader(apiKey=FETCH,
> apiVersion=11, clientId=consumer-29, correlationId=35) due to node 2 being
> disconnected
> (org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient:593)}}
> It gets stuck in a loop constantly disconnecting with request timeout every
> 30s and then trying again.
> Looking at this, I suspected that the problem is the `request.timeout.ms` was
> on the default (30s) and timing out trying to read the topic with many large
> messages, hence I was trying to override the consumer settings to have a
> longer timeout.
>
> The 2 specified versions are the ones I have explicitly tested against.
>
> Properties file I am using to start the cluster:
>
> {{clusters = CLOUD_EU, CLOUD_NA}}
> {{CLOUD_EU.bootstrap.servers = kafka.eu-west-1.amazonaws.com:9092}}
> {{CLOUD_NA.bootstrap.servers = kafka.us-west-2.amazonaws.com:9092}}
> {{CLOUD_EU->CLOUD_NA.enabled = true}}
> {{CLOUD_EU->CLOUD_NA.topics = METRICS_ATTACHMENTS_OVERSIZE_EU}}
> {{CLOUD_NA->CLOUD_EU.enabled = false}}
> {{replication.factor=3}}
> {{tasks.max = 1}}
> {{checkpoints.topic.replication.factor=3}}
> {{heartbeats.topic.replication.factor=3}}
> {{offset-syncs.topic.replication.factor=3}}
> {{offset.storage.replication.factor=3}}
> {{status.storage.replication.factor=3}}
> {{config.storage.replication.factor=3}}
> {{CLOUD_EU.consumer.request.timeout.ms=120000}}
> {{CLOUD_EU.consumer.session.timeout.ms=150000}}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
