Chris Schwarzfischer created KAFKA-7133:
-------------------------------------------
Summary: DisconnectException every 5 minutes in single restore
consumer thread
Key: KAFKA-7133
URL: https://issues.apache.org/jira/browse/KAFKA-7133
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 1.1.0
Environment: Kafka Streams application in Kubernetes.
Kafka Server in Docker on machine in host mode
Reporter: Chris Schwarzfischer
One of our streams applications (and only this one) gets a
{{org.apache.kafka.common.errors.DisconnectException}} almost exactly every 5
minutes.
The application has two of
KStream -> KGroupedStream -> KTable -> KGroupedTable -> KTable
aggregations.
Relevant config is in Streams:
{code:java}
this.properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
StreamsConfig.AT_LEAST_ONCE);
//...
this.properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
this.properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 1024 * 1024
* 500 /* 500 MB */ );
this.properties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 *
1024 * 100 /* 100 MB */);
this.properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 1024 * 1024 * 50 /*
50 MB */);
{code}
On the broker:
{noformat}
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_OFFSETS_RETENTION_MINUTES: 108000
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 2147483000
KAFKA_LOG_RETENTION_HOURS: 2688
KAFKA_OFFSETS_RETENTION_CHECK_INTERVAL_MS: 1200000
KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 12000
{noformat}
Logging gives us a single restore consumer thread that throws exceptions every
5 mins:
{noformat}
July 4th 2018, 15:38:51.560 dockertest03 2018-07-04T13:38:51,559Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:26.379 dockertest03 2018-07-04T13:32:26,378Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:26:53.886 dockertest03 2018-07-04T13:26:53,886Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:26:14.576 dockertest03 2018-07-04T13:26:14,575Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:26:14.575 dockertest03 2018-07-04T13:26:14,575Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:21:13.410 dockertest03 2018-07-04T13:21:13,409Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:20:39.618 dockertest03 2018-07-04T13:20:39,617Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:20:39.617 dockertest03 2018-07-04T13:20:39,617Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:15:31.454 dockertest03 2018-07-04T13:15:31,454Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:15:00.759 dockertest03 2018-07-04T13:15:00,758Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:15:00.758 dockertest03 2018-07-04T13:15:00,758Z INFO
:
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
FetchSessionHandler::handleError:440 - [Consumer
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to
node 2: org.apache.kafka.common.errors.DisconnectException.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
