[
https://issues.apache.org/jira/browse/KAFKA-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206775#comment-16206775
]
Guozhang Wang commented on KAFKA-6047:
--------------------------------------
Thanks for reporting this [~dvsekhvalnov]. I agree that we should make Streams
API be more resilient with such broker unavailability issues. Regarding whether
we should add a separate configuration to cover this case, maybe not need to.
[~mjsax] is already looking into this issue, and I'll leave it to him to
present a proposal.
> Allow retries configuration for InternalTopicManager
> ----------------------------------------------------
>
> Key: KAFKA-6047
> URL: https://issues.apache.org/jira/browse/KAFKA-6047
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 0.11.0.0
> Reporter: Dmitry Vsekhvalnov
> Priority: Minor
>
> There is hardcoded number of retries when kafka-streams attempts to create
> internal topics.
> *InternalTopicManager.MAX_TOPIC_READY_TRY*
> Which is not resilient in some scenarios. Consider setup where replication
> factor for internal streams topics == number of brokers. (RF=3 and x3 kafka
> brokers). When any of brokers dies kafka-streams can shutdown before broker
> is resurrected with approximate log:
> {code}
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #2]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #3]
> [WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
> [Could not create internal topics: Found only 2 brokers, but replication
> factor is 3. Decrease replication factor for internal topics via
> StreamsConfig parameter "replication.factor" or add more brokers to your
> cluster. Retry #4]
> [INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
> [stream-thread [Shutting down]
> {code}
> Would be nice if kafka-streams provides configuration for
> InternalTopicManager retries and ideally for retry backoff strategy. To have
> possibility for resilience tuning. Possibly can re-use corresponding producer
> configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
