Hi Marcus,
1. If you change REPLICATION_FACTOR_CONFIG without resetting the
application (or deleting the changelog and repartition topics) and
redeploy the Streams application, the replication factor of the internal
topics will not change. The replication factor will only change for new
deployments, i.e., Streams applications with a new application ID or
Streams applications that where reset. In both cases the internal topics
will be newly created.
2. Changing the replication factor of a topic directly on the brokers
should be fine. Kafka Streams should not re-create the internal topics
and not throw any exceptions.
3. Unfortunately, I do not know the answer to this question. Hopefully
somebody else can answer it.
I answered your questions to the best of my knowledge. The only way to
confirm my answers is to test (preferably in a test environment).
Best,
Bruno
On 28.04.21 17:34, Marcus Horsley-Rai wrote:
Hi All,
I'm in a sub-optimal situation whereby I have some Kafka Streams apps
deployed to production, but the default replication factor set on the
brokers was 1 when they were first deployed.
As such, any state store changelog topics, and re-partition topics
therefore have RF 1 also.
I'm familiar with the bin/kafka-reassign-partitions.sh tool and how to use
that.
I've since also found the streams replication.factor
(StreamsConfig.REPLICATION_FACTOR_CONFIG) setting that can be supplied in
the streams apps config.
My questions are:
1. Will simply changing the value of REPLICATION_FACTOR_CONFIG and
re-deploying have any effect on already-created internal topics?
2. Conversely, should I just change the RF of the internal topics using
the re-assign-partitions tool? Is that safe to do whilst the apps are still
running?
3. (not Streams-specific) If a broker that was the leader of
partition(s) for a topic with RF 1 died (i.e. no replicas), and was
non-recoverable - would the only way to restore service to those partitions
be to delete the topic and re-create? (with data loss, understandably). I
couldn't seem to achieve this using preferred/unclean leader election, nor
using the re-assign-partitions tool, since that requires all brokers to be
healthy.
Many thanks in advance for any answers,
Marcus
