[
https://issues.apache.org/jira/browse/KAFKA-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938134#comment-17938134
]
Lorcan commented on KAFKA-18054:
--------------------------------
Hi [~cadonna], thanks for the additional insight.
I think you might be right in your last paragraph. The issue I encountered was
when a StreamsBuilder was created without any arguments passed to the
constructor and the topology is created and optimised by passing in properties
to the StreamsBuilder.build method.
The properties are used in optimizeTopology(), but no TopologyConfig object has
been created at this point and the properties don't seem to be persisted. I
checked StreamsConfig.REUSE_KTABLE_SOURCE_TOPICS usages and it only seems to be
applied by the properties and not a configuration object.
Then in the private constructor of the KafkaStreams, there is a chain of method
calls, buildAndRewriteTopology() -> rewriteTopology() -> setStreamsConfig,
which sets the TopologyConfig of the InternalTopologyBuilder. The
TopologyConfigs have not been created by this point so an empty
topologyOverride is created and the StreamsConfig is passed to the
TopologyConstructor.
I thought the topologyOverrides might hold this information as the
Topology_Optimization_Configs use the Properties class but this was empty given
the paragraph above this.
Do you have any suggestions where this information should be persisted in the
TopologyConfig/StreamsConfig?
> Automatically detect missed configs needed by a topology
> --------------------------------------------------------
>
> Key: KAFKA-18054
> URL: https://issues.apache.org/jira/browse/KAFKA-18054
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> See umbrella ticket for problem summary.
> This ticket would help tackle the first issue and would be a signifciant
> improvement without requiring a KIP.
> Basically, we should help users avoid accidentally misconfiguring their
> application by setting one of the topology-specific configs in their
> StreamsConfig but only passing it into the KafkaStreams constructor and not
> into the topology where it actually is applied.
> We should be able to detect this case pretty easily: when the topology and
> application configs are passed into the KafkaStreams constructor, we just
> compare the topology's configs with the provided application configs and look
> for each of the topology-specific configs (see umbrella ticket for the list).
> If any of these configs were set by the user in the application configs but
> not set in the topology's configs, we can throw an exception.
> We should also log a warning if the config is set in both cases but have a
> different value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
