[
https://issues.apache.org/jira/browse/KAFKA-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032511#comment-17032511
]
John Roesler commented on KAFKA-8307:
-------------------------------------
This issue has been lurking for a while, and has been reported a number of
different ways. It seems to take two forms:
1. changing the topology at all (in apparently compatible ways) can renumber
operators and corrupt the application upon restart
2. changing the topology in combination with a rolling bounce results in
members executing a different topology than the leader, which leads to extra
problems (such as NPEs)
https://issues.apache.org/jira/browse/KAFKA-7669 is related, and seems to be
more about just changing the topology at all
https://issues.apache.org/jira/browse/KAFKA-8307 proposes a fix
https://issues.apache.org/jira/browse/KAFKA-8810 seems to be a duplicate of
KAFKA-8307
and https://issues.apache.org/jira/browse/KAFKA-9518 reports an exception that
results from a rolling-bounce topology change.
> Kafka Streams should provide some mechanism to determine topology equality
> and compatibility
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-8307
> URL: https://issues.apache.org/jira/browse/KAFKA-8307
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: John Roesler
> Priority: Major
> Labels: user-experience
>
> Currently, Streams provides no mechanism to compare two topologies. This is a
> common operation when users want to have tests verifying that they don't
> accidentally alter their topology. They would save the known-good topology
> and then add a unit test verifying the current code against that known-good
> state.
> However, because there's no way to do this comparison properly, everyone is
> reduced to using the string format of the topology (from
> `Topology#describe().toString()`). The major drawback is that the string
> format is meant for human consumption. It is neither machine-parseable nor
> stable. So, these compatibility tests are doomed to fail when any minor,
> non-breaking, change is made either to the application, or to the library.
> This trains everyone to update the test whenever it fails, undermining its
> utility.
> We should fix this problem, and provide both a mechanism to serialize the
> topology and to compare two topologies for compatibility. All in all, I think
> we need:
> # a way to serialize/deserialize topology structure in a machine-parseable
> format that is future-compatible. Offhand, I'd recommend serializing the
> topology structure as JSON, and establishing a policy that attributes should
> only be added to the object graph, never removed. Note, it's out of scope to
> be able to actually run a deserialized topology; we only want to save and
> load the structure (not the logic) to facilitate comparisons.
> # a method to verify the *equality* of two topologies... This method tells
> you that the two topologies are structurally identical. We can't know if the
> logic of any operator has changed, only if the structure of the graph is
> changed. We can consider whether other graph properties, like serdes, should
> be included.
> # a method to verify the *compatibility* of two topologies... This method
> tells you that moving from topology A to topology B does not require an
> application reset. Note that this operation is not commutative:
> `A.compatibleWith(B)` does not imply `B.compatibleWith(A)`. We can discuss
> whether `A.compatibleWith(B) && B.compatibleWith(A)` implies `A.equals(B)` (I
> think not necessarily, because we may want "equality" to be stricter than
> "compatibility").
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
