[
https://issues.apache.org/jira/browse/KAFKA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311972#comment-17311972
]
Patrick O'Keeffe commented on KAFKA-12453:
------------------------------------------
Thanks for following up on this [~mjsax], [~ableegoldman]. A few (perhaps
dumb/obvious) questions, but I want to make sure I have understood correctly:
# Are we saying that its safe to enable topology optimisation when we have a
toTable() operation, regardless of whether it is the first node in a
sub-topology or if it is preceded by a key changing operator?
# If so, does this mean there are no caveats re topology optimisation, and its
safe to enable in any topology?
In which case, why not make it a non-configurable default? Its a good
optimisation, which can save a lot of broker load as you avoid the republish to
changelog topics. In our company, we are dealing with fast moving trading data
(50k + / sec), and the reduction in broker load is not insignificant. Also, its
one less thing to consider at deployment time
> Guidance on whether a topology is eligible for optimisation
> -----------------------------------------------------------
>
> Key: KAFKA-12453
> URL: https://issues.apache.org/jira/browse/KAFKA-12453
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Patrick O'Keeffe
> Priority: Major
>
> Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision
> about whether a topology is eligible for optimisation is no longer a simple
> one, and is related to whether toTable() operations are preceded by key
> changing operators.
> This decision requires expert level knowledge, and there are serious
> implications associated with getting it wrong in terms of fault tolerance
> Some ideas spring to mind around how to guide developers to make the correct
> decision:
> # Topology.describe() could indicate whether this topology is eligible for
> optimisation
> # Topologies could be automatically optimised - note this may have an impact
> at deployment time, in that an application reset may be required. The
> developer would need to made aware of this and adjust the deployment plan
> accordingly
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
