[
https://issues.apache.org/jira/browse/KAFKA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317327#comment-17317327
]
Matthias J. Sax commented on KAFKA-12453:
-----------------------------------------
{quote}and why `builder.stream().toTable()` is compatible with optimisation and
`buidler.table()` isn't?
{quote}
If you use `builder.stream().toTable()` the optimization is disabled (because
the table is not directly created from an input topic). So it's "compatible" in
the sense that it's always safe to enable optimization, because we always
create a dedicated changelog anyway. – My worry was the case of
`builder.stream().map().toTable()` – for this case, after the `map()` there
would be repartition topic and thus the table is created directly from an topic
– but as Sophie pointed out, we have a safe guard in place (and a test for it)
and also disable the optimization for this case.
For `builder.table()` we apply the optimization (and it's the only case for it)
if enabled and thus the input topic should be configured with log compaction. –
I don't think there is any "incompatibility" though.
Does this make sense? – Do you wan to do a PR to update the docs accordingly?
> Guidance on whether a topology is eligible for optimisation
> -----------------------------------------------------------
>
> Key: KAFKA-12453
> URL: https://issues.apache.org/jira/browse/KAFKA-12453
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Patrick O'Keeffe
> Priority: Major
>
> Since the introduction of KStream.toTable() in Kafka 2.6.x, the decision
> about whether a topology is eligible for optimisation is no longer a simple
> one, and is related to whether toTable() operations are preceded by key
> changing operators.
> This decision requires expert level knowledge, and there are serious
> implications associated with getting it wrong in terms of fault tolerance
> Some ideas spring to mind around how to guide developers to make the correct
> decision:
> # Topology.describe() could indicate whether this topology is eligible for
> optimisation
> # Topologies could be automatically optimised - note this may have an impact
> at deployment time, in that an application reset may be required. The
> developer would need to made aware of this and adjust the deployment plan
> accordingly
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
