[
https://issues.apache.org/jira/browse/KAFKA-6036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713893#comment-16713893
]
ASF GitHub Bot commented on KAFKA-6036:
---------------------------------------
guozhangwang opened a new pull request #6017: KAFKA-6036: Follow-up; cleanup
sendOldValues logic ForwardingCacheFlushListener
URL: https://github.com/apache/kafka/pull/6017
This is a follow-up PR from the previous PR
https://github.com/apache/kafka/pull/5779, where KTabeSource always get old
values from the store even if sendOldValues. It gets me to make a pass over all
the KTable/KStreamXXX processor to push the `sendOldValues` at the callers in
order to avoid unnecessary store reads.
1. Do the above.
2. `ForwardingCacheFlushListener` and `TupleForwarder` both need
sendOldValues as parameters.
a. For `ForwardingCacheFlushListener` it is not needed at all, since its
callers `XXXCachedStore` already use the sendOldValues values passed from
`TupleForwarder` to avoid getting old values from underlying stores.
b. For `TupleForwarder`, it actually only need to pass the boolean flag to
the cached store; and then it does not need to keep it as its own variable if
we already do 0).
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enable logical materialization to physical materialization
> ----------------------------------------------------------
>
> Key: KAFKA-6036
> URL: https://issues.apache.org/jira/browse/KAFKA-6036
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Guozhang Wang
> Priority: Major
>
> Today whenever users specify a queryable store name for KTable, we would
> always add a physical state store in the translated processor topology.
> For some scenarios, we should consider not physically materialize the KTable
> but only "logically" materialize it when you have some simple transformation
> operations or even join operations that generated new KTables, and which
> needs to be materialized with a state store, you can use the changelog topic
> of the previous KTable and applies the transformation logic upon restoration
> instead of creating a new changelog topic. For example:
> {code}
> table1 = builder.table("topic1");
> table2 = table1.filter(..).join(table3); // table2 needs to be materialized
> for joining
> {code}
> We can actually set the {{getter}} function of table2's materialized store,
> say {{state2}} to be reading from {{topic1}} and then apply the filter
> operator, instead of creating a new {{state2-changelog}} topic in this case.
> We can come up with a general internal impl optimizations to determine when
> to logically materialize, and whether we should actually allow users of DSL
> to "hint" whether to materialize or not (it then may need a KIP).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
