Jungtaek Lim created SPARK-55630:
------------------------------------
Summary: Skip updating matched flag on non-outer side of
stream-stream join
Key: SPARK-55630
URL: https://issues.apache.org/jira/browse/SPARK-55630
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Jungtaek Lim
Updating matched flag in stream-stream join triggers write(s) during retrieval
of the matching values. While this is required for outer join side to track the
unmatched output, it's unnecessary to track the matched flag for non-outer join
side - the cost of updating matched flag is significant. For non-outer join
side, we can skip updating the matched flag.
Applying this optimization would break the case of changing join type during
restarting of the query, but we do not have an explicit support of it, so this
optimization does not break the actual use case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
