[
https://issues.apache.org/jira/browse/SPARK-59698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-59698:
---------------------------------
Component/s: SQL
> ASOF JOIN isn't properly blocked for stream on the right side
> -------------------------------------------------------------
>
> Key: SPARK-59698
> URL: https://issues.apache.org/jira/browse/SPARK-59698
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming
> Affects Versions: 4.3.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Blocker
> Labels: pull-request-available
>
> ASOF JOIN is going to be introduced in Apache Spark 4.3.0. It's feature
> complete for static-static ASOF JOIN, but it doesn't look like we audited
> with streaming.
> From the definition of ASOF JOIN, the join should find the best match among
> multiple matches (by time). This requires the left side to wait for the right
> side to have all the data for a certain time, and when stream is involved,
> this requires event time processing semantic. This requires watermark to be
> set, and the operator implementation must be stateful - left side has to be
> buffered and waited for the time watermark passes the event time, so that we
> know right side has all the data for that event time.
> This is NOT considered and Spark allows *-stream ASOF JOIN which is incorrect
> (it only matches with data within microbatch). We must disallow the case.
> In addition, stream-static ASOF JOIN can be accepted with the same semantic
> of stream-static join but we do not have any test with it. We should also
> address test coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
