Jungtaek Lim created SPARK-59698:
------------------------------------
Summary: ASOF JOIN isn't properly blocked for stream on the right
side
Key: SPARK-59698
URL: https://issues.apache.org/jira/browse/SPARK-59698
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.3.0
Reporter: Jungtaek Lim
ASOF JOIN is going to be introduced in Apache Spark 4.3.0. It's feature
complete for static-static ASOF JOIN, but it doesn't look like we audited with
streaming.
>From the definition of ASOF JOIN, the join should find the best match among
>multiple matches (by time). This requires the left side to wait for the right
>side to have all the data for a certain time, and when stream is involved,
>this requires event time processing semantic. This requires watermark to be
>set, and the operator implementation must be stateful - left side has to be
>buffered and waited for the time watermark passes the event time, so that we
>know right side has all the data for that event time.
This is NOT considered and Spark allows *-stream ASOF JOIN which is incorrect
(it only matches with data within microbatch). We must disallow the case.
In addition, stream-static ASOF JOIN can be accepted with the same semantic of
stream-static join but we do not have any test with it. We should also address
test coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
