Rion Williams created FLINK-41068:
-------------------------------------
Summary: LATERAL SNAPSHOT join stores build-side rows whose join
key can never match
Key: FLINK-41068
URL: https://issues.apache.org/jira/browse/FLINK-41068
Project: Flink
Issue Type: Improvement
Components: Table SQL / Runtime
Affects Versions: 2.3.0
Reporter: Rion Williams
{{LateralSnapshotJoinOperator}} materializes the whole build side.
{{processElement2}} buffers every build-side change in {{buildChangeBuffer}},
and {{applyBuildChanges}} then adds it to {{buildTableState}}, a multiset that
is only bounded by the state TTL. That includes rows with a NULL in a join key
field that isn't null-safe. Such rows can never match: the NULL check only runs
in {{JoinConditionWithNullFilters}} when a probe row is joined.
The state is keyed by the join key, so all NULL-key build rows end up in the
state of the same subtask and stay there for as long as the build-side table
contains them.
h3. Proposed change
The build side is never preserved, so no output depends on build rows whose key
can never match. {{processElement2}} can skip an INSERT or UPDATE_AFTER build
change when {{JoinConditionWithNullFilters#currentKeyCannotMatch}} (from
FLINK-41034) is true, before it is buffered.
Retractions need care:
* Rows stored by an earlier version must still be removed, so retractions of
NULL-key rows should still be applied. Removing a row that was never stored is
a no-op.
* But {{applyBuildChanges}} counts a retraction of a row that isn't stored in
{{numUnmatchedBuildRetractions}}. Retractions of rows whose key can never match
should not be counted, or the metric would report every NULL-key delete.
Optionally, probe rows whose key can never match don't need to be buffered
during the LOAD phase for an inner join. For a left outer join, the null-padded
row still has to be emitted in order with the other buffered probe rows of the
same key.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
