sepuri sai krishna created FLINK-40924:
------------------------------------------
Summary: table.exec.sink.upsert-materialize=NONE silently bypasses
the require-on-conflict check
Key: FLINK-40924
URL: https://issues.apache.org/jira/browse/FLINK-40924
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.3.0, 2.4.0
Reporter: sepuri sai krishna
{{table.exec.sink.require-on-conflict}} defaults to {{true}} and is documented
to throw when the
query's upsert key differs from the sink's primary key and no {{ON CONFLICT}}
clause is given.
Setting {{table.exec.sink.upsert-materialize}} to {{NONE}} suppresses that
error, and the query
plans with no materializer.
h3. Reproducer
The upsert key is the grouping key {{c}}, which is never written to the sink,
so it cannot equal
the primary key {{x}}.
{code:sql}
CREATE TABLE src (a INT, b BIGINT, c STRING)
WITH ('connector' = 'datagen', 'number-of-rows' = '5');
CREATE TABLE snk (x INT, y BIGINT, PRIMARY KEY (x) NOT ENFORCED)
WITH ('connector' = 'blackhole');
SET 'table.exec.sink.upsert-materialize' = 'NONE';
INSERT INTO snk SELECT MAX(a), COUNT(*) FROM src GROUP BY c;
{code}
With {{require-on-conflict}} left at its default of {{true}}:
|| upsert-materialize || result ||
| {{AUTO}} (default) | {{ValidationException}}: "The query has an upsert key
that differs from the primary key of the sink table ... Please specify an ON
CONFLICT clause" |
| {{FORCE}} | plans, {{upsertMaterialize=[true]}} |
| {{NONE}} | plans, {{upsertMaterialize=[false]}} |
Only {{AUTO}} raises the documented error.
h3. Why this looks unintended
The two options document separate concerns, and neither documents this
interaction.
{{require-on-conflict}} documents exactly one way to turn itself off:
{quote}
Set this to false to restore the old behavior where no ON CONFLICT clause was
required. Note that
disabling this check may lead to non-deterministic results in certain streaming
scenarios.
{quote}
{{upsert-materialize}} is described entirely in terms of the materialize
operator and shuffle
disorder, and says nothing about validation or {{ON CONFLICT}}.
So a user who sets {{NONE}} has opted out of materialization, not out of the
check. The
{{FORCE}} case also skips the error but still materializes, so its result stays
deterministic;
{{NONE}} skips the error and does not materialize.
h3. Affects
{{master}} and {{release-2.3}}, where the relevant code is unchanged.
Related to FLINK-40899, which reports the opposite direction on the same
validation: there it
fires too early and masks an NDU error. This one is the check not firing at all.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
