Gustavo de Morais created FLINK-41423:
-----------------------------------------
Summary: SinkUpsertMaterializer compares upsert keys with SQL
equality instead of bytes and drops rows
Key: FLINK-41423
URL: https://issues.apache.org/jira/browse/FLINK-41423
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner, Table SQL / Runtime
Reporter: Gustavo de Morais
{{SinkUpsertMaterializer}} matches records by upsert key with a generated
{{RecordEqualiser}} ({{SinkMaterializeUpsertKeyEqualiser}} in
{{StreamExecSink}}). Unless both rows are {{BinaryRowData}}, the equaliser
compares fields with SQL {{=}} semantics ({{EqualiserCodeGenerator}}):
DOUBLE/FLOAT with {{==}} ({{-0.0 = 0.0}}), MAP and MULTISET ignoring the entry
order, ROW field by field. Keyed state and shuffles compare bytes, so two
different upsert keys are treated as the same record and a delete for one
removes the other.
Reproducer on master, no join needed:
{code:sql}
CREATE TABLE r (id INT NOT NULL, k DOUBLE NOT NULL, v STRING NOT NULL, name
STRING,
PRIMARY KEY (id, k) NOT ENFORCED) WITH ('connector' = 'values',
'changelog-mode' = 'I,UB,UA,D');
CREATE TABLE sink (id INT NOT NULL, k DOUBLE NOT NULL, v STRING NOT NULL, name
STRING,
PRIMARY KEY (id, v) NOT ENFORCED) WITH ('connector' = 'values',
'sink-changelog-mode-enforced' = 'I,UA,D');
INSERT INTO sink SELECT id, k, v, name FROM r ON CONFLICT DO DEDUPLICATE;
{code}
Input: {{+I(1, 0.0, v, r1)}}, {{+I(1, -0.0, v, r2)}}, {{-D(1, 0.0, v, r1)}}.
Expected sink content: {{+I(1, -0.0, v, r2)}}. Actual: empty. The same happens
with {{k}} as {{ROW<d DOUBLE>}} ({{ROW(0.0)}} vs {{ROW(-0.0)}}), {{MAP<INT,
STRING>}} (the same entries in a different order) and {{MULTISET<STRING>}}.
With keys that differ under {{=}} too, e.g. {{ROW(1.0)}} vs {{ROW(2.0)}},
{{r2}} is kept as expected.
FLINK-38753 makes more queries reach this, since it finds upsert keys in more
plans.
We should compare upsert keys byte-wise in the materializer, consistent with
keyed state.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
