hubgeter opened a new pull request, #69188:
URL: https://github.com/apache/doris/pull/69188
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Doris could not write an Iceberg partition by value when the identity
partition field had been renamed away from its source column (`ALTER TABLE t
ADD PARTITION KEY id AS id_identity`):
- `INSERT OVERWRITE TABLE t PARTITION (id = 2) (value, metric) VALUES ('b',
2)` failed with `Unknown partition column 'id' in table 't'. Available
partition columns: [id_identity]`, although `id` is the column Spark resolves
for a static partition clause and the one `SHOW CREATE TABLE` renders for this
table (`PARTITION BY LIST (\`id\`)`);
- `... PARTITION (id_identity = 2) (value, metric) ...` passed the
validation but then failed with `Column has no default value, column=id`:
`id_identity` is a partition FIELD name, not a table column, so the static
literal could not be materialized into any row column;
- `... PARTITION (id_identity = 2) SELECT id, value, metric ...` (no column
list) was accepted and wrote data files whose `id` column value disagreed with
their own partition value.
The two checks contradicted each other: the validation required a name that
the binder then rejected, so there was no usable spelling.
Fix:
- resolve a `PARTITION (col = val)` name to the identity partition field's
SOURCE column, case-insensitively — the identifier the binder can materialize
(`PluginDrivenExternalTable#materializeStaticPartitionValues`);
- reject a renamed partition FIELD name with the source column to use
instead, and list the usable source columns for an unknown name; non-identity
transforms keep their existing rejection;
- rewrite the static partition spec keys to partition FIELD names before the
BE sink (`static_partition_values`) and the commit row filter consume them, so
the thrift contract, the overwrite row filter and the BE writer stay unchanged.
Behavior is aligned with Spark:
- Spark's `Analyzer.partitionColumnNames` takes the identity transform's
field reference, so a static partition column is the SOURCE column (`PARTITION
(id = 2)`), not the renamed field name;
- `Analyzer.addStaticPartitionColumns` materializes that literal into the
source column and rejects the same column in an explicit insert column list —
Doris does the same after this change;
- the resulting overwrite is an `OverwriteFiles.overwriteByRowFilter(id =
2)` (`Iceberg SparkWrite`), which is exactly what Doris builds once the spec is
field-keyed.
Effect after the fix: `PARTITION (id = 2)` overwrites only that partition,
the new files carry `id = 2` next to partition value `2`, and Spark reads the
same rows; `PARTITION (id_identity = 2)` now reports `'id_identity' ... is a
partition field name, not a table column; write the identity partition by its
source column 'id' instead.` instead of a contradictory error or a corrupted
write. Tables whose partition field name already equals its source column are
unaffected.
### Release note
Fix `INSERT [OVERWRITE] ... PARTITION (col = val)` on Iceberg tables whose
identity partition field was renamed with `ADD PARTITION KEY ... AS ...`: the
static value is now written through the identity source column (`PARTITION (id
= 2)`), matching Spark, instead of reporting contradictory errors or writing
files whose data column disagreed with the partition value. Non-identity
transforms and unpartitioned tables keep their existing rejection.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Manual test detail: full `fe-connector-iceberg` module unit tests (93 test
classes) plus checkstyle pass locally with 0 failures on top of current master;
the new regression case (TC39) needs the external docker Iceberg environment to
run, so it is expected to be covered by the regression pipeline.
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
