morningman opened a new pull request, #68874:
URL: https://github.com/apache/doris/pull/68874
### What problem does this PR solve?
Issue Number: None
Related PR: #65868
Problem Summary:
**In short.** This ports the static-partition part of branch-4.1 #65868 to
the Paimon connector. An `INSERT OVERWRITE ... PARTITION (...)` named the
Paimon partition by the literal as the user wrote it, while the rows carry that
literal cast to the column type. Overwrites whose literal is not spelled the
column type's way failed, and overwrites of a TIMESTAMP WITH LOCAL TIME ZONE
partition failed whenever the session zone differed from the FE JVM zone; when
no rows were written to trip Paimon's check, the overwrite silently deleted the
wrong partition. Connectors now get the cast values through a new write-handle
getter, and Paimon uses them.
**Background.** On a Paimon table, the bind step materializes the literal of
`PARTITION (col = <literal>)` into every row with `castIfNotSameType`, so the
rows carry the value in the column's type. The static partition spec handed to
connectors is built separately: `PluginDrivenInsertCommandContext` keeps
`Literal.getStringValue()` of the literal as written, and `PaimonWriteBinding`
passed that string to Paimon's `withOverwrite`. Paimon parses the string with
the column type to find the partition to replace, parses an LTZ value as local
time in the FE JVM zone, and reads the table's `partition.default-name` as the
NULL partition. branch-4.1 #65868 casts each literal to its column type first,
moves an LTZ value from the session zone to the JVM zone, and rejects a value
equal to `partition.default-name`; master has none of this.
**The problem, and what it cost.** On a local FE + BE with a filesystem
Paimon catalog, session zone Asia/Shanghai, FE JVM zone Europe/London:
| Statement | master |
|---|---|
| `INSERT OVERWRITE t PARTITION (p = true) VALUES (...)`, `p INT` | fails:
`NumberFormatException: For input string: "true"` |
| `... PARTITION (dt = 20240101) ...`, `dt DATE` | fails: `the changes in
dt=19723/ does not belong to this partition` (Paimon reads 20240101 as a day
number) |
| `... PARTITION (ts = '2024-01-15 08:30:45') ...`, LTZ `ts` | fails: `the
changes in ts=2024-01-15T00%3A30%3A45/ does not belong to this partition` |
| the same LTZ overwrite with no rows (`SELECT ... LIMIT 0`) | succeeds,
deletes the partition `2024-01-15 16:30:45` (session time) and keeps `08:30:45`
|
| an overwrite with no rows of `region = '<partition.default-name>'` |
succeeds, deletes the NULL partition |
| `INSERT INTO t PARTITION (dt = '2024-13-45') ...` | succeeds, writes `dt =
NULL` |
The two empty overwrites are silent data loss: the statement reports success
and the deleted partition is not the one the user named.
**How this PR fixes it.**
- `ConnectorWriteHandle#getCastStaticPartitionSpec` (new): the static
partition values cast to the types of their columns, keyed like
`getStaticPartitionSpec`, which still carries the literals as written. SQL NULL
keys stay in `getStaticPartitionNullKeys` only; a key that names no column (an
Iceberg partition field) keeps its literal. The default, for a handle that does
not cast, is the spec as written without its null keys.
- fe-core: `InsertIntoTableCommand` records the sink's bound target schema
on `PluginDrivenInsertCommandContext`, and the write handle computes the cast
values on demand with `checkedCastTo`, the cast constant folding applies to the
row value. Only a connector that asks pays for the cast and can fail on it, so
Iceberg, MaxCompute and Hive, which keep reading the spec as written, see no
change. A value that does not fit its column type fails the statement instead
of becoming NULL, as on branch-4.x.
- Paimon: `PaimonWriteBinding` resolves the static partition from the cast
values. An LTZ value is moved to the FE JVM zone and written with a space
between date and time, which Paimon's parser needs: a DATETIMEV2 value from the
session zone, and a TIMESTAMPTZ value (`enable.mapping.timestamp_tz`) from its
own UTC offset. An overwrite value equal to `partition.default-name` is
rejected. Rejecting LTZ values that fall into a DST overlap of the JVM zone,
which branch-4.2 adds with its TIMESTAMPTZ mapping work, is left to that port.
**Results.** The table above, with this PR:
| Statement | this PR |
|---|---|
| `PARTITION (p = true)` | replaces `p = 1` |
| `PARTITION (dt = 20240101)` | replaces `dt = 2024-01-01` |
| LTZ overwrite, DATETIMEV2 or TIMESTAMPTZ mapping | replaces the named
partition |
| LTZ overwrite with no rows | deletes the named partition only |
| `region = '<partition.default-name>'`, with or without rows | fails: `...
cannot be represented in a static overwrite`, no data changes |
| `INSERT INTO ... PARTITION (dt = '2024-13-45')` | fails: `date/datetime
literal [2024-13-45T00:00:00] is invalid` |
Original author: @suxiaogang223
### Release note
Fix INSERT OVERWRITE ... PARTITION on Paimon tables failing, or replacing
the wrong partition, when a static partition value is not spelled in its column
type (`p = true` on an INT column, `dt = 20240101` on a DATE column) or the
column is TIMESTAMP WITH LOCAL TIME ZONE. A static partition value equal to the
table's `partition.default-name`, or one that is not a valid value of its
column type, is now rejected.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_paimon_write_transaction` gains the INT, DATE, LTZ and
`partition.default-name` cases. It needs the Paimon docker environment and was
not run locally; the same statements on a local filesystem Paimon catalog give
the asserted rows with this PR and fail, or delete the wrong partition, on
master.
- [x] Unit Test
- `PaimonWriteBindingTest` ports the branch-4.x cases for the cast
value, the `partition.default-name` rejection and the LTZ zone move, and adds
the TIMESTAMPTZ mapping and SQL NULL. `PluginDrivenTableSinkBindingTest` checks
the cast values on the write handle, the session-zone forms of DATETIMEV2 and
TIMESTAMPTZ values, and that an invalid value fails only when cast.
`ConnectorWriteHandleTest` pins the SPI default, and the plugin surface
baseline is refreshed. Reading the spec as written, dropping the default-name
check, skipping either zone move, or skipping the cast in fe-core makes them
fail.
- The fe-connector-spi suite (147), the Paimon connector tests (683)
and `InsertIntoTableCommandTest` / `InsertOverwriteTableCommandTest` pass.
- [x] Manual test (add detailed scripts or steps below)
- The before/after tables above.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. Paimon static partition values are matched to the rows by
their cast value; an invalid static partition value, or one equal to
`partition.default-name` in an overwrite, is rejected.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
