morningman opened a new pull request, #66344:
URL: https://github.com/apache/doris/pull/66344
### What problem does this PR solve?
Related Issue: #65615
Problem Summary:
Two places where Doris says one thing about an Arrow value and does another.
Both were found
while reading Doris over Arrow Flight SQL, and both reproduce on master on
their own.
**1. `CommandGetTables` describes types the batches do not carry**
`FlightSqlSchemaHelper.getArrowType` is documented as mirroring
`convert_to_arrow_type` in the
backend -- the schema a client reads from `GetTables` is what it types its
columns by. Two of the
mappings had drifted from what BE actually writes:
- A `DATEV2` column was described as `Date(MILLISECOND)` -- Arrow date64 --
while BE writes
`arrow::Date32Type`, a day number. A client that trusts the schema renders
and compares the
column as a datetime and then fails on the first batch with
`not support convert to datetimev2 from arrow type: 16`.
- A complex column was described with a placeholder child: an array's
element type was the Null
type, a map's pair was a bare list, a struct had no fields at all. An
Arrow ARRAY/MAP/STRUCT
carries its element types in its children and nowhere else, so this said
"array of nothing"
about every array in the catalog while BE emitted `ListType(item)`,
`MapType(key, value)` and
`StructType(fields)` in the data.
`describeTables` already reports the whole tree --
`Column.createChildrenColumn` even names an
array's element `item` and a map's pair `key`/`value`, which is what Arrow
calls them -- so the
children are now built from it, recursively. A descriptor that reports no
children keeps the old
placeholders: a source that cannot describe its nested types is no worse off
than before.
**2. A TIMESTAMPTZ column read from Arrow was decoded by copying its bits**
`DataTypeTimeStampTzSerDe` never overrode `read_column_from_arrow`, so the
one it inherited from
`DataTypeNumberSerDe<TYPE_TIMESTAMPTZ>` ran: its fixed-width path `memcpy`s
the array's buffer
straight into the column. Arrow hands it int64 EPOCH values; the column
stores PACKED date/time
values. Both are eight bytes wide, so the width check passes, the scan
succeeds, and every row is
silently wrong -- a Doris datetime read back over Arrow Flight SQL rendered
as `+08:05` with no
date in it.
This is reachable today through the `remote_doris` catalog. Its schema is
the remote table's
`Column` objects deserialized verbatim
(`RemoteDorisRestClient.parseColumns`), so a remote
TIMESTAMPTZ column stays TIMESTAMPTZ locally; `RemoteDorisScanNode` plans
the scan as
`FORMAT_ARROW`; and `remote_doris_reader` materializes it through this
serde. The remote side
writes the epochs with the serde's own `write_column_to_arrow`, so both
halves of that round trip
are Doris code and the mismatch is entirely internal.
The other Arrow readers -- `arrow_stream_reader` for the `arrow` file format,
`python_udtf_function`, `paimon_cpp_reader` -- share the same serde and are
covered by the same
change; whether each of them can present a TIMESTAMPTZ column today is not
something this PR
claims either way.
The reader converts by unit into the microseconds the column stores. NANO is
divided rather than
refused, since no Doris datetime type keeps sub-microsecond digits, and the
division floors so a
pre-1970 instant does not move forward by one microsecond. The value is read
as an instant on the
UTC line, the inverse of what `write_column_to_arrow` emits, so the round
trip is exact.
Two cases are deliberate. A null slot is not converted: the bytes under it
are whatever the source
left there, and running a garbage epoch through the range check would fail a
batch whose rows are
all well-formed. An Arrow type this serde cannot decode is an error rather
than a fallback --
accepting anything eight bytes wide is how the corruption above stayed
invisible.
### Release note
Fix two Arrow type-fidelity bugs.
`GetTables` over Arrow Flight SQL reported `DATEV2` as date64 while the data
is date32, and
described `ARRAY`/`MAP`/`STRUCT` columns with placeholder children instead
of their real element
types; a client that types its columns from that schema failed on the first
batch.
A `TIMESTAMPTZ` column materialized from Arrow was decoded by copying the
epoch integers over the
column's packed values -- same width, no error, every row wrong. This
affects any reader that
builds a TIMESTAMPTZ column from Arrow, including the native Paimon reader.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes.
- An Arrow Flight SQL client now receives, for `DATEV2` and for
nested types, the Arrow
types the batches actually carry. A client that had worked around
the old schema by
ignoring it is unaffected; one that trusted it stops failing.
- A `TIMESTAMPTZ` column read from Arrow now holds the instant the
source sent instead of
a reinterpreted epoch. Values previously returned on those paths
were wrong, so results
change -- to the correct ones.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
