morningman commented on code in PR #69845:
URL: https://github.com/apache/doris/pull/69845#discussion_r4419828177
##########
be/src/format_v2/table/iceberg_reader.cpp:
##########
@@ -1961,7 +2079,16 @@ Status
IcebergTableReader::_append_equality_delete_predicates(format::FileScanRe
} else {
const auto* root = data_path.front();
const auto field_column_id =
format::LocalColumnId(root->file_local_id());
- _append_file_scan_column(request, field_column_id,
&request->predicate_columns);
+ // A hidden key has no ColumnMapping to carry the timestamp
marker that
+ // annotate_file_schema() put on the file column. Without it
Parquet decodes an
+ // INT96 timestamptz key as DATETIMEV2 while the key
expression below, built from
+ // the annotated file schema, expects TIMESTAMPTZ. A projected
key moves into the
+ // predicate columns here and must keep the marker its mapping
carried.
+ auto projection =
format::LocalColumnIndex::top_level(field_column_id);
+
format::attach_timestamp_semantics(root->timestamp_is_adjusted_to_utc,
+ root->children,
&projection);
Review Comment:
Fixed in 4aa737fb648. The projection that the hidden-key path built
(`top_level` plus `attach_timestamp_semantics` from the annotated file column)
is now `equality_delete_root_projection()`, and
`_build_missing_equality_delete_key_expr` uses it to add the missing key's
ancestor as a predicate column, the same way the hidden key and the delete
file's keys are added. The failure was wider than `SELECT id`. When the query
projects `payload` (or `SELECT *`), the mapper's non-predicate projection of
`payload` does carry the marker, but adding the root as a predicate column for
the ancestor read replaced it (`FileScanRequestBuilder::_add_column` drops the
non-predicate entry), so those queries hit the same `check_block_column_type`
error. The key does not need a default either: the ancestor is read whatever
the default is. The new
`IcebergV2ReaderTest.IcebergMissingNestedEqualityKeyAncestorKeepsInt96TimestampTzMarker`
writes an older file with a nullable `payload {existing INT, ts INT96
timestamptz}` and deletes on the later-added `payload.k` (default 7) in
Asia/Shanghai. With `payload` hidden or projected, the only survivor is id 2,
the row under the NULL parent. With a tombstone that matches nothing, all three
rows survive and the projected `ts` reads as the instant. A plan without the
marker reads as before. Reverting only the ancestor read to
`_append_file_scan_column` makes the new test fail at its first marked case
with that error (`Parquet column payload ... is read as
Nullable(Struct(existing:Nullable(INT), ts:Nullable(DateTimeV2(6)))) but its
file block column is Nullable(Struct(existing:Nullable(INT),
ts:Nullable(TimeStampTz(6))))`), after the unmarked case passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
