github-actions[bot] commented on code in PR #69032:
URL: https://github.com/apache/doris/pull/69032#discussion_r4608728224
##########
be/src/format/transformer/viceberg_parquet_writer.cpp:
##########
@@ -104,6 +149,9 @@ Status VIcebergParquetWriter::write(const Block& block) {
if (block.rows() == 0) {
return Status::OK();
}
+ ColumnNumbers column_numbers(block.columns());
+ std::iota(column_numbers.begin(), column_numbers.end(), 0);
+ RETURN_IF_ERROR(validate_spatial_wkb_inputs(block, column_numbers));
Review Comment:
[P1] Reject required spatial container NULLs before Arrow conversion
For an imported V3 Parquet target `items` whose LIST element is required
`STRUCT<g:GEOMETRY>`, `INSERT INTO dst(items) SELECT array(s) FROM src WHERE s
IS NULL` (`src.s` is a nullable STRUCT with the same spatial metadata) passes
FE validation because `s` is a runtime slot. This WKB-only check skips the NULL
STRUCT. `DataTypeStructSerDe` then calls [Arrow 24
StructBuilder::AppendNull](https://github.com/apache/arrow/blob/apache-arrow-24.0.0/cpp/src/arrow/array/builder_nested.h#L711-L717),
which pads `g` with a valid empty binary value; its [Parquet level
builder](https://github.com/apache/arrow/blob/apache-arrow-24.0.0/cpp/src/parquet/arrow/path_internal.cc#L708-L711)
ignores the bitmap of a required STRUCT. The writer can therefore commit a
present `g` containing zero-byte annotated WKB, and reading that column
subsequently fails WKB validation. Validate runtime null maps against the
complete Iceberg schema, including required intermediate containers, before
Arrow conversion and
cover rejection of this write. This differs from the existing nested-literal
validation thread: the NULL here is a runtime parent, and the invalid WKB is
synthesized after validation. Source tracing only; no Doris test was executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]