hubgeter commented on code in PR #65851:
URL: https://github.com/apache/doris/pull/65851#discussion_r3670704478
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -530,30 +594,646 @@ void enableCurrentIcebergScanSemantics() {
params.setIcebergScanSemanticsVersion(ICEBERG_SCAN_SEMANTICS_VERSION);
}
+ /**
+ * Build the schema metadata carrier used by both scanners and
equality-delete readers.
+ *
+ * <p>Batch-mode delete files are planned asynchronously after scan
parameters are sent to BE.
+ * The authenticated manifest preflight therefore supplies the live
equality field IDs before
+ * the schema carrier is serialized. Only historical fields referenced by
those delete files are
+ * added, so an unrelated dropped type cannot make an otherwise supported
scan fail.
+ */
+ @VisibleForTesting
+ List<NestedField> getSchemaFieldsForScan(
+ Schema scanSchema, Set<Integer> equalityDeleteFieldIds) throws
UserException {
+ List<NestedField> fields = new ArrayList<>(scanSchema.columns());
+ if (isSystemTable || equalityDeleteFieldIds.isEmpty()) {
+ return fields;
+ }
+
+ Set<Integer> missingFieldIds = new HashSet<>(equalityDeleteFieldIds);
+
missingFieldIds.removeAll(TypeUtil.indexById(scanSchema.asStruct()).keySet());
Review Comment:
Fixed in 1d0cd97b398. FE now retains the selected historical ancestor tree
instead of appending a nested leaf as a synthetic root. Both V1 and V2 resolve
equality IDs recursively, read the containing top-level struct, and materialize
the primitive key with ancestor and leaf null maps. The focused BE suite covers
current and dropped nested keys in both Parquet and ORC, plus nullable-parent
and required-parent paths.
##########
be/src/format/transformer/vorc_transformer.cpp:
##########
@@ -318,6 +320,30 @@ std::unique_ptr<orc::Type>
VOrcTransformer::_build_orc_type(
}
}
if (nested_field != nullptr) {
+ const PrimitiveType primitive_type = data_type->get_primitive_type();
+ const auto use_iceberg_binary_type = [&](std::string_view binary_type)
{
+ DORIS_CHECK(is_string_type(primitive_type) ||
is_varbinary(primitive_type) ||
+ primitive_type == TYPE_BINARY);
+ type = orc::createPrimitiveType(orc::BINARY);
+ type->setAttribute(ICEBERG_BINARY_TYPE, std::string(binary_type));
+ };
+ switch (nested_field->field_type()->type_id()) {
+ case iceberg::TypeID::UUID:
+ use_iceberg_binary_type("UUID");
+ break;
Review Comment:
Fixed in 1d0cd97b398. ORC writes now recursively normalize Iceberg binary
fields before serialization: legacy UUID STRING values in raw 16-byte, 32-hex,
or 36-character form become the required 16 bytes, while FIXED values are
validated against the schema length. Tests write an explicit nested
36-character UUID, read back the exact 16 bytes, verify FIXED bytes, and reject
invalid UUID/FIXED values.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -505,22 +525,66 @@ private List<String> getOrderedPathPartitionKeys() {
}
public void createScanRangeLocations() throws UserException {
- super.createScanRangeLocations();
+ Schema scanSchema = getQuerySchema();
+ Optional<Map<Integer, List<String>>> nameMapping =
extractNameMapping();
+ Set<Integer> equalityDeleteFieldIds = Collections.emptySet();
+ if (!isSystemTable) {
+ checkNameMappingBackendCompatibility(
+ scanSchema, nameMapping, backendPolicy.getBackends());
Review Comment:
Fixed in 1d0cd97b398. The collision fence now collects both collision
participants and intersects them with projected slots and nested access paths,
while also walking hidden equality-key ancestors. FE tests cover unrelated
projection, a safe nested access path, a colliding projected path, and a hidden
equality dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
