Gabriel39 commented on code in PR #68481:
URL: https://github.com/apache/doris/pull/68481#discussion_r4129157212
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -2589,16 +2589,29 @@ private Split createIcebergSplit(FileScanTask
fileScanTask) throws UserException
}
split.setTableFormatType(TableFormatType.ICEBERG);
split.setTargetSplitSize(selectFeSplitSize(fileScanTask,
targetSplitSize));
- if (isPartitionedTable) {
- int specId = fileScanTask.file().specId();
+ // REPLACE or partition evolution can leave an unpartitioned table
with historical specs.
+ // Row-level deletes must retain each file's spec instead of
defaulting to historical spec 0.
+ int specId = dataFile.specId();
+ split.setPartitionSpecId(specId);
+ PartitionData partitionData = (PartitionData) dataFile.partition();
+ if (partitionData != null) {
Review Comment:
Fixed in 9eda6cafea. The finalized row-ID projection is now inspected once
and cached per scan. Ordinary data scans skip partition JSON serialization and
transmission; unpartitioned reads also skip the unused spec lookup. Per-file
spec IDs remain unconditional, and partition-pruning metadata is handled
independently. Position-delete system-table JSON is unaffected. Added a
regression covering currently partitioned tables and mixed historical/current
specs after dropping the last partition field, asserting no JSON in read scan
ranges while preserving spec IDs and pruning values. Existing delete metadata
tests now explicitly project the hidden row ID. The new read test failed before
the fix and passes now; all 217 selected Iceberg tests and FE Checkstyle pass.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java:
##########
@@ -2589,16 +2589,29 @@ private Split createIcebergSplit(FileScanTask
fileScanTask) throws UserException
}
split.setTableFormatType(TableFormatType.ICEBERG);
split.setTargetSplitSize(selectFeSplitSize(fileScanTask,
targetSplitSize));
- if (isPartitionedTable) {
- int specId = fileScanTask.file().specId();
+ // REPLACE or partition evolution can leave an unpartitioned table
with historical specs.
+ // Row-level deletes must retain each file's spec instead of
defaulting to historical spec 0.
+ int specId = dataFile.specId();
+ split.setPartitionSpecId(specId);
+ PartitionData partitionData = (PartitionData) dataFile.partition();
+ if (partitionData != null) {
PartitionSpec partitionSpec = icebergTable.specs().get(specId);
Preconditions.checkNotNull(partitionSpec, "Partition spec with
specId %s not found for table %s",
specId, icebergTable.name());
- PartitionData partitionData = (PartitionData)
fileScanTask.file().partition();
- if (partitionData != null) {
- split.setPartitionSpecId(specId);
+ try {
split.setPartitionDataJson(IcebergUtils.getPartitionDataJson(
partitionData, partitionSpec,
sessionVariable.getTimeZone()));
+ } catch (UnsupportedOperationException e) {
+ // Dropped source columns leave UNKNOWN types in historical
specs. Reads do not need
+ // this row-ID payload, but DML must never replace an
unrepresentable partition with NULL.
+ boolean requiresRowId = desc.getSlots().stream().anyMatch(slot
-> slot.getColumn() != null
+ &&
Column.ICEBERG_ROWID_COL.equalsIgnoreCase(slot.getColumn().getName()));
+ if (requiresRowId) {
+ throw new UserException("Cannot produce Iceberg row IDs
with unsupported partition types in spec "
Review Comment:
Fixed in 9eda6cafea. getSplits now finds and rethrows UserException through
the exception chain, retaining its actionable message, spec ID, rewrite
guidance, and original serializer cause. This also handles Hadoop doAs, which
wraps checked user exceptions in RuntimeException. The dropped-source-column
regression now exercises the public getSplits entry point instead of invoking
createIcebergSplit directly, and a separate test covers the real Hadoop
authentication wrapper. Both tests reproduced the RuntimeException/message-loss
issue before the fix and now pass. All 217 selected Iceberg tests and FE
Checkstyle pass. The external SQL expectations remain unchanged; the external
suite has not been executed end-to-end locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
