This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 6c9f53c789f [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
omit 8079812f33d [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit d1e03fb7682 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit 942a4dd041d [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit 15165e9e064 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit 0fbd57b3891 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit 5101a6c1c36 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit ff890431947 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit a680703c747 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 9820b0c10ba [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit 21eaf91db7a [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 9920780dca8 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit 215f1981e0f [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit e04c2490721 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit 397aaafe85b [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit fdde0cbced4 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit f1d3096ffa2 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 1cd38df75c3 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 9ba7af0969f [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit eb9ade27810 [refactor](catalog) Catalog spi 11 hive (#65473)
omit 7f1d90ace8b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit bbb3cc0d14a [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit abf3ca1795d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 3bdde9431ab [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit 6d8f0f2e8e0 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit 2be4f515239 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit 6cfce0cf880 [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit d8609b90ff1 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit 6dfce469d94 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 0590c0da1ef [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit 7e775663587 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 369b8a68cc1 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit 901ff505862 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 75702575c5d [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 7abe8fe70f4 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit a0460f200a0 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 481f54e94f1 [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit 7875720ed41 [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 8d007607460 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 35bbba558c2 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 8a94692baa0 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit 7878a5dc29f update P5 handoff and fix compile issue
omit 8b5cb0ddc89 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit 884a95f1b48 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit be8f0e622de [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit 637a4f624e9 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 8c7f0a4c718 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit c6617278765 [doc](connector) add project tracking system for catalog
SPI migration
add d921bebcd90 [fix](agg) Normalize equivalent multi-column distinct
counts (#65206)
add 70a82532325 [feature](iceberg) Support nested column schema change
(#65329)
add 576b0acac04 [fix](test) skip test_flight_record when the frontend is
not on the regression runner (#65933)
add 58e87343513 [fix](be) remove changing segment cache blocks to index
type (#65905)
add 187154675fe [improvement](be) Optimize projected fixed-width Parquet
predicate filtering (#65934)
add 60ba6aef642 [fix](table stream) min delta delete op should return
first op's before value (#65826)
add 098ebc1daf3 [doc](connector) add project tracking system for catalog
SPI migration
add 72a8da3b362 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add b6e616904bf [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 26cf1380ec5 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add b2c08ddba03 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add 9a22738c266 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add ce11f623264 update P5 handoff and fix compile issue
add 48dfc0ecf67 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 6e3448c99ac [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 4345be3c8fe [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add c1e172fd72a [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 0f082d1ceec [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 9ae627ac4ae [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 3074d372f40 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add 237e8a54179 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add c8a7af07230 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 13002b4c02d [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add cc325df1977 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 28768cc7d5e [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add aac18050712 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 50d26f7d16a [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add e59242f5b8f [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add fc0757fdaf9 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add c9d2544c60f [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 3d5e22df9d2 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add ac4a219083d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add fcd94f7bbfd [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 3e886cd543b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 16cd0ac321d [refactor](catalog) Catalog spi 11 hive (#65473)
add 5734baff0bf [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 309c5d4b6ee [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 52625cb00f6 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add 077f0e7a883 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add e83e1a48b83 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add 8bc8d060837 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 1d5496ec511 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 8cb3002d4a1 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add ee7eaae166e [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add 3eeb0b357bc [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add c5ffeaab050 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add aafab75ab74 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add f19cd4d7abe [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add c22513aea1e [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add c005cf8bcfd [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add af5129cb485 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add 9b2724d0728 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 2d93c14a835 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add f95657e0329 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add 67addbbb5cb [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add f1d808e3cf6 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6c9f53c789f)
\
N -- N -- N refs/heads/branch-catalog-spi (f1d808e3cf6)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/benchmark/parquet/parquet_benchmark_scenarios.h | 13 +
be/src/format_v2/parquet/parquet_profile.cpp | 12 +-
be/src/format_v2/parquet/parquet_profile.h | 8 +-
be/src/format_v2/parquet/parquet_scan.cpp | 49 +-
be/src/format_v2/parquet/reader/column_reader.cpp | 8 +-
be/src/format_v2/parquet/reader/column_reader.h | 14 +-
.../parquet/reader/native/column_chunk_reader.cpp | 46 +-
.../parquet/reader/native/column_chunk_reader.h | 35 +-
.../parquet/reader/native/column_reader.cpp | 37 +-
.../parquet/reader/native/column_reader.h | 26 +-
.../parquet/reader/native_column_reader.cpp | 43 +-
.../parquet/reader/native_column_reader.h | 15 +-
be/src/storage/iterator/block_reader.cpp | 13 +-
be/src/storage/segment/segment_writer.cpp | 12 -
.../format/table/iceberg/iceberg_reader_test.cpp | 26 +-
be/test/format_v2/parquet/native_decoder_test.cpp | 35 +-
.../parquet/parquet_benchmark_scenarios_test.cpp | 19 +
be/test/format_v2/parquet/parquet_scan_test.cpp | 133 ++++-
be/test/format_v2/table/iceberg_reader_test.cpp | 12 +-
.../block_reader_change_next_block_test.cpp | 483 ++++++++++++++++++
.../main/java/org/apache/doris/catalog/Column.java | 22 +
.../java/org/apache/doris/catalog/ColumnType.java | 5 +-
.../apache/doris/catalog/info/ColumnPosition.java | 3 +-
.../doris/connector/api/ConnectorCapability.java | 19 +-
.../doris/connector/api/ConnectorColumn.java | 48 +-
.../doris/connector/api/ConnectorTableOps.java | 50 ++
.../connector/api/ddl/ConnectorColumnPath.java | 117 +++++
.../doris/connector/iceberg/IcebergCatalogOps.java | 34 ++
.../doris/connector/iceberg/IcebergConnector.java | 3 +-
.../iceberg/IcebergConnectorMetadata.java | 162 ++++++
.../iceberg/IcebergNestedColumnEvolution.java | 414 +++++++++++++++
.../iceberg/IcebergNestedColumnEvolutionTest.java | 417 ++++++++++++++++
.../main/java/org/apache/doris/alter/Alter.java | 14 +-
.../java/org/apache/doris/analysis/ColumnPath.java | 92 ++++
.../java/org/apache/doris/common/FeNameFormat.java | 9 +-
.../org/apache/doris/datasource/CatalogIf.java | 43 ++
.../apache/doris/datasource/ExternalCatalog.java | 32 ++
.../converter/ConnectorColumnConverter.java | 6 +-
.../plugin/PluginDrivenExternalCatalog.java | 110 ++++
.../plugin/PluginDrivenExternalTable.java | 12 +
.../doris/nereids/parser/LogicalPlanBuilder.java | 146 ++++--
.../parser/LogicalPlanBuilderAssistant.java | 41 +-
.../apache/doris/nereids/parser/NereidsParser.java | 8 +-
.../nereids/rules/analysis/NormalizeAggregate.java | 26 +-
.../PushDownAggWithDistinctThroughJoinOneSide.java | 2 +-
.../trees/plans/commands/AlterTableCommand.java | 150 +++++-
.../trees/plans/commands/info/AddColumnOp.java | 34 +-
.../trees/plans/commands/info/AddColumnsOp.java | 4 +
.../plans/commands/info/ColumnDefinition.java | 112 ++++-
.../trees/plans/commands/info/DropColumnOp.java | 17 +-
.../plans/commands/info/ModifyColumnCommentOp.java | 22 +-
.../trees/plans/commands/info/ModifyColumnOp.java | 23 +-
.../trees/plans/commands/info/RenameColumnOp.java | 24 +-
.../org/apache/doris/nereids/types/DataType.java | 3 +-
.../apache/doris/nereids/types/StructField.java | 26 +-
.../apache/doris/nereids/util/AggregateUtils.java | 10 +-
.../apache/doris/nereids/util/SqlLiteralUtils.java | 96 ++++
.../doris/alter/SchemaChangeHandlerTest.java | 1 +
.../IcebergNestedSchemaEvolutionParserTest.java | 462 +++++++++++++++++
.../NormalizeMultiColumnDistinctCountTest.java | 125 +++++
.../rules/rewrite/PruneNestedColumnTest.java | 27 +-
.../plans/commands/AlterTableCommandTest.java | 273 ++++++++++
.../plans/commands/info/ColumnDefinitionTest.java | 10 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 21 +-
.../java/org/apache/doris/catalog/StructField.java | 19 +-
.../rebase-65329-nested-schema-port-2026-07-23.md | 62 +++
.../iceberg/iceberg_schema_change_ddl.out | 3 +-
.../test_iceberg_nested_schema_evolution_ddl.out | 12 +
...nested_schema_evolution_spark_doris_interop.out | 27 +
.../test_mow_min_delta_delete_before.out} | 10 +-
...olap_table_stream_mixed_history_incremental.out | 1 +
.../test_table_stream_query_comprehensive.out | 1 +
.../test_table_level_compaction_policy.groovy | 30 +-
.../suites/demo_p0/test_flight_record.groovy | 19 +-
.../sql-function/test_array_function.groovy | 2 +-
.../iceberg/iceberg_schema_change_ddl.groovy | 4 +-
...test_iceberg_nested_schema_evolution_ddl.groovy | 175 +++++++
...ted_schema_evolution_spark_doris_interop.groovy | 553 +++++++++++++++++++++
.../agg_strategy/distinct_agg_rewriter.groovy | 15 +-
.../table_stream_p0/test_min_delta_stream.groovy | 2 +-
.../test_mow_min_delta_delete_before.groovy | 125 +++++
81 files changed, 4999 insertions(+), 375 deletions(-)
create mode 100644
be/test/storage/iterator/block_reader_change_next_block_test.cpp
create mode 100644
fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorColumnPath.java
create mode 100644
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolution.java
create mode 100644
fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergNestedColumnEvolutionTest.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnPath.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/util/SqlLiteralUtils.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/IcebergNestedSchemaEvolutionParserTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/NormalizeMultiColumnDistinctCountTest.java
create mode 100644 plan-doc/tasks/rebase-65329-nested-schema-port-2026-07-23.md
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.out
create mode 100644
regression-test/data/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.out
copy regression-test/data/{ann_index_p0/ivf_pq_recall.out =>
table_stream_p0/test_mow_min_delta_delete_before.out} (51%)
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_ddl.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy
create mode 100644
regression-test/suites/table_stream_p0/test_mow_min_delta_delete_before.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
