This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 420b7fa2bb9 [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
discard 3a019c024cf [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
discard 494f9c2a7b7 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
discard a87a844bfbb [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
discard a474d24b475 [fix](be) fail the query instead of aborting BE on a
missing projected column
discard deddc0064c4 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
discard 1ceb9462e30 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
discard 15979ee2efe [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
discard a2e1064819c [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
discard e2d4abd3c07 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
discard 463716cfae3 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
discard 35957447644 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
discard b530227e1d9 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
discard d8f364cf0b0 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
discard c3732f73686 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
discard 6521b547a5c [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
discard 61fcc7536c9 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
discard 2c474d99f72 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
discard 686862ed67e [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
discard ab4e88f9db0 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
discard d3f5dd50ef0 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
discard b1e54bdba3e [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
discard 4d53d6d8b85 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
discard b9eadcbbefe [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
discard 4693c3bbf61 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
discard ab869d515c5 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
discard 8a8a2b45ff5 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
discard f215043a2f8 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
discard cdd7347536c [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
discard 2a6754d7583 [refactor](catalog) Catalog spi 11 hive (#65473)
discard 6cbd3a92f3b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
discard 5a2b3216b3c [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
discard 29294d79e72 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
discard 9ff49338e62 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
discard 081a1ec5c95 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
discard c1455e2cf3d [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
discard e73fc75413c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
discard 8f24e16ee1e [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
discard a79f2a003d9 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
discard 8af09a5752d [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
discard 58bce85b0c6 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
discard 7c47d44a966 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
discard 222ce93f200 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
discard be2b9eba000 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
discard 0a5bc485c20 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
discard 976e9f5870a [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
discard 120700ee6cf [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
discard da60ca72f5d [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
discard c2d6982f858 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
discard 4f7240493bd [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
discard 8ebff3ab098 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
discard 90da92845fa update P5 handoff and fix compile issue
discard d6507f25257 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
discard 8c950ae51df [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
discard 178cd9eb33e [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
discard e6d22fc4f88 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
discard 8c2c8a902ea [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
discard 42a8f3b9170 [doc](connector) add project tracking system for catalog
SPI migration
add 3adf9c8c889 [Chore](Collaborator) Add Breakwater agent to
collaborators (#65980)
add 65492bc162c [fix](rewrite rule) Reject mismatched compare plan in
PullUpJoinFromUnionAll (#65472)
add 91d72f6efad # [fix](topn lazy materialization) Disable topn lazy
materialization on non-light-schema-change tables (#65415)
add a765b36ebc2 [improvement](parquet) Vectorize File Scanner V2 decode
hot paths (#65972)
add b759db26948 [fix](file) Revert split residual predicate ownership
(#65998)
add d3bf6b89bde [fix](cloud) Fix SchemaVariablesScanner crash due to
incomplete FE response (#65994)
add 2dbb35ed137 [doc](connector) add project tracking system for catalog
SPI migration
add 6cec4ec6bb4 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add 234d3e47160 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 1e88f04386d [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 5a8f496eef1 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add f1d28dd0867 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 2ddf31716e5 update P5 handoff and fix compile issue
add b75ebb5d028 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add f03770fb8d2 [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add 9e9f7c3f90d [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add 91e3694feee [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add f5d43da45fc [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 92ac46673c5 [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 341e0533723 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add d7daa3c6cdb [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 4e9443243fa [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 1cbd3963bb0 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 2304b5672dc [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add b8524b4d063 [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add 72bcb5b8fef [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 95d5623c993 [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add 48ad870404a [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add d0edf25ca38 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 93fa0d6fbf2 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add f8586e9671b [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 5c77077d593 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 823450a104d [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add f3665d78361 [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add b552c2a8083 [refactor](catalog) Catalog spi 11 hive (#65473)
add e0e93a59601 [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add 3c1b272f540 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add f7c428ec597 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add ea1c4371b69 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 1e0ec56b0cc [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add 31f9423c24c [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 21347cf0a92 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add 29d16fb7976 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add 484186e887f [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add db7537bed55 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 3cbfedab461 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add 21befabae13 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 19f349faa40 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 4c477d59b26 [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add d17544385e8 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add a0679eba473 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add afec33703d6 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add d862f77bdfe [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 15e613e4656 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add 76b6068624b [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add fcf003f4856 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add 9582857c790 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add f00a2e5fe10 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add 1976332dd24 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add 33b2b49bd4f [fix](be) fail the query instead of aborting BE on a
missing projected column
add d76d939ad36 [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add 2e14fcf5202 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add c7b10e61ffc [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add 91b157990ce [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
add e717b560da7 [test](nereids) rebase-adapt PullUpJoinFromUnionAllTest to
the connector-migrated Hudi scan path
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (420b7fa2bb9)
\
N -- N -- N refs/heads/branch-catalog-spi (e717b560da7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
be/benchmark/benchmark_main.cpp | 1 +
be/benchmark/parquet/AGENTS.md | 54 +-
be/benchmark/parquet/README.md | 19 +
be/benchmark/parquet/benchmark_parquet_decoder.hpp | 187 +++++-
be/benchmark/parquet/benchmark_parquet_kernels.hpp | 243 ++++++++
be/benchmark/parquet/parquet_benchmark_scenarios.h | 71 +++
.../core/data_type_serde/parquet_decode_source.cpp | 78 +++
.../core/data_type_serde/parquet_decode_source.h | 8 +-
be/src/exec/operator/scan_operator.cpp | 20 +-
be/src/exec/operator/scan_operator.h | 9 +-
be/src/exec/scan/file_scanner_v2.cpp | 145 +----
be/src/exec/scan/file_scanner_v2.h | 33 +-
be/src/exec/scan/scanner.cpp | 16 +-
be/src/exec/scan/scanner.h | 8 -
be/src/exprs/vectorized_fn_call.cpp | 33 +-
be/src/format_v2/column_mapper.cpp | 69 +--
be/src/format_v2/column_mapper.h | 15 +-
be/src/format_v2/file_reader.h | 4 +-
.../reader/native/byte_stream_split_decoder.cpp | 23 +-
.../parquet/reader/native/column_chunk_reader.cpp | 8 +
.../parquet/reader/native/delta_bit_pack_decoder.h | 11 +-
be/src/format_v2/table/hudi_reader.cpp | 22 -
be/src/format_v2/table/hudi_reader.h | 2 -
.../iceberg_position_delete_sys_table_reader.cpp | 31 +-
be/src/format_v2/table/paimon_reader.cpp | 22 -
be/src/format_v2/table/paimon_reader.h | 2 -
be/src/format_v2/table_reader.cpp | 143 +----
be/src/format_v2/table_reader.h | 128 ++--
.../schema_variables_scanner.cpp | 9 +
.../segment/adaptive_block_size_predictor.cpp | 7 +-
.../segment/adaptive_block_size_predictor.h | 1 -
be/src/util/simd/parquet_kernels.cpp | 651 +++++++++++++++++++++
be/src/util/simd/parquet_kernels.h | 48 ++
be/test/exec/scan/file_scanner_v2_test.cpp | 35 +-
be/test/exec/scan/scanner_late_arrival_rf_test.cpp | 57 +-
...eberg_position_delete_sys_table_reader_test.cpp | 73 ---
be/test/format_v2/column_mapper_test.cpp | 68 +--
be/test/format_v2/parquet/native_decoder_test.cpp | 8 +-
.../parquet/parquet_benchmark_scenarios_test.cpp | 64 +-
.../parquet/parquet_simd_kernels_test.cpp | 222 +++++++
be/test/format_v2/table/hudi_reader_test.cpp | 147 -----
be/test/format_v2/table/iceberg_reader_test.cpp | 11 +
be/test/format_v2/table/paimon_reader_test.cpp | 146 -----
be/test/format_v2/table_reader_test.cpp | 340 +----------
.../segment/adaptive_block_size_predictor_test.cpp | 12 -
.../catalog/stream/OlapTableStreamWrapper.java | 4 +
.../post/materialize/MaterializeProbeVisitor.java | 36 +-
.../rules/rewrite/PullUpJoinFromUnionAll.java | 11 +-
.../plans/logical/LogicalCatalogRelation.java | 42 ++
.../trees/plans/logical/LogicalFileScan.java | 29 +
.../trees/plans/logical/LogicalOdbcScan.java | 5 +
.../trees/plans/logical/LogicalOlapScan.java | 25 +
.../plans/logical/LogicalOlapTableStreamScan.java | 25 +
.../trees/plans/logical/LogicalSchemaScan.java | 13 +
.../trees/plans/logical/LogicalTestScan.java | 5 +
.../java/org/apache/doris/nereids/util/Utils.java | 7 +
.../materialize/MaterializeProbeVisitorTest.java | 2 +
.../rules/rewrite/PullUpJoinFromUnionAllTest.java | 515 ++++++++++++++++
.../topn_lazy/topn_lazy_light_schema_change.out} | 12 +-
.../topn_lazy/topn_lazy_light_schema_change.groovy | 119 ++++
61 files changed, 2642 insertions(+), 1513 deletions(-)
create mode 100644 be/benchmark/parquet/benchmark_parquet_kernels.hpp
create mode 100644 be/src/core/data_type_serde/parquet_decode_source.cpp
create mode 100644 be/src/util/simd/parquet_kernels.cpp
create mode 100644 be/src/util/simd/parquet_kernels.h
create mode 100644 be/test/format_v2/parquet/parquet_simd_kernels_test.cpp
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/PullUpJoinFromUnionAllTest.java
copy regression-test/data/{ann_index_p0/scalar_quantization.out =>
query_p0/topn_lazy/topn_lazy_light_schema_change.out} (53%)
create mode 100644
regression-test/suites/query_p0/topn_lazy/topn_lazy_light_schema_change.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
