This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-catalog-spi
in repository https://gitbox.apache.org/repos/asf/doris.git
omit cdbcc7f164e [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
omit 23b06e04767 [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
omit f1d808e3cf6 [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
omit 67addbbb5cb [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
omit f95657e0329 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
omit 2d93c14a835 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
omit 9b2724d0728 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
omit af5129cb485 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
omit c005cf8bcfd [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
omit c22513aea1e [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
omit f19cd4d7abe [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
omit aafab75ab74 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
omit c5ffeaab050 [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
omit 3eeb0b357bc [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
omit ee7eaae166e [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
omit 8cb3002d4a1 [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
omit 1d5496ec511 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
omit 8bc8d060837 [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
omit e83e1a48b83 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
omit 077f0e7a883 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
omit 52625cb00f6 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
omit 309c5d4b6ee [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
omit 5734baff0bf [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
omit 16cd0ac321d [refactor](catalog) Catalog spi 11 hive (#65473)
omit 3e886cd543b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
omit fcd94f7bbfd [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
omit ac4a219083d [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
omit 3d5e22df9d2 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
omit c9d2544c60f [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
omit fc0757fdaf9 [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
omit e59242f5b8f [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
omit 50d26f7d16a [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
omit aac18050712 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
omit 28768cc7d5e [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
omit cc325df1977 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
omit 13002b4c02d [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
omit c8a7af07230 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
omit 237e8a54179 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
omit 3074d372f40 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
omit 9ae627ac4ae [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
omit 0f082d1ceec [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
omit c1e172fd72a [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
omit 4345be3c8fe [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
omit 6e3448c99ac [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
omit 48dfc0ecf67 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
omit ce11f623264 update P5 handoff and fix compile issue
omit 9a22738c266 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
omit b2c08ddba03 [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
omit 26cf1380ec5 [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
omit b6e616904bf [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
omit 72a8da3b362 [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
omit 098ebc1daf3 [doc](connector) add project tracking system for catalog
SPI migration
add ec5f97257ed [fix](variant) Prefer sparse variant fields over root
value (#65660)
add 07bd92a3b7e [fix](stream) Fix stream scan partition prune state
propagation (#65657)
add c345e92d93d [feature](fe) Add meta service RPC rate limiting in FE
(#65694)
add 7c922d5d2ea [fix](file-cache): speed up LRU restore startup path
(#65174)
add 67489f6e006 [fix](build) Fix insecure Python dependency pins in
doris-compose, dbt-doris, and cost_model (#64928)
add 8a1bf788e29 [fix](replica) Resolve conflicting default replica
properties (#65836)
add bd16e5d68fd [fix](fe) Guard auto partition result against concurrent
drops (#65282)
add 27cb451229c [improvement](file) Execute split residual predicates in
TableReader (#65925)
add e0d8db50e1a [fix](filecache) fix flaky be UT for LRU dump (#65427)
add 7038bb70bf2 [opt](expr) Vectorize AVG serialized state merge (#65729)
add 8a76160b646 [feature](streaming-job) Support OceanBase CDC streaming
jobs (#65588)
add 805b746addd [fix](regression) use JDK 17 for Java UDF builds (#65888)
add b80247f1200 [fix](file) Resolve remaining FileScannerV2 audit issues
(#65931)
add 14aaf84f9f7 [test](regression) Complete Iceberg/Paimon schema time
travel P0 coverage (#65960)
add 60e56ec867a [fix](fe) Cancel rebuilt VCG warm up jobs on drop (#65426)
add 488242f4eb5 [improvement](nereids) Support CTE split for DISTINCT
aggregate functions (#65664)
add 72989876caa [improvement](parquet) Lazily materialize complex residual
columns (#65965)
add f1460f89230 [feat](fs) Add GCS, MinIO and Ozone filesystem plugins on
a shared S3 base (#65834)
add 42a8f3b9170 [doc](connector) add project tracking system for catalog
SPI migration
add 8c2c8a902ea [feat](connector) P0 SPI baseline + DDL/Partition + import
gate (T03-T27) (#63582)
add e6d22fc4f88 [P1-T03-T05] route plugin-driven scans first in nereids
translator (#63641)
add 178cd9eb33e [feat](connector) P2 migrate trino-connector to catalog
SPI (T01-T13) (#64096)
add 8c950ae51df [feat](connector) P3 hudi connector hardening + test
baseline + dispatch design (hybrid, T02-T08) (#64143)
add d6507f25257 [refactor](connector) P4 maxcompute: remove legacy
subsystem from fe-core + make fe-core odps-free (T07-T09) (#64300)
add 90da92845fa update P5 handoff and fix compile issue
add 8ebff3ab098 [refactor](catalog) P5 paimon: migrate to catalog SPI +
cutover (#64446) (#64446)
add 4f7240493bd [refactor](catalog) P5 paimon: remove legacy subsystem
from fe-core + make fe-core paimon-SDK-free (T29) (#64653)
add c2d6982f858 [refactor](catalog) P3b: consolidate kerberos
authentication into fe-kerberos (#64655)
add da60ca72f5d [refactor](catalog) P6 iceberg: migrate to catalog SPI +
cutover + remove legacy fe-core subsystem (#64688)
add 120700ee6cf [fix](catalog) jdbc: keep driver classloaders alive per
URL to stop Metaspace leak
add 976e9f5870a [fix](catalog) P4 maxcompute: drop dangling legacy
MaxComputeExternalTableTest after rebase
add 0a5bc485c20 [fix](catalog) port upstream #65332/#65094 datasource
fixes into SPI connectors
add be2b9eba000 [fix](catalog) tier3 #65094: preserve external-table
top-level column-name case on the read path
add 222ce93f200 [feat](catalog) P6.6-C6 #63068 SPI: neutral
delegated-credential DTO + user-session capability
add 7c47d44a966 [feat](catalog) P6.6-C6 #63068 FE: capability-gated
credential injection + per-user cache bypass
add 58bce85b0c6 [feat](catalog) P6.6-C6 #63068 iceberg: per-user REST
session routing (metadata + scan/write/procedure)
add 8af09a5752d [doc](catalog) P6.6-C6 #63068 OIDC session migration:
design + research notes + HANDOFF (Tasks 0-7 DONE)
add a79f2a003d9 [feat](catalog) P6.6-C6 #63068 FE: re-migrate
ExternalDatabaseSessionContextTest (cross-user cache-leakage data-flow)
add 8f24e16ee1e [doc](catalog) P6.6-C6 #63068 HANDOFF: record
cache-leakage data-flow test + Task 0 retained-base verify
add e73fc75413c [fix](catalog) reconcile upstream #65365 with connector
SPI after rebase to 4f48ccfaa7d
add c1455e2cf3d [fix](build) fix parallel -T shade race in
fe-connector-paimon-hive-shade
add 081a1ec5c95 [doc](catalog) HANDOFF: rebase to upstream 30a44d61a8a +
register $position_deletes connector gap as flip blocker
add 9ff49338e62 [doc](catalog) P6.6 $position_deletes connector port:
research notes + design (T0 evidence in, ready to implement)
add 29294d79e72 [doc](catalog) HANDOFF: point next step at the
$position_deletes port design (T0 evidence in)
add 5a2b3216b3c [feat](catalog) P6.6: support the iceberg
$position_deletes system table on the connector SPI
add 6cbd3a92f3b [doc](catalog) HANDOFF: $position_deletes port done; e2e
is the only remaining flip gate
add 2a6754d7583 [refactor](catalog) Catalog spi 11 hive (#65473)
add cdd7347536c [refactor](catalog) fe-connector: replace fat
hive-catalog-shade with a slim HMS metastore-client shade (#65721)
add f215043a2f8 [refactor](catalog) fe-core: remove dead Trino-ported
DirectoryListercache cluster (#65734)
add 8a8a2b45ff5 [refactor](catalog) fe-core: remove dead
ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
stubs (#65736)
add ab869d515c5 [refactor](catalog) fe-core: remove 6 dead datasource
classes orphaned by the SPI migration (#65740)
add 4693c3bbf61 [refactor](catalog) drop
ReauthenticatingRestSessionCatalog test orphaned by master rebase
add b9eadcbbefe [feat](catalog) fe-connector-iceberg: port #64966 REST 401
re-auth to the connector
add 4d53d6d8b85 [fix](catalog) restore jfs hive-write hadoop config +
iceberg system-table scan projection
add b1e54bdba3e [fix](catalog) iceberg system-table scan: restore #65262
positional JNI read + order-preserving projection
add d3f5dd50ef0 [fix](catalog) port #65548 external COUNT(*)/COUNT(col)
semantics to the connector scan path
add ab4e88f9db0 [fix](catalog) port #65676 iceberg deletion-vector
metadata validation to the connector scan path
add 686862ed67e [perf](catalog) fe-connector-iceberg hot-path caching +
fe-core per-statement metadata funnel, with session=user cache isolation
add 2c474d99f72 [perf](catalog) two-level cross-query cache for external
partition derived views (#65829)
add 61fcc7536c9 [fix](catalog) port #65782 collect_column_stats sink flag
+ write-metrics tests to the iceberg connector
add 6521b547a5c [fix](catalog) port #65784 authoritative iceberg name
mapping to the connector scan path
add c3732f73686 [refactor](catalog) fe-core: strip residual
iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors,
remove hudi_meta TVF (#65893)
add d8f364cf0b0 [refactor](catalog) extract fe-trino-connector-common so
fe-common no longer depends on Trino
add b530227e1d9 [fix](catalog) migrate rebased-in
PhysicalStorageLayerAggregateTest to PluginDrivenExternalTable
add 35957447644 [fix](catalog) bundle fastutil-core into BE
java-extensions that eager-load ConcurrentLong2LongHashMap
add 463716cfae3 [fix](catalog) port #65904 paimon special-char partition
names into fe-connector
add e2d4abd3c07 [fix](catalog) reconcile upstream #65329 iceberg nested
column schema change with connector SPI (Tier 1: compile + flat DDL routing)
add a2e1064819c [feat](catalog) port upstream #65329 iceberg nested column
schema change to the connector SPI (Tier 2)
add 15979ee2efe [fix](catalog) catalog-SPI abstraction-mismatch review:
paimon time-travel crash, partition/schema fidelity, SPI convergence &
dead-code cleanup (#65970)
add 1ceb9462e30 [fix](catalog) preserve nested STRUCT field comment on
omitted COMMENT in complex MODIFY (build 1004182)
add deddc0064c4 [fix](paimon) read latest schema in getColumnHandles so a
renamed column is not dropped
add a474d24b475 [fix](be) fail the query instead of aborting BE on a
missing projected column
add a87a844bfbb [refactor](catalog) drop generic-S3 minio.* folding
superseded by upstream #65834's dedicated fe-filesystem-minio
add 494f9c2a7b7 [fix](iceberg) scan same-table multi-version references
against each reference's pinned snapshot schema
add 3a019c024cf [fix](paimon) read a branch's independent schema via its
own schema dir on the scan path
add 420b7fa2bb9 [test](paimon) align stale schema-evolution matrix tests
to shipped behavior
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cdbcc7f164e)
\
N -- N -- N refs/heads/branch-catalog-spi (420b7fa2bb9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
be/benchmark/parquet/AGENTS.md | 10 +-
be/benchmark/parquet/README.md | 15 +-
be/benchmark/parquet/benchmark_parquet_reader.hpp | 94 ++-
be/benchmark/parquet/parquet_benchmark_scenarios.h | 22 +-
be/src/common/compiler_util.h | 10 +-
be/src/core/column/column_variant.cpp | 5 +
be/src/exec/operator/scan_operator.cpp | 20 +-
be/src/exec/operator/scan_operator.h | 9 +-
be/src/exec/scan/file_scanner_v2.cpp | 145 +++-
be/src/exec/scan/file_scanner_v2.h | 33 +-
be/src/exec/scan/scanner.cpp | 16 +-
be/src/exec/scan/scanner.h | 8 +
be/src/exprs/aggregate/aggregate_function_avg.h | 6 +-
.../aggregate/aggregate_function_avg_weighted.h | 9 +-
be/src/exprs/aggregate/aggregate_function_sum.h | 5 +-
be/src/format/parquet/vparquet_reader.cpp | 13 +
be/src/format/table/table_schema_change_helper.h | 16 +
be/src/format_v2/column_mapper.cpp | 78 ++-
be/src/format_v2/column_mapper.h | 16 +-
be/src/format_v2/expr/cast.h | 3 +
be/src/format_v2/file_reader.h | 4 +-
be/src/format_v2/jni/jdbc_reader.cpp | 15 +
be/src/format_v2/jni/jdbc_reader.h | 2 +
be/src/format_v2/json/json_reader.cpp | 54 +-
be/src/format_v2/json/json_reader.h | 44 +-
be/src/format_v2/parquet/native_schema_desc.cpp | 16 +-
be/src/format_v2/parquet/native_schema_node.cpp | 35 +-
be/src/format_v2/parquet/parquet_profile.cpp | 3 +
be/src/format_v2/parquet/parquet_profile.h | 2 +
be/src/format_v2/parquet/parquet_scan.cpp | 274 ++++++--
be/src/format_v2/parquet/parquet_scan.h | 13 +-
.../parquet/reader/native/column_chunk_reader.cpp | 162 ++++-
.../parquet/reader/native/column_chunk_reader.h | 32 +-
.../parquet/reader/native/column_reader.cpp | 16 +-
.../parquet/reader/native/column_reader.h | 6 +
.../parquet/reader/native/delta_bit_pack_decoder.h | 6 +
.../parquet/reader/native_column_reader.cpp | 19 +-
.../parquet/reader/native_column_reader.h | 1 +
be/src/format_v2/table/hudi_reader.cpp | 29 +
be/src/format_v2/table/hudi_reader.h | 7 +
.../iceberg_position_delete_sys_table_reader.cpp | 32 +-
be/src/format_v2/table/iceberg_reader.h | 1 +
be/src/format_v2/table/paimon_reader.cpp | 29 +
be/src/format_v2/table/paimon_reader.h | 7 +
be/src/format_v2/table/remote_doris_reader.cpp | 185 +++++-
be/src/format_v2/table/remote_doris_reader.h | 2 +
be/src/format_v2/table_reader.cpp | 143 +++-
be/src/format_v2/table_reader.h | 131 ++--
be/src/io/cache/block_file_cache_factory.cpp | 137 +++-
be/src/io/cache/block_file_cache_factory.h | 5 +
be/src/io/cache/cache_lru_dumper.cpp | 18 +-
be/src/io/cache/cache_lru_dumper.h | 4 +-
be/src/runtime/exec_env_init.cpp | 28 +-
.../segment/adaptive_block_size_predictor.cpp | 7 +-
.../segment/adaptive_block_size_predictor.h | 1 +
be/test/core/column/column_variant_test.cpp | 27 +
be/test/exec/scan/file_scanner_v2_test.cpp | 35 +-
be/test/exec/scan/scanner_late_arrival_rf_test.cpp | 57 +-
...eberg_position_delete_sys_table_reader_test.cpp | 73 +++
.../table/table_schema_change_helper_test.cpp | 37 ++
be/test/format_v2/column_mapper_test.cpp | 96 ++-
be/test/format_v2/jni/jdbc_reader_test.cpp | 49 ++
be/test/format_v2/json/json_reader_test.cpp | 24 +-
be/test/format_v2/parquet/native_decoder_test.cpp | 381 ++++++++++-
.../parquet/parquet_benchmark_scenarios_test.cpp | 24 +-
be/test/format_v2/parquet/parquet_scan_test.cpp | 291 +++++++-
be/test/format_v2/parquet/parquet_schema_test.cpp | 112 ++++
be/test/format_v2/table/hudi_reader_test.cpp | 154 +++++
be/test/format_v2/table/iceberg_reader_test.cpp | 11 -
be/test/format_v2/table/paimon_reader_test.cpp | 153 +++++
.../format_v2/table/remote_doris_reader_test.cpp | 269 ++++++++
be/test/format_v2/table_reader_test.cpp | 340 +++++++++-
be/test/io/cache/block_file_cache_test.cpp | 67 ++
.../io/cache/block_file_cache_test_meta_store.cpp | 3 +-
be/test/io/cache/cache_lru_dumper_test.cpp | 42 ++
.../segment/adaptive_block_size_predictor_test.cpp | 12 +
build.sh | 4 +-
docker/runtime/doris-compose/requirements.txt | 2 +-
.../docker-compose/oceanbase/oceanbase.env | 1 +
.../docker-compose/oceanbase/oceanbase.yaml.tpl | 6 +-
extension/dbt-doris/dev-requirements.txt | 2 +-
.../main/java/org/apache/doris/common/Config.java | 33 +
.../MetaServiceRpcRateLimitConfigValidator.java | 103 +++
.../connector/paimon/PaimonConnectorMetadata.java | 53 +-
.../paimon/PaimonConnectorMetadataTest.java | 40 +-
fe/fe-core/pom.xml | 4 +
.../doris/catalog/InternalSchemaInitializer.java | 8 +-
.../org/apache/doris/catalog/TableProperty.java | 28 +-
.../cloud/catalog/CloudInstanceStatusChecker.java | 11 +-
.../apache/doris/cloud/rpc/MetaServiceProxy.java | 97 ++-
.../rpc/MetaServiceRateLimitException.java} | 12 +-
.../doris/cloud/rpc/MetaServiceRpcRateLimiter.java | 278 ++++++++
.../doris/cloud/system/CloudSystemInfoService.java | 5 +-
.../doris/common/profile/SummaryProfile.java | 44 +-
.../doris/datasource/scan/FileQueryScanNode.java | 25 +-
.../datasource/scan/PluginDrivenScanNode.java | 53 ++
.../apache/doris/job/common/DataSourceType.java | 3 +-
.../streaming/DataSourceConfigValidator.java | 79 ++-
.../insert/streaming/StreamingInsertJob.java | 19 +-
.../streaming/StreamingJdbcUrlNormalizer.java | 1 +
.../apache/doris/job/util/StreamingJobUtils.java | 15 +-
.../java/org/apache/doris/metric/CloudMetrics.java | 17 +
.../nereids/rules/analysis/NormalizeAggregate.java | 5 +-
.../nereids/rules/analysis/NormalizeRepeat.java | 3 +
.../rules/rewrite/DistinctAggStrategySelector.java | 13 +-
.../rules/rewrite/SplitMultiDistinctStrategy.java | 6 +-
.../expressions/functions/agg/CollectList.java | 15 +
.../functions/agg/ExponentialMovingAverage.java | 6 +
.../expressions/functions/agg/GroupBitmapXor.java | 11 +-
.../trees/expressions/functions/agg/Histogram.java | 5 +
.../trees/expressions/functions/agg/MapAgg.java | 2 +-
.../trees/expressions/functions/agg/MapAggV2.java | 2 +-
.../expressions/functions/agg/Percentile.java | 5 +
.../functions/agg/PercentileApprox.java | 5 +
.../functions/agg/PercentileApproxWeighted.java | 5 +
.../expressions/functions/agg/PercentileArray.java | 5 +
.../functions/agg/PercentileReservoir.java | 5 +
.../trees/expressions/functions/agg/TopN.java | 5 +
.../trees/expressions/functions/agg/TopNArray.java | 5 +
.../expressions/functions/agg/TopNWeighted.java | 5 +
.../plans/logical/LogicalOlapTableStreamScan.java | 4 +-
.../apache/doris/nereids/util/AggregateUtils.java | 11 +
.../apache/doris/service/FrontendServiceImpl.java | 438 +++++++------
.../doris/alter/InternalSchemaAlterTest.java | 5 +
.../apache/doris/catalog/TablePropertyTest.java | 86 +++
.../catalog/CloudInstanceStatusCheckerTest.java | 116 +++-
.../doris/cloud/rpc/MetaServiceProxyTest.java | 190 ++++++
.../cloud/rpc/MetaServiceRpcRateLimiterTest.java | 275 ++++++++
.../doris/common/profile/SummaryProfileTest.java | 30 +
.../scan/PluginDrivenScanNodePinnedSchemaTest.java | 95 +++
.../streaming/DataSourceConfigValidatorTest.java | 188 ++++++
.../streaming/StreamingJdbcUrlNormalizerTest.java | 10 +
.../doris/job/util/StreamingJobUtilsTest.java | 13 +
.../trees/plans/ExplainTableStreamPlanTest.java | 31 +
.../doris/service/FrontendServiceImplTest.java | 49 ++
.../pom.xml | 59 +-
.../src/main/assembly/plugin-zip.xml | 0
.../filesystem/gcs/GcsFileSystemProperties.java | 199 ++++++
.../filesystem/gcs/GcsFileSystemProvider.java | 85 +++
....apache.doris.filesystem.spi.FileSystemProvider | 31 +-
.../gcs/GcsFileSystemPropertiesTest.java | 196 ++++++
.../filesystem/gcs/GcsFileSystemProviderTest.java | 191 ++++++
.../pom.xml | 59 +-
.../src/main/assembly/plugin-zip.xml | 0
.../minio/MinioFileSystemProperties.java | 180 +++++
.../filesystem/minio/MinioFileSystemProvider.java | 82 +++
....apache.doris.filesystem.spi.FileSystemProvider | 31 +-
.../minio/MinioFileSystemPropertiesTest.java | 127 ++++
.../minio/MinioFileSystemProviderTest.java | 153 +++++
.../pom.xml | 59 +-
.../src/main/assembly/plugin-zip.xml | 0
.../ozone/OzoneFileSystemProperties.java | 181 +++++
.../filesystem/ozone/OzoneFileSystemProvider.java | 82 +++
....apache.doris.filesystem.spi.FileSystemProvider | 31 +-
.../ozone/OzoneFileSystemPropertiesTest.java | 138 ++++
.../ozone/OzoneFileSystemProviderTest.java | 109 +++
.../pom.xml | 53 +-
.../s3/AbstractDelegatingS3Properties.java | 246 +++++++
.../doris/filesystem/s3/S3CompatSignals.java | 270 ++++++++
.../s3/S3CredentialsProviderFactory.java | 0
.../filesystem/s3/S3CredentialsProviderType.java | 0
.../apache/doris/filesystem/s3/S3FileSystem.java | 8 +-
.../filesystem/s3/S3FileSystemProperties.java | 63 +-
.../apache/doris/filesystem/s3/S3ObjStorage.java | 15 +-
.../doris/filesystem/s3/S3CompatSignalsTest.java | 144 ++++
.../s3/S3CredentialsProviderFactoryTest.java | 0
.../doris/filesystem/s3/S3FileSystemEnvTest.java | 0
.../filesystem/s3/S3FileSystemPropertiesTest.java | 105 ---
.../doris/filesystem/s3/S3FileSystemTest.java | 0
.../doris/filesystem/s3/S3ObjStorageEnvTest.java | 0
.../doris/filesystem/s3/S3ObjStorageMockTest.java | 0
.../doris/filesystem/s3/S3ObjStorageTest.java | 0
fe/fe-filesystem/fe-filesystem-s3/pom.xml | 24 +-
.../doris/filesystem/s3/S3FileSystemProvider.java | 24 +-
.../filesystem/s3/S3FileSystemProviderTest.java | 103 ++-
fe/fe-filesystem/pom.xml | 4 +
fe/pom.xml | 6 +
.../mysql/MySqlStreamingChangeEventSource.java | 92 ++-
.../doris/cdcclient/source/factory/DataSource.java | 3 +-
.../source/factory/SourceReaderFactory.java | 2 +
.../source/reader/mysql/MySqlSourceReader.java | 9 +
.../reader/oceanbase/OceanBaseSourceReader.java | 15 +-
.../mysql/MySqlStreamingChangeEventSourceTest.java | 48 ++
.../cdcclient/itcase/CdcClientWriteHarness.java | 53 +-
.../itcase/OceanBaseSchemaChangeITCase.java | 121 ++++
.../itcase/OceanBaseStartupOffsetITCase.java | 148 +++++
.../doris/cdcclient/itcase/OceanBaseTestBase.java | 150 +++++
.../cdcclient/itcase/OceanBaseWriteDmlITCase.java | 112 ++++
.../source/factory/SourceReaderFactoryTest.java | 9 +
.../source/reader/mysql/MySqlSourceReaderTest.java | 31 +-
regression-test/conf/regression-conf.groovy | 3 +-
.../cdc/test_streaming_oceanbase_job.out | 17 +
.../cdc/test_streaming_oceanbase_job_all_type.out | 8 +
.../cdc/test_streaming_oceanbase_job_offset.out | 11 +
.../test_streaming_oceanbase_job_restart_fe.out | 5 +
.../test_streaming_oceanbase_job_sc_restart_fe.out | 11 +
.../test_streaming_oceanbase_job_schema_change.out | 9 +
.../agg_function/agg_distinct_function.out | 393 +++++++++++
.../test_topn_lazy_materialize_sparse_variant.out | 17 +
regression-test/java-udf-src/pom.xml | 7 +-
.../pipeline/external/conf/regression-conf.groovy | 1 +
.../suites/doc/external/jdbc/oceanbase.md.groovy | 2 +-
.../iceberg/test_iceberg_jdbc_catalog.groovy | 50 +-
...test_iceberg_schema_dual_relation_matrix.groovy | 174 +++++
...eberg_schema_equality_delete_time_travel.groovy | 145 ++++
...iceberg_schema_metadata_atomicity_matrix.groovy | 163 +++++
...t_iceberg_schema_position_dv_time_travel.groovy | 180 +++++
.../test_iceberg_schema_ref_actions_matrix.groovy | 242 +++++++
.../test_iceberg_schema_time_travel_matrix.groovy | 728 +++++++++++++++++++++
.../iceberg_paimon_schema_time_travel_coverage.md | 119 ++++
.../paimon/test_paimon_jdbc_catalog.groovy | 89 ++-
...st_paimon_schema_branch_partition_matrix.groovy | 225 +++++++
.../test_paimon_schema_dual_relation_matrix.groovy | 171 +++++
..._paimon_schema_metadata_atomicity_matrix.groovy | 170 +++++
.../test_paimon_schema_time_travel_matrix.groovy | 600 +++++++++++++++++
.../jdbc/test_oceanbase_jdbc_catalog.groovy | 4 +-
.../cdc/test_streaming_oceanbase_job.groovy | 143 ++++
.../test_streaming_oceanbase_job_all_type.groovy | 152 +++++
.../cdc/test_streaming_oceanbase_job_offset.groovy | 203 ++++++
.../test_streaming_oceanbase_job_restart_fe.groovy | 150 +++++
...st_streaming_oceanbase_job_sc_restart_fe.groovy | 156 +++++
...st_streaming_oceanbase_job_schema_change.groovy | 138 ++++
.../agg_function/agg_distinct_function.groovy | 212 ++++++
.../distinct_agg_strategy_selector.groovy | 39 ++
...est_topn_lazy_materialize_sparse_variant.groovy | 86 +++
run-regression-test.sh | 99 +--
tools/cost_model_evaluate/requirements.txt | 2 +-
227 files changed, 14400 insertions(+), 1227 deletions(-)
create mode 100644 be/test/format_v2/jni/jdbc_reader_test.cpp
create mode 100644
fe/fe-common/src/main/java/org/apache/doris/common/MetaServiceRpcRateLimitConfigValidator.java
copy fe/fe-core/src/main/java/org/apache/doris/{job/common/DataSourceType.java
=> cloud/rpc/MetaServiceRateLimitException.java} (69%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/cloud/rpc/MetaServiceRpcRateLimiter.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/cloud/rpc/MetaServiceRpcRateLimiterTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodePinnedSchemaTest.java
copy fe/fe-filesystem/{fe-filesystem-s3 => fe-filesystem-gcs}/pom.xml (63%)
copy fe/fe-filesystem/{fe-filesystem-jfs =>
fe-filesystem-gcs}/src/main/assembly/plugin-zip.xml (100%)
create mode 100644
fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProperties.java
create mode 100644
fe/fe-filesystem/fe-filesystem-gcs/src/main/java/org/apache/doris/filesystem/gcs/GcsFileSystemProvider.java
copy docker/thirdparties/docker-compose/oceanbase/oceanbase.yaml.tpl =>
fe/fe-filesystem/fe-filesystem-gcs/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
(55%)
create mode 100644
fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemPropertiesTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-gcs/src/test/java/org/apache/doris/filesystem/gcs/GcsFileSystemProviderTest.java
copy fe/fe-filesystem/{fe-filesystem-s3 => fe-filesystem-minio}/pom.xml (63%)
copy fe/fe-filesystem/{fe-filesystem-jfs =>
fe-filesystem-minio}/src/main/assembly/plugin-zip.xml (100%)
create mode 100644
fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProperties.java
create mode 100644
fe/fe-filesystem/fe-filesystem-minio/src/main/java/org/apache/doris/filesystem/minio/MinioFileSystemProvider.java
copy docker/thirdparties/docker-compose/oceanbase/oceanbase.yaml.tpl =>
fe/fe-filesystem/fe-filesystem-minio/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
(55%)
create mode 100644
fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemPropertiesTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-minio/src/test/java/org/apache/doris/filesystem/minio/MinioFileSystemProviderTest.java
copy fe/fe-filesystem/{fe-filesystem-s3 => fe-filesystem-ozone}/pom.xml (63%)
copy fe/fe-filesystem/{fe-filesystem-jfs =>
fe-filesystem-ozone}/src/main/assembly/plugin-zip.xml (100%)
create mode 100644
fe/fe-filesystem/fe-filesystem-ozone/src/main/java/org/apache/doris/filesystem/ozone/OzoneFileSystemProperties.java
create mode 100644
fe/fe-filesystem/fe-filesystem-ozone/src/main/java/org/apache/doris/filesystem/ozone/OzoneFileSystemProvider.java
copy docker/thirdparties/docker-compose/oceanbase/oceanbase.yaml.tpl =>
fe/fe-filesystem/fe-filesystem-ozone/src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider
(55%)
create mode 100644
fe/fe-filesystem/fe-filesystem-ozone/src/test/java/org/apache/doris/filesystem/ozone/OzoneFileSystemPropertiesTest.java
create mode 100644
fe/fe-filesystem/fe-filesystem-ozone/src/test/java/org/apache/doris/filesystem/ozone/OzoneFileSystemProviderTest.java
copy fe/fe-filesystem/{fe-filesystem-s3 => fe-filesystem-s3-base}/pom.xml (64%)
create mode 100644
fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/AbstractDelegatingS3Properties.java
create mode 100644
fe/fe-filesystem/fe-filesystem-s3-base/src/main/java/org/apache/doris/filesystem/s3/S3CompatSignals.java
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/main/java/org/apache/doris/filesystem/s3/S3CredentialsProviderFactory.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/main/java/org/apache/doris/filesystem/s3/S3CredentialsProviderType.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/main/java/org/apache/doris/filesystem/s3/S3FileSystem.java
(87%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/main/java/org/apache/doris/filesystem/s3/S3FileSystemProperties.java
(85%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/main/java/org/apache/doris/filesystem/s3/S3ObjStorage.java
(97%)
create mode 100644
fe/fe-filesystem/fe-filesystem-s3-base/src/test/java/org/apache/doris/filesystem/s3/S3CompatSignalsTest.java
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3CredentialsProviderFactoryTest.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemEnvTest.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemPropertiesTest.java
(72%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3FileSystemTest.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageEnvTest.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageMockTest.java
(100%)
rename fe/fe-filesystem/{fe-filesystem-s3 =>
fe-filesystem-s3-base}/src/test/java/org/apache/doris/filesystem/s3/S3ObjStorageTest.java
(100%)
copy fe/fe-core/src/main/java/org/apache/doris/job/common/DataSourceType.java
=>
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/source/reader/oceanbase/OceanBaseSourceReader.java
(62%)
create mode 100644
fs_brokers/cdc_client/src/test/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSourceTest.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/itcase/OceanBaseSchemaChangeITCase.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/itcase/OceanBaseStartupOffsetITCase.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/itcase/OceanBaseTestBase.java
create mode 100644
fs_brokers/cdc_client/src/test/java/org/apache/doris/cdcclient/itcase/OceanBaseWriteDmlITCase.java
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job.out
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_all_type.out
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_offset.out
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_restart_fe.out
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_sc_restart_fe.out
create mode 100644
regression-test/data/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_schema_change.out
create mode 100644
regression-test/data/nereids_function_p0/agg_function/agg_distinct_function.out
create mode 100644
regression-test/data/variant_p0/test_topn_lazy_materialize_sparse_variant.out
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_dual_relation_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_equality_delete_time_travel.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_metadata_atomicity_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_position_dv_time_travel.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_ref_actions_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_time_travel_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/iceberg_paimon_schema_time_travel_coverage.md
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_schema_branch_partition_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_schema_dual_relation_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_schema_metadata_atomicity_matrix.groovy
create mode 100644
regression-test/suites/external_table_p0/paimon/test_paimon_schema_time_travel_matrix.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_all_type.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_offset.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_restart_fe.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_sc_restart_fe.groovy
create mode 100644
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_oceanbase_job_schema_change.groovy
create mode 100644
regression-test/suites/nereids_function_p0/agg_function/agg_distinct_function.groovy
create mode 100644
regression-test/suites/variant_p0/test_topn_lazy_materialize_sparse_variant.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
