This is an automated email from the ASF dual-hosted git repository. lijibing pushed a change to branch high-priority-column in repository https://gitbox.apache.org/repos/asf/doris.git
omit 6038033cbe6 Support high priority column stats auto collection. * fix visible column (#33023) * Collect high priority columns. (#31235) * High priority queue and map. (#31509) * Support column level health value. (#31794) * Support follower sync query columns to master. (#31859) * Support show auto analyze pending jobs. (#31926) * Check column health value earlier, show job priority. (#32064) * support window (#32094) * Refactor. (#32273) * refactor2 (#32278) * Unit test (#32398) [...] add 4ebc91fa148 [chore](spill) add timers for performance tuning (#33185) add 8d9e12a4050 [FIX](agg) fix vertical_compaction_reader for agg table with array/map type (#33130) add 4c1599f1161 [Fix](trino-connector) Disable JUL from printing logs to the console (#33178) add 398c3116723 [Fix](nereids) forward the user define variables to master (#33013) add d58c7a812e0 [Fix](nereids) modify the binding aggregate function in order by (#32758) add 343c0c1a1ae [fix](nereids) runtime filter with probe expr should be pushed though set operator #33010 add e277382776d [fix](fe)reduce memory usage in alter (#32810) add 9ea9f44cb63 [fix](inverted index) fix the incorrect result issue of COUNT_ON_INDEX for key columns (#33164) add a1db6ec6f89 [fix](Nereids) convert agg state type failed in some cases (#33208) add 84f5fa63129 [fix](tvf) Support fs.defaultFS with postfix '/' (#33202) add 2674e222544 [fix](nereids)SimplifyRange didn't process NULL value correctly (#32854) add c20864ff4c7 [fix](plsql) Fix plsql variable initialization (#33186) add 6ebf320210b [bugfix](stop) should skip the loop when graceful stop (#33212) add f978079830a [fix](Export) Set the default value of the `data_consistence` property of export to `partition` (#32830) add 94c862e605e [refactor](coordinator) split profile logic and instance report logic (#32010) add c741ff8357c [Bug](runtime-filter) set need_local_merge to false when rf is broadcast (#33211) add 51508cfe5e6 [fix](memory) Fix memory log compile by ASAN (#33162) add b0f773d17af [fix](Nereids) support update without filter (#33214) add c34b52c02a0 [fix](move-memtable) fix initial use count of streams for auto partition (#33165) add 98b04fc3ed0 [FIX](inverted_index) fix inverted index write array with _doc is empty (#33170) add 63105615f9f [Fix](executor)Fix normal group can not be appended when image exits #33197 add d178a65f5d8 [test](cast) add test for stream load cast (#33189) add 939766df919 [fix](plsql) Fix regression test execute the same name procedure in parallel #33234 add 031a4c039a6 [enhance](cloud_p0) morecases for cloud_p0 (#32652) add 23f9ee042fe [feature](hive)support ExternalTransaction for writing exteral table (#32726) add 2ad3f940ff4 [feature](merge-cloud) Add table stat metrics in FE (#32876) add 50f10d16df6 (cloud-merge) Fix the is not same in invertedIndex and catalog (#33183) add c50176fdeeb [fix](cloud) fix failed to get delete bitmap update lock key info (#33238) add 394e40d4bb2 [fix](compile) fix gcc compile failed (#30596) (#33150) add 39b53360305 [feature](index compaction)support index compaction in cloud mode (#33168) add 71377498e63 [fix](case) ignore single_replica_compaction in cloud mode (#33072) add b3d57c727c2 [fix](cloud-cases) exclude cache ttl (#33253) add 3ceff6a43ad [fix](cloud) Fix coredump during graceful shutdown (#33250) add 44090a523bf [debug](log) Add some debug logs (#33237) add 6d4e750bae6 [fix](merge-cloud) fix backend id in tablet invert index (#33194) add 82f86a8730b [enhancement](merge-on-write) consider version count on size-based cloud cu compaction policy (#33244) add 5a3be311a20 (cloud-merge) Fix some problem about the file cache strategy `ttl` (#33128) add 790cde1190a [Feature](Cloud) Add show storage vault stmt (#33249) add 4886ad8333d [fix](cloud) fix routine load can not consume in cloud mode (#33248) add 8c850d8250a [enhance](cloud) Validate and normalize hdfs conf when adding new vault (#33255) add f71dd71bda1 [fix](inverted index) fix incorrect case test_index_delete (#33246) add 25d7242e559 [feature](merge-cloud) Add `SecurityChecker` for cloud (#32906) add 01490139ef6 [improve](txn insert) txn insert support delete command (#33100) add 605aa8d9240 [fix](move-memtable) fix timeout to get tablet schema (#33256) add 3e2d9014920 [improve](function) the offset params in lead/lag function could use 0 (#33174) add 4d6d2263cd4 [Enhancement](partial update)Add timezone case for partial update timestamp #33177 add f6782d6c6e7 [Fix](point query) avoid nullptr in _block_pool (#33120) add c0b937b0afb [fix](nereids)reject nse-rf for nested loop outer join #33122 add b417ef3be3b [BugFix](Iceberg Catalog) Fix iceberg catalog of hms and hadoop not support iceberg properties (#33113) add 900ca7bd6e8 [fix](cloud) Always use absolute path in hdfs (#33259) add 45e36c8bb03 [fix](regression) fix mtmv statistics case failure (#33266) add 95d42aefb1f [fix](merge-cloud) add injection test http api and fix cloud compaction http api (#33268) add 31dfc40ecb2 [Improvement](executor)Add BypassWorkloadGroup to pass query queue #33101 add 026ef3c4bdd [fix](merge-cloud) update modify backend's cluster name (#33263) add 8429353e79e [fix](routine-load) fix routine load lock do not release (#33275) add e7a468b7b31 [chore](ci) set enableKafkaTest=true for test_routine_load (#33271) add 99f1127b554 [fix](merge-cloud) change cloud rebalance config to masterOnly=true (#33261) add 39b1bd40f07 [test](Cloud) Add more storage vault ut (#33269) add 8ea87892bb3 [fix](cloud_p0) exclude cache ttl in cloud_p0 (#33258) add d7203657e00 [fix(cloud-merge) Modify the cluster name in ttl test (#33279) add a8596ecf3e9 [fix](UT) Fix incorrect command for cloud run_all_tests.sh (#33274) add 68e1413417b [feature&fix](cloud) Support hdfs accessor in Checker (#33280) add d5b1eb1d0a9 [chore](cloud) Fix clear file cache implementation and regression case (#33281) add 8cc8933d30e (httpaction) log response of http (#33270) add 9c9ee9c5a1a [fix](cloud) add load.groovy for cache cases (#33287) add 7977976ef07 [Bug](materialized-view) fix wrong result when salias name same with base slot on mv (#33198) add e4e763aeccc [improvement](serde) Handle NaN values in number for MySQL result write (#33227) add 3f98b1f4ce7 [pipelineX](fix) Fix coredump by incorrect cancel order (#33294) add 4feb98030fc [fix](spill) fix hash join error 'invalid slot id' (#33273) add d0777780685 [chore](test) nereids support window function but some cases does not open yet (#33098) add 6df632d0383 [improve](exec) reduce copy on store_string_ref (#33232) add 7cbdaa7e89a [chore](license) fix incomplete license header (#33306) add 79a7718f9f3 [fix](iceberg) fix iceberg predicate conversion bug (#33283) add 0de7d9f4093 [feature](replica) support force set replicate allocation for olap tables (#32916) add 77e28b5ab5a [Feature][Enhancement](hive-writer) Add hive-writer runtime profiles, change output file names (#33245) add c4247cd2177 [chore](Nereids) load command always could fallback (#33233) add a65901578bb [bug](cloud) fix null pointer exception in get partition visible version rpc (#33070) add dd110460621 [fix](Nereids) array_range not support amount without unit (#33231) add aa3a3fdac37 [chore](Nereids) create policy always allow fallback (#33226) add 7c73df569f9 (cloud-merge) Fix the case timeout because the sql fetches too many datas (#33300) add 119dc4d660d [fix](variables) change column type of @@autocommit to BIGINT (#33282) add 1d4b99cfbd4 [fix](cloud) change to masterOnly=false (#33316) add bb938f93abd [Bug](case) fix wrong case test_mv_partition (#33324) add 26ad7417284 [improvement](doris-compose) Add more config for custom file (#32949) add 90d5faf2e80 Fix compatibility issues with GLIBC(>= 2.34) for prebuilt thirdparty packages (#33314) add 15527ec8426 [fix](Nereids) variant column prune push down failed on variant literal (#33328) add aff6d93a15d [fix](Nereids) partial update generate column in wrong way (#33326) add 8eea3cb92d7 [fix](hdfs-writer) Catch error information after `hdfsCloseFile()` (#33195) add e680afbbe24 [fix](parquet) fix time zone error in parquet reader (#33217) add f0ce519ecab [chore](multi-table-load) add context info in log when using single-stream-multi-table load (#33317) add 18f997a18e0 [chore](cloud) Fix code style and run cloud UT script (#33284) add c2fd8931afd [config](cloud) Set meta_schema_value_version to 1 by default (#33288) add e45c86c2f17 [fix](txn insert) Fix txn insert into values for sequence column or column name is keyword (#33336) add f9de308dfff [Bug](array) fix array column core dump in get_shrinked_column as not check type (#33295) add fb5454c213c [improvement](spill) avoid unnecessary spilling in hash join build phase (#33277) add 82bce1b566e [fix](compile) fix implicit float-to-int conversion in mem_info calculation (#33311) add dd17f310b5c [fix](group commit) Fix syntax error when insert into table which column names contain keyword (#33322) add 6dbf3494f88 [fix](merge-cloud) Fix drop rollup/mv leak meta kvs and data objects resource (#33321) add 1eb0a7511c6 [fix](npe) fix kafka be id npe (#33151) add 6f4b426d9e3 [fix](jdbc catalog) fix db2 test connection sql (#33335) add 74c5549e540 [fix](nereids) make runtime filter targets in fixed order (#33191) add c485c350abe [Improvement](materialized-view) adjust priority of materialized view match rule (#33305) add 9d2cfc072ab [fix](test) Fix auto partition dbgen product data (#33313) add b7e44d84750 [Fix](grace-exit) Fix threadpool hang on grace exit (#33132) add ee48fa2305c [fix](Nereids): don't transpose agg and join if join is mark join (#33312) add 0d3a2a93548 [fix](pipeline)group by and output is empty (#33192) add 16313a76aae [improvement](spill) improve cancel of spill and improve log printing (#33229) add bd684c316eb [scan](improvement) Adjust parallel scanners num (#33223) add a4a42bf35b2 [improve](cloud) Optimize meta service proxy retry strategy (#32902) add 86ba42d56ee Set enable_unique_key_partial_update to false in statistics session variable. (#33220) new b97cee9f067 Support high priority column stats auto collection. * fix visible column (#33023) * Collect high priority columns. (#31235) * High priority queue and map. (#31509) * Support column level health value. (#31794) * Support follower sync query columns to master. (#31859) * Support show auto analyze pending jobs. (#31926) * Check column health value earlier, show job priority. (#32064) * support window (#32094) * Refactor. (#32273) * refactor2 (#32278) * Unit test (#32398) [...] This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6038033cbe6) \ N -- N -- N refs/heads/high-priority-column (b97cee9f067) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/CMakeLists.txt | 4 + be/src/cloud/cloud_base_compaction.cpp | 3 +- be/src/cloud/cloud_compaction_action.cpp | 171 +-- be/src/cloud/cloud_compaction_action.h | 3 - be/src/cloud/cloud_cumulative_compaction.cpp | 2 +- .../cloud/cloud_cumulative_compaction_policy.cpp | 16 +- be/src/cloud/cloud_cumulative_compaction_policy.h | 5 +- be/src/cloud/cloud_storage_engine.cpp | 8 +- be/src/cloud/cloud_storage_engine.h | 15 + be/src/cloud/cloud_tablet.cpp | 20 +- be/src/cloud/cloud_tablet.h | 2 +- be/src/cloud/cloud_tablet_mgr.cpp | 12 +- be/src/cloud/cloud_tablet_mgr.h | 5 +- be/src/cloud/injection_point_action.cpp | 213 ++++ .../src/cloud/injection_point_action.h | 23 +- be/src/clucene | 2 +- be/src/common/config.cpp | 5 + be/src/common/config.h | 5 +- be/src/exprs/minmax_predicate.h | 12 +- be/src/exprs/runtime_filter.cpp | 1 + be/src/exprs/runtime_filter.h | 2 +- be/src/http/action/clear_file_cache_action.cpp | 12 +- be/src/io/cache/block_file_cache.cpp | 86 +- be/src/io/cache/block_file_cache.h | 10 +- be/src/io/cache/block_file_cache_factory.cpp | 13 +- be/src/io/cache/block_file_cache_factory.h | 10 +- be/src/io/cache/block_file_cache_profile.cpp | 54 +- be/src/io/cache/block_file_cache_profile.h | 18 +- be/src/io/cache/cached_remote_file_reader.cpp | 6 + be/src/io/cache/fs_file_cache_storage.cpp | 10 +- be/src/io/cache/fs_file_cache_storage.h | 2 +- be/src/io/fs/file_writer.h | 4 +- be/src/io/fs/hdfs_file_system.cpp | 7 +- be/src/io/fs/hdfs_file_writer.cpp | 8 +- be/src/io/fs/multi_table_pipe.cpp | 26 +- be/src/io/hdfs_util.cpp | 18 +- be/src/olap/base_tablet.cpp | 12 + be/src/olap/base_tablet.h | 2 + be/src/olap/compaction.cpp | 43 +- be/src/olap/compaction.h | 12 +- be/src/olap/olap_server.cpp | 3 +- be/src/olap/rowset/beta_rowset_reader.cpp | 7 +- be/src/olap/rowset/beta_rowset_writer.cpp | 4 +- be/src/olap/rowset/beta_rowset_writer.h | 8 + .../rowset/segment_v2/inverted_index_writer.cpp | 9 +- be/src/olap/rowset/segment_v2/segment_iterator.cpp | 20 +- be/src/olap/rowset/segment_v2/segment_iterator.h | 5 +- be/src/olap/storage_engine.cpp | 4 + be/src/olap/tablet.cpp | 34 +- be/src/olap/tablet.h | 25 +- be/src/pipeline/exec/file_scan_operator.h | 2 + be/src/pipeline/exec/hashjoin_build_sink.h | 1 + .../exec/nested_loop_join_build_operator.cpp | 2 +- .../exec/partitioned_aggregation_sink_operator.cpp | 46 +- .../exec/partitioned_aggregation_sink_operator.h | 9 +- .../partitioned_aggregation_source_operator.cpp | 43 +- .../exec/partitioned_hash_join_probe_operator.cpp | 209 ++-- .../exec/partitioned_hash_join_probe_operator.h | 36 +- .../exec/partitioned_hash_join_sink_operator.cpp | 272 ++-- .../exec/partitioned_hash_join_sink_operator.h | 29 +- be/src/pipeline/exec/scan_operator.cpp | 7 +- be/src/pipeline/exec/scan_operator.h | 2 + be/src/pipeline/exec/spill_sort_sink_operator.cpp | 150 ++- .../pipeline/exec/spill_sort_source_operator.cpp | 63 +- be/src/pipeline/pipeline_fragment_context.cpp | 4 +- be/src/pipeline/pipeline_x/dependency.cpp | 18 +- be/src/pipeline/pipeline_x/dependency.h | 6 +- .../local_exchange_sink_operator.cpp | 6 +- be/src/pipeline/pipeline_x/operator.h | 6 + .../pipeline_x/pipeline_x_fragment_context.cpp | 30 +- be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 13 +- be/src/runtime/exec_env.cpp | 6 +- be/src/runtime/exec_env.h | 16 +- be/src/runtime/exec_env_init.cpp | 14 +- be/src/runtime/memory/mem_tracker_limiter.cpp | 4 + be/src/runtime/query_context.cpp | 4 +- .../routine_load/routine_load_task_executor.cpp | 2 +- be/src/service/http_service.cpp | 9 + be/src/service/internal_service.cpp | 12 +- be/src/service/point_query_executor.cpp | 6 +- be/src/util/doris_metrics.cpp | 7 + be/src/util/doris_metrics.h | 4 + be/src/util/mem_info.cpp | 11 +- be/src/util/perf_counters.cpp | 4 + be/src/vec/columns/column.h | 9 +- be/src/vec/columns/column_array.cpp | 10 +- be/src/vec/columns/column_array.h | 2 + be/src/vec/columns/column_const.h | 2 + be/src/vec/columns/column_map.cpp | 10 +- be/src/vec/columns/column_map.h | 2 + be/src/vec/columns/column_nullable.cpp | 12 +- be/src/vec/columns/column_nullable.h | 2 + be/src/vec/columns/column_string.h | 9 +- be/src/vec/columns/column_struct.cpp | 13 +- be/src/vec/columns/column_struct.h | 1 + .../data_types/serde/data_type_number_serde.cpp | 14 +- .../exec/format/parquet/parquet_column_convert.h | 6 +- be/src/vec/exec/format/parquet/parquet_pred_cmp.h | 2 +- be/src/vec/exec/vaggregation_node.cpp | 4 +- be/src/vec/exec/vaggregation_node.h | 2 + be/src/vec/olap/block_reader.cpp | 17 +- be/src/vec/olap/vertical_block_reader.cpp | 15 +- be/src/vec/olap/vertical_block_reader.h | 2 +- be/src/vec/sink/load_stream_map_pool.cpp | 126 ++ ...d_stream_stub_pool.h => load_stream_map_pool.h} | 63 +- be/src/vec/sink/load_stream_stub.cpp | 35 +- be/src/vec/sink/load_stream_stub.h | 14 +- be/src/vec/sink/load_stream_stub_pool.cpp | 72 -- be/src/vec/sink/volap_table_sink_v2.cpp | 2 - be/src/vec/sink/writer/vhive_partition_writer.cpp | 54 +- be/src/vec/sink/writer/vhive_partition_writer.h | 11 +- be/src/vec/sink/writer/vhive_table_writer.cpp | 245 ++-- be/src/vec/sink/writer/vhive_table_writer.h | 27 +- be/src/vec/sink/writer/vtablet_writer_v2.cpp | 183 ++- be/src/vec/sink/writer/vtablet_writer_v2.h | 10 +- be/src/vec/spill/spill_stream.cpp | 14 +- be/src/vec/spill/spill_stream.h | 4 +- be/src/vec/spill/spill_stream_manager.cpp | 2 - be/src/vec/spill/spill_writer.cpp | 18 +- be/src/vec/spill/spill_writer.h | 5 +- be/test/io/fs/stream_sink_file_writer_test.cpp | 4 +- be/test/olap/tablet_cooldown_test.cpp | 1 + be/test/vec/exec/load_stream_stub_map_test.cpp | 59 + be/test/vec/exec/load_stream_stub_pool_test.cpp | 51 - build.sh | 6 + cloud/script/run_all_tests.sh | 18 +- cloud/src/common/config.h | 8 +- cloud/src/common/stopwatch.h | 9 +- cloud/src/meta-service/codec.cpp | 2 + cloud/src/meta-service/keys.cpp | 20 +- cloud/src/meta-service/meta_server.cpp | 3 +- cloud/src/meta-service/meta_service.cpp | 42 +- cloud/src/meta-service/meta_service.h | 48 +- cloud/src/meta-service/meta_service_helper.h | 1 + cloud/src/meta-service/meta_service_http.cpp | 3 +- cloud/src/meta-service/meta_service_partition.cpp | 9 +- cloud/src/meta-service/meta_service_resource.cpp | 86 +- cloud/src/meta-service/meta_service_txn.cpp | 7 + cloud/src/recycler/checker.cpp | 97 +- cloud/src/recycler/checker.h | 17 +- cloud/src/recycler/hdfs_accessor.cpp | 26 +- cloud/src/recycler/hdfs_accessor.h | 3 +- cloud/src/recycler/meta_checker.cpp | 6 +- cloud/src/recycler/obj_store_accessor.h | 14 +- cloud/src/recycler/recycler.cpp | 8 +- cloud/src/recycler/s3_accessor.cpp | 3 +- cloud/src/recycler/util.cpp | 22 +- cloud/test/fdb_injection_test.cpp | 3 +- cloud/test/hdfs_accessor_test.cpp | 4 +- cloud/test/meta_service_test.cpp | 323 ++++- cloud/test/recycler_test.cpp | 5 +- docker/runtime/doris-compose/Readme.md | 6 +- docker/runtime/doris-compose/cluster.py | 7 +- docker/runtime/doris-compose/command.py | 76 +- .../org/apache/doris/jdbc/BaseJdbcExecutor.java | 3 +- .../org/apache/doris/jdbc/DefaultJdbcExecutor.java | 3 +- .../trinoconnector/TrinoConnectorPluginLoader.java | 6 +- .../cloud/security/DummySecurityChecker.java} | 28 +- .../doris/cloud/security/SecurityChecker.java | 56 + .../doris/cloud/security/UrlSecurityChecker.java | 76 ++ .../main/java/org/apache/doris/common/Config.java | 36 +- .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 2 +- .../antlr4/org/apache/doris/nereids/PLLexer.g4 | 4 - .../antlr4/org/apache/doris/nereids/PLParser.g4 | 2 +- fe/fe-core/src/main/cup/sql_parser.cup | 4 + .../org/apache/doris/alter/CloudRollupJobV2.java | 4 +- .../apache/doris/alter/CloudSchemaChangeJobV2.java | 7 +- .../doris/alter/MaterializedViewHandler.java | 11 + .../org/apache/doris/alter/SchemaChangeJobV2.java | 51 +- .../java/org/apache/doris/analysis/ExportStmt.java | 18 +- .../org/apache/doris/analysis/LiteralExpr.java | 20 + .../apache/doris/analysis/NativeInsertStmt.java | 5 + .../doris/analysis/ShowStorageVaultStmt.java | 60 + .../apache/doris/catalog/CloudTabletStatMgr.java | 5 +- .../main/java/org/apache/doris/catalog/Env.java | 4 + .../java/org/apache/doris/catalog/OlapTable.java | 8 + .../org/apache/doris/catalog/StorageVault.java | 37 + .../apache/doris/catalog/TabletInvertedIndex.java | 8 + .../cloud/datasource/CloudInternalCatalog.java | 31 + .../org/apache/doris/cloud/qe/SnapshotProxy.java | 19 +- .../apache/doris/cloud/rpc/MetaServiceClient.java | 11 + .../apache/doris/cloud/rpc/MetaServiceProxy.java | 10 + .../doris/cloud/system/CloudSystemInfoService.java | 29 + .../main/java/org/apache/doris/common/Status.java | 5 + .../doris/common/profile/ExecutionProfile.java | 309 ++--- .../org/apache/doris/common/profile/Profile.java | 80 +- .../doris/common/profile/SummaryProfile.java | 16 +- .../org/apache/doris/common/util/HttpURLUtil.java | 26 +- .../apache/doris/common/util/ProfileManager.java | 82 +- .../apache/doris/common/util/PropertyAnalyzer.java | 22 + .../apache/doris/common/util/RuntimeProfile.java | 34 +- .../org/apache/doris/common/util/SmallFileMgr.java | 7 +- .../java/org/apache/doris/common/util/Util.java | 31 +- .../apache/doris/datasource/ExternalCatalog.java | 2 + .../apache/doris/datasource/InternalCatalog.java | 4 + .../apache/doris/datasource/es/EsRestClient.java | 17 +- .../apache/doris/datasource/hive/HMSCommitter.java | 754 ----------- .../doris/datasource/hive/HMSExternalCatalog.java | 6 +- .../doris/datasource/hive/HMSTransaction.java | 1322 ++++++++++++++++++++ .../doris/datasource/hive/HiveMetadataOps.java | 24 +- .../datasource/hive/HivePartitionStatistics.java | 2 +- .../hive/HivePartitionWithStatistics.java | 6 +- .../iceberg/IcebergHMSExternalCatalog.java | 3 +- .../iceberg/IcebergHadoopExternalCatalog.java | 5 +- .../doris/datasource/iceberg/IcebergUtils.java | 73 +- .../datasource/iceberg/source/IcebergScanNode.java | 18 +- .../doris/datasource/jdbc/client/JdbcClient.java | 5 +- .../datasource/jdbc/client/JdbcDB2Client.java | 4 + .../apache/doris/datasource/kafka/KafkaUtil.java | 8 +- .../org/apache/doris/httpv2/rest/LoadAction.java | 46 +- .../org/apache/doris/httpv2/rest/UploadAction.java | 4 + .../main/java/org/apache/doris/load/DeleteJob.java | 19 +- .../main/java/org/apache/doris/load/ExportJob.java | 2 +- .../java/org/apache/doris/load/TxnDeleteJob.java | 67 + .../apache/doris/load/loadv2/BrokerLoadJob.java | 21 +- .../apache/doris/load/loadv2/LoadLoadingTask.java | 16 +- .../load/routineload/KafkaRoutineLoadJob.java | 3 +- .../doris/load/routineload/RoutineLoadJob.java | 24 + .../doris/metric/PrometheusMetricVisitor.java | 93 ++ .../doris/nereids/parser/LogicalPlanBuilder.java | 4 +- .../processor/post/RuntimeFilterGenerator.java | 18 + .../nereids/rules/analysis/BindExpression.java | 57 +- .../nereids/rules/analysis/FillUpMissingSlots.java | 2 +- .../rules/exploration/TransposeAggSemiJoin.java | 4 +- .../exploration/TransposeAggSemiJoinProject.java | 4 +- .../expression/rules/ConvertAggStateCast.java | 38 +- .../rules/expression/rules/SimplifyRange.java | 10 + .../mv/AbstractSelectMaterializedIndexRule.java | 33 +- .../mv/SelectMaterializedIndexWithAggregate.java | 118 +- .../scalar/PushDownToProjectionFunction.java | 4 +- .../trees/expressions/functions/window/Lag.java | 2 +- .../trees/expressions/functions/window/Lead.java | 2 +- .../trees/plans/commands/CreatePolicyCommand.java | 1 + .../plans/commands/DeleteFromUsingCommand.java | 1 + .../trees/plans/commands/ExportCommand.java | 16 +- .../nereids/trees/plans/commands/LoadCommand.java | 5 +- .../trees/plans/commands/UpdateCommand.java | 3 +- .../plans/commands/insert/HiveInsertExecutor.java | 34 +- .../plans/physical/PhysicalNestedLoopJoin.java | 20 + .../trees/plans/physical/RuntimeFilter.java | 3 +- .../apache/doris/planner/GroupCommitPlanner.java | 2 +- .../src/main/java/org/apache/doris/plsql/Exec.java | 6 - .../doris/plsql/executor/DorisRowResult.java | 2 +- .../java/org/apache/doris/qe/ConnectContext.java | 8 + .../java/org/apache/doris/qe/ConnectProcessor.java | 25 +- .../java/org/apache/doris/qe/CoordInterface.java | 2 - .../main/java/org/apache/doris/qe/Coordinator.java | 635 +++++----- .../java/org/apache/doris/qe/MasterOpExecutor.java | 17 +- .../java/org/apache/doris/qe/PointQueryExec.java | 6 - .../main/java/org/apache/doris/qe/QeProcessor.java | 2 - .../java/org/apache/doris/qe/QeProcessorImpl.java | 79 +- .../java/org/apache/doris/qe/SessionVariable.java | 34 +- .../java/org/apache/doris/qe/ShowExecutor.java | 26 + .../java/org/apache/doris/qe/StmtExecutor.java | 83 +- .../main/java/org/apache/doris/qe/VariableMgr.java | 97 +- .../resource/workloadgroup/WorkloadGroupMgr.java | 23 +- .../org/apache/doris/rpc/BackendServiceClient.java | 3 +- .../org/apache/doris/rpc/BackendServiceProxy.java | 7 +- .../doris/statistics/util/StatisticsUtil.java | 2 +- .../org/apache/doris/system/SystemInfoService.java | 4 - .../doris/transaction/HiveTransactionManager.java | 79 ++ .../org/apache/doris/transaction/Transaction.java | 19 +- .../doris/transaction/TransactionManager.java | 26 +- .../transaction/TransactionManagerFactory.java | 19 +- .../analysis/ExportToOutfileLogicalPlanTest.java | 20 +- .../doris/common/util/RuntimeProfileTest.java | 2 +- .../doris/datasource/hive/HmsCommitTest.java | 72 +- .../datasource/iceberg/IcebergPredicateTest.java | 256 ++++ .../datasource/iceberg/TestIcebergPredict.java | 135 -- .../rules/analysis/FillUpMissingSlotsTest.java | 2 +- .../exploration/TransposeAggSemiJoinTest.java | 18 + .../rules/expression/SimplifyRangeTest.java | 25 +- .../rewrite/TransposeSemiJoinAggProjectTest.java | 15 + .../java/org/apache/doris/qe/StmtExecutorTest.java | 4 +- .../java/org/apache/doris/qe/VariableMgrTest.java | 19 + gensrc/proto/cloud.proto | 5 +- gensrc/thrift/FrontendService.thrift | 2 +- regression-test/conf/regression-conf.groovy | 1 + .../data/cloud_p0/cache/ttl/test_show_ttl.out | 16 + .../test_compaction_agg_keys_with_array_map.out | 13 + .../test_always_nullable_window_function.out | 12 + .../data_model_p0/unique/test_unique_table.out | 2 +- .../unique/test_unique_table_new_sequence.out | 4 +- .../double/test_double_nan.out} | 5 +- .../float/test_float_nan.out} | 5 +- .../hive/test_hive_basic_type.out | 24 +- .../tvf/test_s3_tvf_parquet_compress.out | 120 +- .../hive/test_external_catalog_hive.out | 9 + .../iceberg/test_iceberg_predicate_conversion.out | 611 +++++++++ .../data/insert_p0/insert_group_commit_into.out | 8 + regression-test/data/insert_p0/txn_insert.out | 19 + .../test_index_delete.out} | 16 +- .../data/load_p0/stream_load/test_cast1.csv | 1 + .../data/load_p0/stream_load/test_cast1.json | 1 + .../data/load_p0/stream_load/test_cast2.csv | 1 + .../data/load_p0/stream_load/test_cast2.json | 1 + .../data/load_p0/stream_load/test_cast3.csv | 1 + .../data/load_p0/stream_load/test_cast3.json | 1 + .../load_p0/stream_load/test_stream_load_cast.out | 19 + .../test_base/test_base.out} | 9 +- .../test_upper_alias/test_upper_alias.out} | 13 +- .../load.out | 0 .../data/nereids_function_p0/scalar_function/A.out | 16 +- .../data/nereids_hint_tpcds_p0/shape/query24.out | 2 +- .../data/nereids_hint_tpcds_p0/shape/query64.out | 4 +- .../data/nereids_p0/aggregate/aggregate.out | 322 ++++- .../window_functions/test_window_fn.out | 46 +- .../window_functions/test_window_function.out | 132 +- .../nereids_syntax_p0/order_by_bind_priority.out | 25 + .../shape/query14.out | 2 +- .../shape/query17.out | 2 +- .../nereids_tpcds_shape_sf1000_p0/shape/query2.out | 2 +- .../shape/query24.out | 2 +- .../shape/query25.out | 2 +- .../shape/query29.out | 2 +- .../nereids_tpcds_shape_sf1000_p0/shape/query4.out | 2 +- .../nereids_tpcds_shape_sf1000_p0/shape/query5.out | 10 +- .../shape/query64.out | 2 +- .../shape/query71.out | 4 +- .../noStatsRfPrune/query5.out | 6 +- .../noStatsRfPrune/query64.out | 4 +- .../noStatsRfPrune/query65.out | 2 +- .../noStatsRfPrune/query71.out | 4 +- .../no_stats_shape/query14.out | 2 +- .../no_stats_shape/query17.out | 2 +- .../no_stats_shape/query2.out | 2 +- .../no_stats_shape/query24.out | 2 +- .../no_stats_shape/query25.out | 2 +- .../no_stats_shape/query29.out | 2 +- .../no_stats_shape/query4.out | 2 +- .../no_stats_shape/query5.out | 10 +- .../no_stats_shape/query64.out | 4 +- .../no_stats_shape/query65.out | 2 +- .../no_stats_shape/query71.out | 4 +- .../rf_prune/query5.out | 6 +- .../rf_prune/query64.out | 2 +- .../rf_prune/query71.out | 4 +- .../nereids_tpcds_shape_sf100_p0/shape/query17.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query2.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query24.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query25.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query29.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query4.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query5.out | 10 +- .../nereids_tpcds_shape_sf100_p0/shape/query64.out | 2 +- .../nereids_tpcds_shape_sf100_p0/shape/query71.out | 4 +- .../nostats_rf_prune/q20-rewrite.out | 2 +- .../nostats_rf_prune/q20.out | 2 +- .../nostats_rf_prune/q5.out | 2 +- .../rf_prune/q20-rewrite.out | 2 +- .../nereids_tpch_shape_sf1000_p0/rf_prune/q20.out | 2 +- .../runtime_filter/test_pushdown_setop.out | 25 + .../shape/q20-rewrite.out | 2 +- .../nereids_tpch_shape_sf1000_p0/shape/q20.out | 2 +- .../data/nereids_tpch_shape_sf1000_p0/shape/q5.out | 2 +- .../shape_no_stats/q20-rewrite.out | 2 +- .../shape_no_stats/q20.out | 2 +- .../shape_no_stats/q5.out | 2 +- .../test_plsql_variable.out} | 11 +- .../data/query_p0/test_array_orderby_limit.out | 3 + .../partial_update/test_partial_update.out | 6 + .../doris/regression/action/HttpCliAction.groovy | 2 + .../cloud_p0/conf/regression-conf-custom.groovy | 12 +- .../pipeline/p0/conf/regression-conf.groovy | 1 + .../cache/compaction/test_stale_rowset.groovy | 233 ++++ .../suites/cloud_p0/cache/ddl/customer.sql | 31 + .../suites/cloud_p0/cache/ddl/customer_delete.sql | 18 + .../suites/cloud_p0/cache/ddl/customer_load.sql | 23 + .../suites/cloud_p0/cache/ddl/customer_ttl.sql | 29 + .../cloud_p0/cache/ddl/customer_ttl_delete.sql | 18 + .../cloud_p0/cache/ddl/customer_ttl_load.sql | 23 + .../suites/cloud_p0/cache/ddl/nation.sql | 25 + .../suites/cloud_p0/cache/ddl/nation_delete.sql | 18 + .../suites/cloud_p0/cache/ddl/nation_load.sql | 23 + .../cache/http/test_cache_metrics_format.groovy | 108 ++ .../cloud_p0/cache/http/test_clear_cache.groovy | 161 +++ .../cache/http/test_clear_cache_async.groovy | 131 ++ regression-test/suites/cloud_p0/cache/load.groovy | 79 ++ .../suites/cloud_p0/cache/ttl/alter_ttl_1.groovy | 164 +++ .../suites/cloud_p0/cache/ttl/alter_ttl_2.groovy | 164 +++ .../suites/cloud_p0/cache/ttl/alter_ttl_3.groovy | 124 ++ .../suites/cloud_p0/cache/ttl/alter_ttl_4.groovy | 199 +++ .../cloud_p0/cache/ttl/alter_ttl_error.groovy | 86 ++ .../cloud_p0/cache/ttl/alter_ttl_max_int64.groovy | 119 ++ .../cloud_p0/cache/ttl/alter_ttl_random.groovy | 107 ++ .../cloud_p0/cache/ttl/alter_ttl_seconds.groovy | 119 ++ .../cache/ttl/create_table_as_select.groovy | 181 +++ .../cloud_p0/cache/ttl/create_table_like.groovy | 155 +++ .../suites/cloud_p0/cache/ttl/test_show_ttl.sql | 40 + .../suites/cloud_p0/cache/ttl/test_ttl.groovy | 151 +++ .../cloud_p0/cache/ttl/test_ttl_preempt.groovy | 183 +++ .../test_compaction_agg_keys_with_array_map.groovy | 154 +++ .../test_always_nullable_window_function.groovy | 8 +- .../data_model_p0/unique/test_unique_table.groovy | 8 +- .../unique/test_unique_table_new_sequence.groovy | 14 +- .../unique/test_unique_table_sequence.groovy | 16 +- .../datatype_p0/double/test_double_nan.groovy | 25 +- .../suites/datatype_p0/float/test_float_nan.groovy | 25 +- .../suites/export_p0/test_export_basic.groovy | 33 +- .../export_p0/test_export_data_consistency.groovy | 3 +- .../test_hive_read_parquet_comlex_type.groovy | 4 +- .../jdbc/test_db2_jdbc_catalog.groovy | 2 +- .../hive/test_external_catalog_hive.groovy | 5 +- .../test_iceberg_predicate_conversion.groovy | 79 ++ .../test_index_compaction_fault_injection.groovy | 51 +- .../insert_p0/insert_group_commit_into.groovy | 36 + regression-test/suites/insert_p0/txn_insert.groovy | 100 ++ .../test_index_compaction_dup_keys.groovy | 20 +- .../test_index_compaction_null.groovy | 19 +- .../test_index_compaction_unique_keys.groovy | 19 +- ...dex_compaction_with_multi_index_segments.groovy | 44 +- .../inverted_index_p0/test_index_delete.groovy | 69 + .../stream_load/test_stream_load_cast.groovy | 336 +++++ .../test_stream_load_move_memtable.groovy | 2 +- .../suites/mv_p0/test_base/test_base.groovy | 57 + .../mv_p0/test_dup_mv_year/test_dup_mv_year.groovy | 1 - .../test_mv_partition/test_mv_partition.groovy | 27 +- .../mv_p0/test_upper_alias/test_upper_alias.groovy | 69 + .../load.groovy | 0 .../suites/nereids_p0/aggregate/aggregate.groovy | 99 +- .../suites/nereids_p0/join/test_join_15.groovy | 10 +- .../suites/nereids_p0/keyword/order_group.groovy | 34 +- .../test_select_stddev_variance_window.groovy | 258 ++-- .../sql_functions/window_functions/test_sum.groovy | 11 +- .../window_functions/test_window_fn.groovy | 482 +++---- .../window_functions/test_window_function.groovy | 840 ++++++------- .../suites/nereids_p0/test_prune_tablet_mv.groovy | 2 +- .../order_by_bind_priority.groovy | 38 + .../runtime_filter/test_pushdown_setop.groovy | 7 +- .../auto_partition/sql/multi_thread_load.groovy | 12 + .../sql/stress_test_insert_into.groovy | 1 + .../diff_data/stress_test_diff_date_list.groovy | 1 - .../suites/plsql_p0/test_plsql_loop_cursor.groovy | 7 + .../suites/plsql_p0/test_plsql_variable.groovy | 115 ++ .../query_p0/test_array_orderby_limit.groovy | 16 + .../schema_change/test_alter_table_property.groovy | 32 +- .../schema_change_p0/test_rename_column.groovy | 4 +- .../test_statistic_partial_update.groovy | 56 + .../partial_update/test_partial_update.groovy | 8 + .../workload_manager_p0/test_curd_wlg.groovy | 13 + run-cloud-ut.sh | 8 +- thirdparty/download-thirdparty.sh | 11 + thirdparty/patches/krb5-1.19.patch | 13 + tools/cost_model_evaluate/requirements.txt | 2 + 444 files changed, 13054 insertions(+), 4573 deletions(-) create mode 100644 be/src/cloud/injection_point_action.cpp copy regression-test/suites/nereids_p0/sql_functions/window_functions/test_sum.groovy => be/src/cloud/injection_point_action.h (64%) create mode 100644 be/src/vec/sink/load_stream_map_pool.cpp rename be/src/vec/sink/{load_stream_stub_pool.h => load_stream_map_pool.h} (59%) delete mode 100644 be/src/vec/sink/load_stream_stub_pool.cpp create mode 100644 be/test/vec/exec/load_stream_stub_map_test.cpp delete mode 100644 be/test/vec/exec/load_stream_stub_pool_test.cpp copy fe/{fe-core/src/main/java/org/apache/doris/qe/CoordInterface.java => fe-common/src/main/java/org/apache/doris/cloud/security/DummySecurityChecker.java} (64%) create mode 100644 fe/fe-common/src/main/java/org/apache/doris/cloud/security/SecurityChecker.java create mode 100644 fe/fe-common/src/main/java/org/apache/doris/cloud/security/UrlSecurityChecker.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/analysis/ShowStorageVaultStmt.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSCommitter.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/load/TxnDeleteJob.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/transaction/HiveTransactionManager.java copy regression-test/suites/nereids_p0/sql_functions/window_functions/test_sum.groovy => fe/fe-core/src/main/java/org/apache/doris/transaction/Transaction.java (65%) copy regression-test/suites/nereids_p0/sql_functions/window_functions/test_sum.groovy => fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManager.java (65%) copy regression-test/suites/nereids_p0/sql_functions/window_functions/test_sum.groovy => fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionManagerFactory.java (65%) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergPredicateTest.java delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/TestIcebergPredict.java create mode 100644 regression-test/data/cloud_p0/cache/ttl/test_show_ttl.out create mode 100644 regression-test/data/compaction/test_compaction_agg_keys_with_array_map.out copy regression-test/data/{query_p0/test_array_orderby_limit.out => datatype_p0/double/test_double_nan.out} (82%) copy regression-test/data/{query_p0/test_array_orderby_limit.out => datatype_p0/float/test_float_nan.out} (82%) create mode 100644 regression-test/data/external_table_p2/iceberg/test_iceberg_predicate_conversion.out copy regression-test/data/{query_p0/test_array_orderby_limit.out => inverted_index_p0/test_index_delete.out} (58%) create mode 100644 regression-test/data/load_p0/stream_load/test_cast1.csv create mode 100644 regression-test/data/load_p0/stream_load/test_cast1.json create mode 100644 regression-test/data/load_p0/stream_load/test_cast2.csv create mode 100644 regression-test/data/load_p0/stream_load/test_cast2.json create mode 100644 regression-test/data/load_p0/stream_load/test_cast3.csv create mode 100644 regression-test/data/load_p0/stream_load/test_cast3.json create mode 100644 regression-test/data/load_p0/stream_load/test_stream_load_cast.out copy regression-test/data/{query_p0/test_array_orderby_limit.out => mv_p0/test_base/test_base.out} (63%) copy regression-test/data/{query_p0/test_array_orderby_limit.out => mv_p0/test_upper_alias/test_upper_alias.out} (54%) rename regression-test/data/{nereids_shape_check => nereids_broadcast_shuffle_join}/load.out (100%) create mode 100644 regression-test/data/nereids_syntax_p0/order_by_bind_priority.out copy regression-test/data/{query_p0/test_array_orderby_limit.out => plsql_p0/test_plsql_variable.out} (65%) create mode 100644 regression-test/suites/cloud_p0/cache/compaction/test_stale_rowset.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer_delete.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer_load.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer_ttl.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer_ttl_delete.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/customer_ttl_load.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/nation.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/nation_delete.sql create mode 100644 regression-test/suites/cloud_p0/cache/ddl/nation_load.sql create mode 100644 regression-test/suites/cloud_p0/cache/http/test_cache_metrics_format.groovy create mode 100644 regression-test/suites/cloud_p0/cache/http/test_clear_cache.groovy create mode 100644 regression-test/suites/cloud_p0/cache/http/test_clear_cache_async.groovy create mode 100644 regression-test/suites/cloud_p0/cache/load.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_1.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_2.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_3.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_4.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_error.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_max_int64.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_random.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/alter_ttl_seconds.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/create_table_as_select.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/create_table_like.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.sql create mode 100644 regression-test/suites/cloud_p0/cache/ttl/test_ttl.groovy create mode 100644 regression-test/suites/cloud_p0/cache/ttl/test_ttl_preempt.groovy create mode 100644 regression-test/suites/compaction/test_compaction_agg_keys_with_array_map.groovy copy fe/fe-core/src/main/java/org/apache/doris/qe/CoordInterface.java => regression-test/suites/datatype_p0/double/test_double_nan.groovy (62%) copy fe/fe-core/src/main/java/org/apache/doris/qe/CoordInterface.java => regression-test/suites/datatype_p0/float/test_float_nan.groovy (62%) create mode 100644 regression-test/suites/external_table_p2/iceberg/test_iceberg_predicate_conversion.groovy create mode 100644 regression-test/suites/inverted_index_p0/test_index_delete.groovy create mode 100644 regression-test/suites/load_p0/stream_load/test_stream_load_cast.groovy create mode 100644 regression-test/suites/mv_p0/test_base/test_base.groovy create mode 100644 regression-test/suites/mv_p0/test_upper_alias/test_upper_alias.groovy rename regression-test/suites/{nereids_shape_check => nereids_broadcast_shuffle_join}/load.groovy (100%) create mode 100644 regression-test/suites/nereids_syntax_p0/order_by_bind_priority.groovy create mode 100644 regression-test/suites/plsql_p0/test_plsql_variable.groovy create mode 100644 regression-test/suites/statistics/test_statistic_partial_update.groovy create mode 100644 thirdparty/patches/krb5-1.19.patch --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org