This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a change to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
discard becace04d80 refactor2 (#32278)
omit b5ba91a9ec0 Refactor. (#32273)
omit c73acc0b8ea support window (#32094)
omit 16a1d332ae7 Check column health value earlier, show job priority.
(#32064)
omit b7145ca484b Support show auto analyze pending jobs. (#31926)
omit 9764ef3aa89 Support follower sync query columns to master. (#31859)
omit 5cac30ed252 Support column level health value. (#31794)
omit 077a4c0575c High priority queue and map. (#31509)
omit 0b5a2a58b8c Collect high priority columns. (#31235)
add 0292dd479b4 [feature](pipelineX) support paritition tablet sink
shuffle (#31689)
add 43ce4864fb2 [opt](Nereids) outer join with is null stats estimation
enhancement (#31875)
add f3fdb306e4f [feat](nereids) add merge aggregate rule (#31811)
add 684766e8ce0 [Enhancement](jdbc catalog) Add a property to test the
connection when creating a Jdbc catalog (#32125)
add 7786f95682a [fix](Nereids): ignore project and distribute in
test_cte_filter_pushdown and push_down_expression_in_hash_join (#32083)
add 7d5a64e6be2 [fix](nereids)EliminateSemiJoin should consider empty
table (#32107)
add e368c36589b [enhancement](Nereids) Optimize expression (#32067)
add 3842427c487 [Chore](workflow) do not check java env on clang-tidy
preparation (#32161)
add b4f471050a7 [fix](merge-clod) fix file not found when load for mow
table (#32144)
add 99680e9ec13 [Bug](top-n) do not update topn filter when sort node and
scan node are not in the… (#32159)
add ca092139a00 [fix](Nereids): slot set in condition can be empty (#32169)
add 9f628f3449b [improvement](mtmv) Remove useless code and add some doc
(#32047)
add 50b1aa6e3db [fix](merge-cloud) fix no cluster for common user (#32097)
add 954750b59a8 [Fix](smooth-upgrade) Fix incompatibility when upgrade
from 2.0 to 2.1
add 31da1bb9e42 [fix](regression-test) upgrade groovy to 4.x and enable
run test by jdk17/21 again
add ae7d0d2bbad [Opt](exec) Reduce unnecessary find function calls (#32165)
add f2373fcc948 [fix](index compaction)Fix MOW index compaction core
(#32121)
add 0e344246aab [Optimization][Scanner] Skip `_init_variant_columns` when
there are no variant columns, and ensure `inherit_tablet_index` is called only
once (#32174)
add 0b0e527ce1f [enhancement](cloud) Enable FileCache for internal
statistics info table (#32142)
add 191982c8782 [Fix](Variant) fix inverted index lost due to wrong column
unique id (#31992)
add 7137ed36be3 [fix](routineload) fix consume data too slow in partial
partitions (#32126)
add 65fc66072bb [fix](merge-cloud) fix cloud cumu compaction thread not
initializing thread loaca context (#32181)
add 67064e1442b [improvement](Nereids) Support to remove sort which is
under table sink (#31751)
add 50be40429f3 [enhancement](nereids) support unnest subquery with group
by and having clause (#32002)
add 70e1a17c82a [chore](tools) update tools config (#32193)
add 3e8e3a7660e [Chore](build) add DISABLE_BE_JAVA_EXTENSIONS to generate
compile_commands.json (#32189)
add 74660237ed5 [fix](jdbc catalog) Fix the wrong method used in
loadColumnNames of JdbcIdentifierMapping (#32168)
add cfa403fb900 [fix](mysql) Fix Connection Attributes Parsing Logic for
Compatibility (#32177)
add 50300474cf3 [fix](script) Fix the JAVA_OPTS set on macOS (#32170)
add 3a4c1ea73e7 [fix](merge-cloud) fix check failed in cloud delta writer
(#32197)
new 32184b35cb4 Collect high priority columns. (#31235)
new 452646e86d1 High priority queue and map. (#31509)
new b8f61c4b6a7 Support column level health value. (#31794)
new 8bccfb5cc15 Support follower sync query columns to master. (#31859)
new f74a289e2f2 Support show auto analyze pending jobs. (#31926)
new 4d426f6dbaa Check column health value earlier, show job priority.
(#32064)
new c707a66a243 support window (#32094)
new e76be97881b Refactor. (#32273)
new 3fb4b214172 refactor2 (#32278)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (becace04d80)
\
N -- N -- N refs/heads/high-priority-column (3fb4b214172)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/code-checks.yml | 2 +-
.github/workflows/sonarcloud.yml | 2 +-
be/src/agent/be_exec_version_manager.h | 9 +-
be/src/cloud/cloud_cumulative_compaction.cpp | 3 +
be/src/cloud/cloud_delta_writer.cpp | 2 +-
be/src/clucene | 2 +-
be/src/common/config.cpp | 2 +
be/src/common/config.h | 2 +
be/src/exprs/runtime_filter.cpp | 3 +-
be/src/http/action/stream_load.cpp | 5 +-
be/src/index-tools/index_tool.cpp | 143 +++++++++++-
be/src/io/fs/hdfs_file_system.h | 2 +-
be/src/olap/compaction.cpp | 39 ++++
be/src/olap/rowset/beta_rowset_writer.cpp | 5 +
be/src/olap/rowset/segment_creator.cpp | 13 +-
be/src/olap/rowset/segment_creator.h | 9 +-
be/src/olap/tablet_reader.cpp | 2 -
be/src/olap/tablet_schema.cpp | 6 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 142 ++++++++++--
be/src/pipeline/exec/exchange_sink_operator.h | 34 ++-
be/src/pipeline/exec/sort_sink_operator.cpp | 14 +-
be/src/pipeline/task_scheduler.cpp | 9 +-
be/src/runtime/load_channel_mgr.cpp | 6 +-
be/src/runtime/routine_load/data_consumer_group.h | 8 +-
be/src/runtime/routine_load/data_consumer_pool.cpp | 5 +-
be/src/runtime/runtime_predicate.h | 10 +-
be/src/service/internal_service.cpp | 60 +++++
be/src/service/internal_service.h | 5 +
.../aggregate_function_bitmap.h | 19 +-
.../aggregate_function_window_funnel.h | 5 +-
be/src/vec/exec/scan/new_olap_scanner.cpp | 5 +-
be/src/vec/exec/vjdbc_connector.cpp | 27 ++-
be/src/vec/exec/vjdbc_connector.h | 5 +
be/src/vec/exec/vsort_node.cpp | 14 +-
be/src/vec/functions/function_utility.cpp | 12 +-
be/src/vec/functions/simple_function_factory.h | 4 +-
be/src/vec/sink/load_stream_stub.cpp | 22 +-
be/src/vec/sink/load_stream_stub.h | 2 +-
be/src/vec/sink/vdata_stream_sender.cpp | 44 ++--
be/src/vec/sink/writer/async_result_writer.cpp | 7 +-
be/src/vec/sink/writer/vtablet_writer_v2.cpp | 2 +-
be/test/exec/test_data/wal_scanner/wal | Bin 220 -> 180 bytes
bin/start_be.sh | 4 +-
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 2 +
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 2 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 3 +-
.../Create/CREATE-ASYNC-MATERIALIZED-VIEW.md | 1 +
env.sh | 2 +-
.../org/apache/doris/jdbc/BaseJdbcExecutor.java | 20 ++
.../org/apache/doris/jdbc/DefaultJdbcExecutor.java | 20 ++
.../main/java/org/apache/doris/common/Config.java | 5 +-
.../org/apache/doris/catalog/JdbcResource.java | 9 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 99 ++++++++
.../datasource/jdbc/JdbcIdentifierMapping.java | 2 +-
.../doris/datasource/jdbc/client/JdbcClient.java | 18 ++
.../datasource/jdbc/client/JdbcOracleClient.java | 5 +
.../datasource/jdbc/client/JdbcSapHanaClient.java | 5 +
.../datasource/mapping/IdentifierMapping.java | 3 +-
.../main/java/org/apache/doris/mtmv/MTMVCache.java | 19 +-
.../org/apache/doris/mysql/MysqlAuthPacket.java | 5 +-
.../doris/nereids/jobs/executor/Rewriter.java | 51 +----
.../jobs/joinorder/hypergraph/edge/JoinEdge.java | 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 3 +
.../nereids/rules/analysis/FillUpMissingSlots.java | 8 +-
.../nereids/rules/analysis/NormalizeAggregate.java | 6 +-
.../nereids/rules/analysis/NormalizeRepeat.java | 12 +-
.../rules/exploration/join/InnerJoinLAsscom.java | 6 +-
.../doris/nereids/rules/rewrite/ColumnPruning.java | 46 +++-
.../rules/rewrite/ConvertInnerOrCrossJoin.java | 4 +-
.../rules/rewrite/ConvertOuterJoinToAntiJoin.java | 4 +-
.../rewrite/EliminateNullAwareLeftAntiJoin.java | 2 +-
.../nereids/rules/rewrite/EliminateOuterJoin.java | 4 +-
.../nereids/rules/rewrite/EliminateSemiJoin.java | 58 ++---
.../doris/nereids/rules/rewrite/EliminateSort.java | 17 +-
.../nereids/rules/rewrite/MergeAggregate.java | 211 ++++++++++++++++++
.../nereids/rules/rewrite/NormalizeToSlot.java | 28 ++-
...CorrelatedFilterUnderApplyAggregateProject.java | 66 ++++--
.../rewrite/TransposeSemiJoinLogicalJoin.java | 2 +-
.../TransposeSemiJoinLogicalJoinProject.java | 10 +-
.../rewrite/UnCorrelatedApplyAggregateFilter.java | 85 ++++---
.../doris/nereids/stats/FilterEstimation.java | 31 ++-
.../doris/nereids/stats/StatsCalculator.java | 112 +++++++++-
.../org/apache/doris/nereids/trees/TreeNode.java | 4 +-
.../doris/nereids/trees/expressions/Alias.java | 50 +++--
.../doris/nereids/trees/expressions/Any.java | 4 +
.../trees/expressions/ArrayItemReference.java | 4 +-
.../nereids/trees/expressions/BinaryOperator.java | 12 -
.../nereids/trees/expressions/Expression.java | 69 ++++--
.../trees/expressions/MarkJoinSlotReference.java | 2 +-
.../nereids/trees/expressions/SlotReference.java | 54 +++--
.../trees/expressions/VirtualSlotReference.java | 6 +-
.../nereids/trees/expressions/WhenClause.java | 15 --
.../trees/expressions/WindowExpression.java | 2 +-
.../trees/expressions/functions/BoundFunction.java | 26 +--
.../doris/nereids/trees/plans/AbstractPlan.java | 14 +-
.../org/apache/doris/nereids/trees/plans/Plan.java | 8 +-
.../nereids/trees/plans/logical/LogicalFilter.java | 6 +
.../trees/plans/logical/LogicalHiveTableSink.java | 2 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 9 +-
.../trees/plans/logical/LogicalOlapTableSink.java | 2 +-
.../{OutputPrunable.java => LogicalTableSink.java} | 21 +-
.../nereids/trees/plans/logical/LogicalUnion.java | 6 +
.../plans/physical/PhysicalHiveTableSink.java | 2 +-
.../plans/physical/PhysicalOlapTableSink.java | 2 +-
.../{PhysicalSink.java => PhysicalTableSink.java} | 24 +-
.../nereids/trees/plans/visitor/SinkVisitor.java | 10 +
.../apache/doris/nereids/util/ExpressionUtils.java | 16 +-
.../org/apache/doris/nereids/util/PlanUtils.java | 86 +++++--
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../org/apache/doris/rpc/BackendServiceClient.java | 5 +
.../org/apache/doris/rpc/BackendServiceProxy.java | 12 +
.../org/apache/doris/statistics/AnalysisJob.java | 2 +-
.../apache/doris/statistics/OlapAnalysisTask.java | 3 +-
.../org/apache/doris/statistics/Statistics.java | 4 +
.../doris/statistics/util/StatisticsUtil.java | 22 +-
.../rules/analysis/FillUpMissingSlotsTest.java | 2 +-
.../rules/rewrite/EliminateSemiJoinTest.java | 71 +++++-
.../nereids/rules/rewrite/EliminateSortTest.java | 110 ++++++++-
.../doris/nereids/stats/FilterEstimationTest.java | 2 +-
.../doris/nereids/util/HyperGraphBuilder.java | 4 +-
gensrc/proto/internal_service.proto | 11 +
.../jdbc/test_clickhouse_jdbc_catalog.out | Bin 4991 -> 5426 bytes
.../nereids_p0/cte/test_cte_filter_pushdown.out | 69 ++----
.../nereids_p0/subquery/subquery_unnesting.out | 11 +
.../cte/test_cte_filter_pushdown.out | 69 ++----
.../eliminate_join_condition.out | 4 +-
.../push_down_expression_in_hash_join.out | 158 ++++---------
.../merge_aggregate/merge_aggregate.out | 248 +++++++++++++++++++++
regression-test/framework/pom.xml | 89 +++-----
.../jdbc/test_clickhouse_jdbc_catalog.groovy | 30 ++-
.../test_insert_from_tvf_with_common_user.groovy | 8 +
.../test_memtable_flush_fault.groovy | 3 +-
.../load_p0/stream_load/test_stream_load.groovy | 8 +
.../nereids_p0/cte/test_cte_filter_pushdown.groovy | 1 +
.../test_outerjoin_isnull_estimation.groovy | 54 +++++
.../insert_into_table/no_partition.groovy | 11 -
.../nereids_p0/subquery/subquery_unnesting.groovy | 1 +
.../cte/test_cte_filter_pushdown.groovy | 1 +
.../push_down_expression_in_hash_join.groovy | 1 +
.../merge_aggregate/merge_aggregate.groovy | 177 +++++++++++++++
thirdparty/build-thirdparty.sh | 4 +-
tools/tpcds-tools/conf/opt/opt_sf1.sql | 1 -
tools/tpcds-tools/conf/opt/opt_sf100.sql | 1 -
tools/tpch-tools/conf/opt/opt_sf1.sql | 6 -
tools/tpch-tools/conf/opt/opt_sf100.sql | 7 -
tools/tpch-tools/conf/opt/opt_sf1000.sql | 6 -
tools/tpch-tools/conf/opt/opt_sf10000.sql | 6 -
147 files changed, 2559 insertions(+), 825 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeAggregate.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/{OutputPrunable.java
=> LogicalTableSink.java} (55%)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/{PhysicalSink.java
=> PhysicalTableSink.java} (66%)
create mode 100644
regression-test/data/nereids_rules_p0/merge_aggregate/merge_aggregate.out
create mode 100644
regression-test/suites/nereids_p0/explain/test_outerjoin_isnull_estimation.groovy
create mode 100644
regression-test/suites/nereids_rules_p0/merge_aggregate/merge_aggregate.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
