This is an automated email from the ASF dual-hosted git repository.
englefly pushed a change to branch topn-lazy-materialize-poc
in repository https://gitbox.apache.org/repos/asf/doris.git
omit e6ad28d15ee fix ut compile after rebase
omit af350dc48a6 set stats for lazyMaterializeFileScan
omit 38ae9a2dd00 fix external core. (#51133)
omit 9005f46b45c support variant alias
omit df4b53e9fe9 fix materialize node local shuffle
omit 5d8ae4f1f87 clickbench conf revise
omit 60092438b28 add query id to print
omit 4f751cb90db Revert "add log to debug the id mapping null"
omit fd08d4e2bfe add rt for only one materialization node
omit 2b6bf8c5409 fmt
omit 7ccdeac8f01 shape-update
omit f38b0fc5f2a only one materializeNode
omit beef9dda6cd skip sync mv
omit 4e9a53bd7b7 fix right join nullable
omit 4b6f44ee5fb add log to debug the id mapping null
omit a34e6fe8a69 fix rebase error
omit dd44c4f40bd fix the ut
omit f44fa81caaf turn off lazy materialization for defer_materialization
cases
omit 99a717411c4 fix the block column not match bug
omit 56a57f09e5b fix alias issue
omit 989900f8dce shape update
omit e14162cf540 fix missing unused key col
omit f5acd43d21f fix the p0 failed by BE
omit 06fde02f84b fix wg null pointer
omit 90207cbea77 fix the ut compile
omit 68a27f1b7fe apply licesen
omit 7e6eb436502 diable key lazy materialize in mor/agg key
omit 8066540e13e fix char padding not remove bug
omit bea2687997d fix typo
omit 6a76e2909c2 translate_fix
omit 746b4dcb349 topn filter ut
omit 4ed1f10d8f1 fix test failed
omit a02d00d4304 fix compile error
omit 4ad6e4d2758 do code format
omit 56aa53200c8 fe
omit 8f560159ed7 Support topn-lazy-materialize
add fdb8409d466 [feat](doris compose) Add extra hosts option for up
command (#51098)
add 8fe3b735904 [Enhancement] (nereids)implement
revokeResourcePrivilegeCommand in ne… (#50419)
add c737662c09b [clean](planner)delete original planner (#51056)
add f4d78f5ef56 [enhancement](cloud-schema-change) Enable new tablet to do
compactionby default (#51070)
add 51f89f1beb4 [Opt](multi-catalog)Disable dict filter in parquet/orc
reader if have non-single conjuncts. (#44777)
add 33e0ec7d3c8 [opt](Nereids) optimize view related capabilities (#51028)
add 5fdaf7b495a [enhancement](cloud) make file cache version upgrade
faster (#50726)
add 4a8d2cc1e18 [fix](cloud-schema-change) Make SC tablet job abort logic
really work (#50908)
add af382359274 [chore](cataglog) Unlimit db data size quota (#51108)
add 8488e9c0c83 [Refactor] Refactor ConnectContext.getCluster (#50783)
add 80c4a0b2c68 [fix](jdbc test) Add more connections to mysql docker
(#50970)
add de1fcd37ce7 [fix](agent) cancel agent task when it is rejected by
agent-task-pool (#51138)
add ef003cde1aa [Fix](orc-reader) Fix merge range not sorted in new merge
io facility of orc-reader. (#51102)
add 22d293269db [Optimize](orc-reader) Optimize stripe footer multiple
reads in orc reader. (#51117)
add f94e0b41ee4 [feat](desc) add comment column in desc statement (#51047)
add 2c257f2a8e4 [fix](cloud) Cloud tablet cache should not be pruned
(#51193)
add f2432586297 [Enhancement] (nereids)implement showStorageVaultCommand
in nereids (#44805)
add 5dd341edddc [bugfix](nerids) complete the implementation of the concat
method. (#51141)
add 2f60e2d7bb9 [Fix](function) Add lost check of function SHA2 in nereids
(#51163)
add 2f795bc21dc [fix](export) remove task from map finally (#51171)
add 17dea2e8b99 fe-part
add 94343f0d662 BE-part
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e6ad28d15ee)
\
N -- N -- N refs/heads/topn-lazy-materialize-poc (94343f0d662)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/actions/action-pr-title | 2 +-
.github/actions/ccache-action | 2 +-
.github/actions/get-workflow-origin | 2 +-
.github/actions/paths-filter | 2 +-
.github/actions/setup-maven | 2 +-
be/src/apache-orc | 2 +-
be/src/cloud/cloud_schema_change_job.cpp | 7 +
be/src/cloud/cloud_tablet_mgr.cpp | 2 +-
be/src/cloud/config.cpp | 2 +-
be/src/common/config.cpp | 9 +-
be/src/common/config.h | 1 +
be/src/common/status.h | 2 +
be/src/io/cache/fs_file_cache_storage.cpp | 267 +-
be/src/io/cache/fs_file_cache_storage.h | 8 +
be/src/io/fs/err_utils.cpp | 2 +
be/src/runtime/exec_env_init.cpp | 3 +-
be/src/vec/exec/format/orc/vorc_reader.cpp | 57 +-
be/src/vec/exec/format/orc/vorc_reader.h | 1 +
.../exec/format/parquet/vparquet_group_reader.cpp | 98 +-
.../exec/format/parquet/vparquet_group_reader.h | 1 -
be/test/io/cache/block_file_cache_test.cpp | 414 ++-
cloud/src/meta-service/meta_service_job.cpp | 4 +-
docker/runtime/doris-compose/Dockerfile | 9 +-
docker/runtime/doris-compose/Readme.md | 31 +
docker/runtime/doris-compose/cluster.py | 33 +-
docker/runtime/doris-compose/command.py | 12 +-
docker/thirdparties/custom_settings.env | 2 +-
docker/thirdparties/docker-compose/mysql/my.cnf | 3 +-
.../main/java/org/apache/doris/common/Config.java | 2 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 26 +-
.../main/java/org/apache/doris/alter/Alter.java | 9 +-
.../java/org/apache/doris/alter/AlterJobV2.java | 11 +
.../org/apache/doris/alter/CloudRollupJobV2.java | 2 +-
.../apache/doris/alter/CloudSchemaChangeJobV2.java | 4 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 6 +-
.../doris/analysis/CreateTableAsSelectStmt.java | 54 +-
.../org/apache/doris/analysis/DescribeStmt.java | 5 +
.../org/apache/doris/cloud/catalog/CloudEnv.java | 5 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 54 +-
.../cloud/datasource/CloudInternalCatalog.java | 3 +-
.../doris/cloud/load/CloudBrokerLoadJob.java | 17 +-
.../doris/cloud/load/CloudLoadLoadingTask.java | 10 +-
.../doris/cloud/system/CloudSystemInfoService.java | 2 +-
.../doris/common/proc/IndexSchemaProcNode.java | 49 +-
.../common/proc/RemoteIndexSchemaProcDir.java | 2 +-
.../common/proc/RemoteIndexSchemaProcNode.java | 2 +-
.../doris/datasource/FederationBackendPolicy.java | 8 +-
.../org/apache/doris/httpv2/rest/LoadAction.java | 4 +-
.../doris/httpv2/rest/TableQueryPlanAction.java | 2 +-
.../apache/doris/load/loadv2/BrokerLoadJob.java | 2 +-
.../apache/doris/load/loadv2/LoadLoadingTask.java | 13 +-
.../org/apache/doris/load/loadv2/LoadManager.java | 26 +-
.../apache/doris/load/loadv2/MysqlLoadManager.java | 14 +
.../doris/load/routineload/RoutineLoadJob.java | 2 +
.../org/apache/doris/master/ReportHandler.java | 10 +-
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 4 +-
.../org/apache/doris/mysql/privilege/Auth.java | 16 +
.../org/apache/doris/nereids/NereidsPlanner.java | 4 +-
.../apache/doris/nereids/analyzer/MappingSlot.java | 5 -
.../apache/doris/nereids/analyzer/UnboundSlot.java | 5 -
.../org/apache/doris/nereids/cost/CostModelV1.java | 8 +-
.../glue/translator/ExpressionTranslator.java | 4 +-
.../glue/translator/PhysicalPlanTranslator.java | 10 +-
.../glue/translator/PlanTranslatorContext.java | 4 +-
.../nereids/load/NereidsLoadPlanInfoCollector.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 54 +-
.../processor/post/RuntimeFilterPruner.java | 4 +-
.../post/materialize/MaterializeProbeVisitor.java | 2 +-
.../doris/nereids/rules/analysis/AnalyzeCTE.java | 2 -
.../nereids/rules/analysis/BindExpression.java | 18 +-
.../doris/nereids/rules/analysis/BindRelation.java | 4 +-
.../mv/AbstractMaterializedViewAggregateRule.java | 4 +-
.../exploration/mv/MaterializedViewUtils.java | 13 +-
.../rules/expression/QueryColumnCollector.java | 4 +-
.../nereids/rules/expression/rules/AddMinMax.java | 2 +-
.../rules/implementation/AggregateStrategies.java | 36 +-
.../LogicalOlapScanToPhysicalOlapScan.java | 8 +-
...ogicalPartitionTopNToPhysicalPartitionTopN.java | 2 +-
.../nereids/rules/rewrite/CheckPrivileges.java | 4 +-
.../nereids/rules/rewrite/ForeignKeyContext.java | 8 +-
.../rules/rewrite/OperativeColumnDerive.java | 2 +-
.../rewrite/PushDownFilterIntoSchemaScan.java | 2 +-
.../nereids/rules/rewrite/SetPreAggStatus.java | 18 +-
.../rules/rewrite/UnequalPredicateInfer.java | 6 +-
.../rules/rewrite/VariantSubPathPruning.java | 10 +-
.../doris/nereids/stats/FilterEstimation.java | 8 +-
.../doris/nereids/stats/StatsCalculator.java | 8 +-
.../doris/nereids/trees/expressions/Alias.java | 26 +-
.../trees/expressions/ArrayItemReference.java | 3 +-
.../nereids/trees/expressions/Expression.java | 6 +-
.../doris/nereids/trees/expressions/Slot.java | 8 +-
.../nereids/trees/expressions/SlotReference.java | 152 +-
.../trees/expressions/functions/agg/AnyValue.java | 2 +-
.../functions/executable/StringArithmetic.java | 9 +-
.../trees/expressions/functions/scalar/Sha2.java | 21 +
.../apache/doris/nereids/trees/plans/PlanType.java | 4 +-
.../trees/plans/commands/AddConstraintCommand.java | 4 +-
.../trees/plans/commands/CopyIntoCommand.java | 1 +
.../commands/CreateMaterializedViewCommand.java | 4 +-
.../trees/plans/commands/CreateTableCommand.java | 4 +-
.../trees/plans/commands/DeleteFromCommand.java | 4 +-
.../trees/plans/commands/DescribeCommand.java | 17 +-
.../commands/RevokeResourcePrivilegeCommand.java | 157 ++
.../plans/commands/ShowStorageVaultCommand.java | 96 +
.../trees/plans/commands/info/AlterViewInfo.java | 10 +-
.../trees/plans/commands/info/CreateViewInfo.java | 3 +-
.../BackendDistributedPlanWorkerManager.java | 70 +-
.../trees/plans/logical/LogicalCTEConsumer.java | 7 +-
.../plans/logical/LogicalCatalogRelation.java | 2 +-
.../trees/plans/logical/LogicalOlapScan.java | 2 +-
.../nereids/trees/plans/logical/LogicalView.java | 16 +-
.../plans/physical/PhysicalCatalogRelation.java | 2 +-
.../plans/physical/PhysicalLazyMaterialize.java | 2 +-
.../trees/plans/visitor/CommandVisitor.java | 10 +
.../org/apache/doris/nereids/util/PlanUtils.java | 4 +-
.../org/apache/doris/planner/AnalyticPlanner.java | 931 -------
.../apache/doris/planner/DistributedPlanner.java | 1343 ---------
.../org/apache/doris/planner/OriginalPlanner.java | 628 -----
.../org/apache/doris/planner/ProjectPlanner.java | 83 -
.../org/apache/doris/planner/RuntimeFilter.java | 44 +-
.../apache/doris/planner/SingleNodePlanner.java | 2921 --------------------
.../java/org/apache/doris/qe/ConnectContext.java | 80 +-
.../org/apache/doris/qe/PointQueryExecutor.java | 2 +-
.../java/org/apache/doris/qe/SessionVariable.java | 9 +
.../java/org/apache/doris/qe/StmtExecutor.java | 32 +-
.../resource/computegroup/ComputeGroupMgr.java | 2 +-
.../resource/computegroup/MergedComputeGroup.java | 16 +-
.../scheduler/manager/TransientTaskManager.java | 2 +-
.../apache/doris/statistics/AnalysisManager.java | 4 +-
.../org/apache/doris/statistics/Statistics.java | 2 +-
.../doris/statistics/hbo/ScanPlanStatistics.java | 4 +-
.../main/java/org/apache/doris/task/AgentTask.java | 11 +
.../org/apache/doris/task/AgentTaskExecutor.java | 19 +-
.../java/org/apache/doris/task/AgentTaskQueue.java | 13 +
.../analysis/CreateTableAsSelectStmtTest.java | 692 -----
.../apache/doris/catalog/CreateFunctionTest.java | 4 +-
.../apache/doris/common/proc/DbsProcDirTest.java | 4 +-
.../org/apache/doris/mtmv/MTMVPlanUtilTest.java | 4 +-
.../nereids/jobs/cascades/DeriveStatsJobTest.java | 5 +-
.../rules/analysis/AnalyzeWhereSubqueryTest.java | 4 +-
.../expression/ExpressionRewriteTestHelper.java | 2 +-
.../rules/rewrite/EliminateGroupByTest.java | 2 +-
.../doris/nereids/stats/StatsCalculatorTest.java | 10 +-
.../doris/nereids/trees/plans/ConstraintTest.java | 4 +-
...ava => RevokeResourcePrivilegeCommandTest.java} | 43 +-
.../doris/planner/SingleNodePlannerTest.java | 1085 --------
.../java/org/apache/doris/qe/CoordinatorTest.java | 877 ------
.../org/apache/doris/qe/OlapQueryCacheTest.java | 16 +
.../apache/doris/resource/ComputeGroupTest.java | 36 +-
.../workloadgroup/WorkloadGroupMgrTest.java | 3 +-
.../doris/statistics/AnalysisManagerTest.java | 7 +-
gensrc/thrift/Status.thrift | 1 +
.../data/ddl_p0/test_create_or_replace_view.out | Bin 505 -> 605 bytes
.../test_external_and_internal_describe.out | Bin 0 -> 2972 bytes
.../test_nereids_alter_database_set_quota.out | Bin 725 -> 731 bytes
.../tpcds_sf100/noStatsRfPrune/query24.out | Bin 3152 -> 3122 bytes
.../tpcds_sf100/noStatsRfPrune/query6.out | Bin 3007 -> 2984 bytes
.../tpcds_sf100/no_stats_shape/query24.out | Bin 3308 -> 3278 bytes
.../tpcds_sf100/no_stats_shape/query6.out | Bin 3217 -> 3194 bytes
.../shape_check/tpcds_sf100/rf_prune/query24.out | Bin 3152 -> 3122 bytes
.../shape_check/tpcds_sf100/rf_prune/query6.out | Bin 3256 -> 3233 bytes
.../data/shape_check/tpcds_sf100/shape/query24.out | Bin 3307 -> 3277 bytes
.../data/shape_check/tpcds_sf100/shape/query6.out | Bin 3310 -> 3287 bytes
.../tpcds_sf1000/bs_downgrade_shape/query6.out | Bin 3310 -> 3287 bytes
.../data/shape_check/tpcds_sf1000/hint/query24.out | Bin 3465 -> 3435 bytes
.../data/shape_check/tpcds_sf1000/hint/query6.out | Bin 3310 -> 3287 bytes
.../shape_check/tpcds_sf1000/shape/query24.out | Bin 3312 -> 3282 bytes
.../data/shape_check/tpcds_sf1000/shape/query6.out | Bin 3310 -> 3287 bytes
.../shape_check/tpcds_sf10t_orc/shape/query24.out | Bin 3302 -> 3272 bytes
.../shape_check/tpcds_sf10t_orc/shape/query6.out | Bin 3215 -> 3192 bytes
.../data/shape_check/tpch_sf1000/hint/q11.out | Bin 2104 -> 2074 bytes
.../tpch_sf1000/nostats_rf_prune/q11.out | Bin 2197 -> 2167 bytes
.../tpch_sf1000/nostats_rf_prune/q22.out | Bin 1376 -> 1350 bytes
.../data/shape_check/tpch_sf1000/rf_prune/q11.out | Bin 2205 -> 2175 bytes
.../data/shape_check/tpch_sf1000/rf_prune/q22.out | Bin 1313 -> 1287 bytes
.../data/shape_check/tpch_sf1000/shape/q11.out | Bin 2205 -> 2175 bytes
.../data/shape_check/tpch_sf1000/shape/q22.out | Bin 1313 -> 1287 bytes
.../shape_check/tpch_sf1000/shape_no_stats/q11.out | Bin 2197 -> 2167 bytes
.../shape_check/tpch_sf1000/shape_no_stats/q22.out | Bin 1376 -> 1350 bytes
.../doris/regression/suite/SuiteCluster.groovy | 17 +-
.../test_base_compaction_after_sc_fail.groovy | 100 +
.../ddl_p0/test_create_or_replace_view.groovy | 5 +
.../test_external_and_internal_describe.groovy | 163 ++
.../suites/mysql_compatibility_p0/metadata.groovy | 59 +-
.../cte/test_cte_with_duplicate_consumer.groovy | 8 +-
.../fold_constant/fold_constant_by_fe.groovy | 4 +-
...oovy => test_show_storage_vault_command.groovy} | 20 +-
.../subquery/test_duplicate_name_in_view.groovy | 175 --
.../test_convert_median_to_percentile.groovy | 4 +-
.../suites/nereids_syntax_p0/explain.groovy | 5 +-
.../query_p0/cte/query_with_dup_column.groovy | 112 -
.../encryption_digest/test_digest.groovy | 25 +-
192 files changed, 2191 insertions(+), 9702 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RevokeResourcePrivilegeCommand.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowStorageVaultCommand.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/AnalyticPlanner.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/DistributedPlanner.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/ProjectPlanner.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/analysis/CreateTableAsSelectStmtTest.java
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/{GrantResourcePrivilegeCommandTest.java
=> RevokeResourcePrivilegeCommandTest.java} (80%)
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/planner/SingleNodePlannerTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/qe/CoordinatorTest.java
create mode 100644
regression-test/data/external_table_p0/test_external_and_internal_describe.out
create mode 100644
regression-test/suites/compaction/test_base_compaction_after_sc_fail.groovy
create mode 100644
regression-test/suites/external_table_p0/test_external_and_internal_describe.groovy
copy regression-test/suites/nereids_p0/show/{test_help_command.groovy =>
test_show_storage_vault_command.groovy} (62%)
delete mode 100644
regression-test/suites/nereids_p0/subquery/test_duplicate_name_in_view.groovy
delete mode 100644
regression-test/suites/query_p0/cte/query_with_dup_column.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
