[GitHub] [doris] 924060929 merged pull request #13957: [fix](Nerieds) fix tpch and support trace plan's change event
924060929 merged PR #13957: URL: https://github.com/apache/doris/pull/13957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](Nerieds) fix tpch and support trace plan's change event (#13957)
This is an automated email from the ASF dual-hosted git repository. huajianlan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 06a1efdb01 [fix](Nerieds) fix tpch and support trace plan's change event (#13957) 06a1efdb01 is described below commit 06a1efdb018e6328e1b2e7426f193433602ed721 Author: 924060929 <924060...@qq.com> AuthorDate: Fri Nov 4 15:01:06 2022 +0800 [fix](Nerieds) fix tpch and support trace plan's change event (#13957) This pr fix some bugs for run tpc-h 1. fix the avg(decimal) crash the backend. The fix code in `Avg.getFinalType()` and every child class of `ComputeSinature` 2. fix the ReorderJoin dead loop. The fix code in `ReorderJoin.findInnerJoin()` 3. fix the TimestampArithmetic can not bind the functions in the child. The fix code in `BindFunction.FunctionBinder.visitTimestampArithmetic()` New feature: support trace the plan's change event, you can `set enable_nereids_trace=true` to open trace log and see some log like this: ``` 2022-11-03 21:07:38,391 INFO (mysql-nio-pool-0|208) [Job.printTraceLog():128] == RewriteBottomUpJob ANALYZE_FILTER_SUBQUERY == before: LogicalProject ( projects=[S_ACCTBAL#17, S_NAME#13, N_NAME#4, P_PARTKEY#19, P_MFGR#21, S_ADDRESS#14, S_PHONE#16, S_COMMENT#18] ) +--LogicalFilter ( predicates=P_PARTKEY#19 = PS_PARTKEY#7) AND (S_SUPPKEY#12 = PS_SUPPKEY#8)) AND (P_SIZE#24 = 15)) AND (P_TYPE#23 like '%BRASS')) AND (S_NATIONKEY#15 = N_NATIONKEY#3)) AND (N_REGIONKEY#5 = R_REGIONKEY#0)) AND (R_NAME#1 = 'EUROPE')) AND (PS_SUPPLYCOST#10 = (SCALARSUBQUERY) (QueryPlan: LogicalAggregate ( phase=LOCAL, outputExpr=[min(PS_SUPPLYCOST#31) AS `min(PS_SUPPLYCOST)`#33], groupByExpr=[] )), (CorrelatedSlots: [P_PARTKEY#19, S_SUPPKEY#12, S_NATIONKEY#15, N [...] +--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | | |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | | | |--LogicalOlapScan ( qualified=default_cluster:regression_test_tpch_sf1_p1_tpch_sf1.part, output=[P_PARTKEY#19, P_NAME#20, P_MFGR#21, P_BRAND#22, P_TYPE#23, P_SIZE#24, P_CONTAINER#25, P_RETAILPRICE#26, P_COMMENT#27], candidateIndexIds=[], selectedIndexId=11076, preAgg=ON ) | | | +--LogicalOlapScan ( qualified=default_cluster:regression_test_tpch_sf1_p1_tpch_sf1.supplier, output=[S_SUPPKEY#12, S_NAME#13, S_ADDRESS#14, S_NATIONKEY#15, S_PHONE#16, S_ACCTBAL#17, S_COMMENT#18], candidateIndexIds=[], selectedIndexId=11124, preAgg=ON ) | | +--LogicalOlapScan ( qualified=default_cluster:regression_test_tpch_sf1_p1_tpch_sf1.partsupp, output=[PS_PARTKEY#7, PS_SUPPKEY#8, PS_AVAILQTY#9, PS_SUPPLYCOST#10, PS_COMMENT#11], candidateIndexIds=[], selectedIndexId=11092, preAgg=ON ) | +--LogicalOlapScan ( qualified=default_cluster:regression_test_tpch_sf1_p1_tpch_sf1.nation, output=[N_NATIONKEY#3, N_NAME#4, N_REGIONKEY#5, N_COMMENT#6], candidateIndexIds=[], selectedIndexId=11044, preAgg=ON ) +--LogicalOlapScan ( qualified=default_cluster:regression_test_tpch_sf1_p1_tpch_sf1.region, output=[R_REGIONKEY#0, R_NAME#1, R_COMMENT#2], candidateIndexIds=[], selectedIndexId=11108, preAgg=ON ) after: LogicalProject ( projects=[S_ACCTBAL#17, S_NAME#13, N_NAME#4, P_PARTKEY#19, P_MFGR#21, S_ADDRESS#14, S_PHONE#16, S_COMMENT#18] ) +--LogicalFilter ( predicates=P_PARTKEY#19 = PS_PARTKEY#7) AND (S_SUPPKEY#12 = PS_SUPPKEY#8)) AND (P_SIZE#24 = 15)) AND (P_TYPE#23 like '%BRASS')) AND (S_NATIONKEY#15 = N_NATIONKEY#3)) AND (N_REGIONKEY#5 = R_REGIONKEY#0)) AND (R_NAME#1 = 'EUROPE')) AND (PS_SUPPLYCOST#10 = min(PS_SUPPLYCOST)#33)) ) +--LogicalProject ( projects=[P_PARTKEY#19, P_NAME#20, P_MFGR#21, P_BRAND#22, P_TYPE#23, P_SIZE#24, P_CONTAINER#25, P_RETAILPRICE#26, P_COMMENT#27, S_SUPPKEY#12, S_NAME#13, S_ADDRESS#14, S_NATIONKEY#15, S_PHONE#16, S_ACCTBAL#17, S_COMMENT#18, PS_PARTKEY#7, PS_SUPPKEY#8, PS_AVAILQTY#9, PS_SUPPLYCOST#10, PS_COMMENT#11, N_NATIONKEY#3, N_NAME#4, N_REGIONKEY#5, N_COMMENT#6, R_REGIONKEY#0, R_NAME#1, R_COMMENT#2, min(PS_SUPPLYCOST)#33] ) +--LogicalApply ( correlationSlot=[P_PARTKEY#19, S_SUPPKEY#12, S_NATIONKEY#15, N_NATIONKEY#3, N_REGIONKEY#5, R_REGIONKEY#0, R_NAME#1], correlationFilter=Optional.empty ) |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | | |--LogicalJoin ( type=CROSS_JOIN, hashJoinCondition=[], otherJoinCondition=[] ) | | | |--LogicalJoin ( ty
[GitHub] [doris] nextdreamblue commented on a diff in pull request #12801: [enhancement](broker) broker load support tencent cos
nextdreamblue commented on code in PR #12801:
URL: https://github.com/apache/doris/pull/12801#discussion_r1013557768
##
fs_brokers/apache_hdfs_broker/build.sh:
##
@@ -41,6 +41,10 @@ rm -rf "${BROKER_OUTPUT}"
install -d "${BROKER_OUTPUT}/bin" "${BROKER_OUTPUT}/conf" \
"${BROKER_OUTPUT}/lib"
+# download tencent hadoop cos jar
+wget
https://github.com/tencentyun/hadoop-cos/releases/download/v8.1.7/hadoop-cos-2.7.5-8.1.7.jar
+mv hadoop-cos-2.7.5-8.1.7.jar "${BROKER_OUTPUT}/lib/"
+
Review Comment:
1.目前 Hadoop-COS 已经正式被 Apache Hadoop-3.3.0集成。但是目前doris依赖的hadoop版本比较低，无法使用该版本；
2.在 Apache Hadoop-3.3.0之前的版本，需要下载单独 Hadoop-cos jar 包。
3.目前单独的Hadoop-cos jar 包，腾讯云未提交到maven，因此目前只能使用单独下载的方式
后续如果hadoop库升级或者支持maven方式下载，再调整此处的下载方式
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated (ba918b40e2 -> 06a1efdb01)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git from ba918b40e2 [chore](macOS) Fix compilation errors caused by the deprecated function (#13890) add 6f3db8b4b4 [enhancement](Nereids) add eliminate unnecessary project rule (#13886) add f2a0adf34e [fix](fe) Inconsistent behavior for string comparison in FE and BE (#13604) add 699ffbca0e [enhancement](Nereids) generate correct distribution spec after project (#13725) add 1bafb26217 [fix](Nereids) throw NPE when call getOutputExprIds in LogicalProperties (#13898) add a871fef815 [Improve](Nereids): refactor eliminate outer join (#13402) add 6eea855e78 [feature](Nereids) Support lots of scalar function and fix some bug (#13764) add b26d8f284c [fix](rpc) The proxy removed when rpc exception occurs is not an abnormal proxy (#13836) add 374303186c [Vectorized](function) support topn_array function (#13869) add 53814e466b [Enhancement](Nereids)optimize merge group in memo #13900 add 62f765b7f5 [improvement](scan) speed up inserting strings into ColumnString (#13397) add fbc8b7311f [Opt](function) opt the function of ndv (#13887) add 0ea7f85986 [fix](keyword) add BIN as keyword (#13907) add b83744d2f6 [feature](function)add regexp functions: regexp_replace_one, regexp_extract_all (#13766) add e021705053 [feature](nereids) support common table expression (#12742) add 228e5afad8 [Load](Sink) remove validate the column data when data is NULL (#13919) add 7b4c2cabb4 [feature](new-scan) support transactional insert in new scan framework (#13858) add 636bdffe62 [fix](doc) fix 404 link (#13908) add cc37ef0f2b [regression-test](query) Add the regression case of the query under the large wide table. #13897 add 28a4a8dc17 [fix](storage) evaluate_and of ComparisonPredicateBase has logical error (#13895) add 1ee6518e00 [fix](unique-key-merge-on-write) Types don't match when calling IndexedColumnIterator::seek_at_or_after (#13885) add 37e4a1769d [fix](sequence) fix that update table core dump with sequence column (#13847) add b3c6af0059 [Bugfix](MV) Fixed load negative values into bitmap type materialized views successfully under non-vectorization (#13719) add 32a029d9dc [enhancement](memtracker) Refactor load channel + memtable mem tracker (#13795) add 5a700223fe [fix](function) fix coredump cause by return type mismatch of vectorized repeat function (#13868) add a4a991207b [fix](agg)fix group by constant value bug (#13827) add 31d8fdd9e4 [fix](Nereids) finalize local aggregate should not turn on stream pre agg (#13922) add 57ee5c4a65 [feature](nereids) Support authentication (#13434) add bfba058ecf [Feature](join) Support null aware left anti join (#13871) add 29e01db7ce [Fix](Nereids) add comments to CostAndEnforcerJob and fix view test case (#13046) add 5fe3342aa3 [Vectorized](function) support bitmap_to_array function (#13926) add 6ff306b1ea [docs](round) complement round function documentation (#13838) add b1816d49e7 [fix](typo) check catalog enable exception message spelling mistake (#13925) add ee934483eb [Enhancement](function) optimize the `upper` and `lower` functions using the simd instruction. (#13326) add c1438cbad6 [revert](Nereids): revert GroupExpression Children ImmutableList. (#13918) add 8043418db4 [optimization](array-type) update the exception message when create table with array column (#13731) add d183199319 [Bug](array-type) Fix array product calculate decimal type return wrong result (#13794) add ff935ca1a0 [enhancement](chore) remove debug log which is really too frequent #13909 add 1b36843664 [doc](jsonb type)add documents for JSONB datatype (#13792) add 5d7b51dcc2 [BugFix](Concat) output of string concat function exceeds UINT makes crash (#13916) add 0a228a68d6 [Improvement](javaudf) support different date argument for date/datetime type (#13920) add 9869915279 [refactor](crossjoin) refactor cross join (#13896) add 5d56fe6d32 [fix](meta)(recover) fix recover info persist bug (#13948) add 698541e58d [improvement](exec) add more debug info on fragment exec error (#13899) add f2d84d81e6 [feature-wip][refactor](multi-catalog) Persist external catalog related metadata. (#13746) add e09033276e [fix](runtime-filter) build thread destruct first may cause probe thread coredump (#13911) add efb2596c7a [enhancment](Nereids) enable push down filter through aggregation (#13938) add 9bf20a7b5d [enhancement](Nereids) remove unnecessary int cast (#13881) add dc01fb4085 [enhancement](Nereids) remove unnecessary string cast (#13730) add 948e080b31 [minor](error msg) Fix wrong error message (#13950) add 554f566217 [enhancement](compaction) introduce segm
[GitHub] [doris] Jibing-Li commented on a diff in pull request #13883: [feature-wip](nereids) Statistics persistence
Jibing-Li commented on code in PR #13883:
URL: https://github.com/apache/doris/pull/13883#discussion_r1013687795
##
fe/fe-core/src/main/java/org/apache/doris/statistics/DropAnalysisJobLog.java:
##
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics;
+
+import org.apache.doris.common.io.Writable;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+public class DropAnalysisJobLog implements Writable {
+public final long id;
+
+public DropAnalysisJobLog(long id) {
+this.id = id;
+}
+
+@Override
+public void write(DataOutput out) throws IOException {
+out.writeLong(id);
Review Comment:
Use Gson, see other Writable classes, e.g.
fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogLog.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #12620: [refactor](cv)wait on condition variable more gently
yiguolei commented on code in PR #12620:
URL: https://github.com/apache/doris/pull/12620#discussion_r1013691162
##
be/src/util/blocking_queue.hpp:
##
@@ -47,25 +48,19 @@ class BlockingQueue {
// are no more elements available.
bool blocking_get(T* out) {
MonotonicStopWatch timer;
+timer.start();
std::unique_lock unique_lock(_lock);
-
-while (true) {
-if (!_list.empty()) {
-*out = _list.front();
-_list.pop_front();
-_total_get_wait_time += timer.elapsed_time();
-unique_lock.unlock();
-_put_cv.notify_one();
-return true;
-}
-
-if (_shutdown) {
-return false;
-}
-
-timer.start();
-_get_cv.wait(unique_lock);
-timer.stop();
+_get_cv.wait(unique_lock, [this] { return _shutdown || !_list.empty();
});
+_total_get_wait_time += timer.elapsed_time();
+
Review Comment:
timer.stop lost? or it is useless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013691586
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupingSets.java:
##
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Grouping sets.
+ */
+public class LogicalGroupingSets extends
LogicalRepeat {
+private final List> groupingSets;
+
+public LogicalGroupingSets(List> groupingSets,
+List outputExpressions,
+CHILD_TYPE child) {
+this(groupingSets, genOriginalGroupByExpressions(groupingSets),
outputExpressions,
+Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalGroupingSets(List> groupingSets,
+List groupByExpressions,
+List outputExpressions,
+List groupingIdList,
+Set virtualSlotRefs,
+List virtualGroupingExprs,
+boolean changedOutput,
+boolean isNormalized,
+CHILD_TYPE child) {
+this(groupingSets, groupByExpressions, outputExpressions,
groupingIdList,
+virtualSlotRefs, virtualGroupingExprs, changedOutput,
+isNormalized,
+Optional.empty(), Optional.empty(), child);
+}
+
+public LogicalGroupingSets(List> groupingSets,
+List groupByExpressions,
+List outputExpressions,
+Optional groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions,
outputExpressions,
+groupExpression, logicalProperties, child);
+this.groupingSets = groupingSets;
+}
+
+public LogicalGroupingSets(
+List> groupingSets,
+List groupByExpressions,
+List outputExpressions,
+List groupingIdList,
+Set virtualSlotRefs,
+List virtualGroupingExprs,
+boolean changedOutput,
+boolean isNormalized,
+Optional groupExpression,
+Optional logicalProperties, CHILD_TYPE child) {
+super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions,
outputExpressions, groupingIdList,
+virtualSlotRefs, virtualGroupingExprs, changedOutput,
+isNormalized,
+groupExpression, logicalProperties, child);
+this.groupingSets = groupingSets;
+}
+
+@Override
+public List> getGroupingSets() {
+return groupingSets;
+}
+
+@Override
+public String toString() {
+return Utils.toSqlString("LogicalGroupingSets",
+"outputExpressions", outputExpressions,
+"grouping sets", groupingSets,
+"original groupByExpressions", groupByExpressions,
+"groupByExpressions", getGroupByExpressions()
+);
+}
+
+@Override
+public R accept(PlanVisitor visitor, C context) {
+return visitor.visitLogicalGroupingSets(this, context);
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+
[GitHub] [doris] yiguolei commented on a diff in pull request #12620: [refactor](cv)wait on condition variable more gently
yiguolei commented on code in PR #12620:
URL: https://github.com/apache/doris/pull/12620#discussion_r1013694741
##
be/src/util/blocking_priority_queue.hpp:
##
@@ -49,143 +49,126 @@ class BlockingPriorityQueue {
// -- timeout_ms: 0 means wait indefinitely
bool blocking_get(T* out, uint32_t timeout_ms = 0) {
MonotonicStopWatch timer;
+timer.start();
std::unique_lock unique_lock(_lock);
-
-while (true) {
-if (!_queue.empty()) {
-// 定期提高队列中残留的任务优先级
-// 保证优先级较低的大查询不至于完全饿死
-if (_upgrade_counter >
config::priority_queue_remaining_tasks_increased_frequency) {
-std::priority_queue tmp_queue;
-while (!_queue.empty()) {
-T v = _queue.top();
-_queue.pop();
-++v;
-tmp_queue.push(v);
-}
-swap(_queue, tmp_queue);
-_upgrade_counter = 0;
+bool wait_successful = false;
+if (timeout_ms > 0) {
+wait_successful = _get_cv.wait_for(unique_lock,
std::chrono::milliseconds(timeout_ms),
+ [this] { return _shutdown ||
!_queue.empty(); });
+} else {
+_get_cv.wait(unique_lock, [this] { return _shutdown ||
!_queue.empty(); });
+wait_successful = true;
+}
+_total_get_wait_time += timer.elapsed_time();
Review Comment:
我感觉这个语义不一致了， 在原来的代码里，即使wait 失败，也会调整队列中已有元素的优先级； 但是在你改的新的代码里，只有wait 成功了，才调整？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013695443
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalRepeat.java:
##
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Logical Repeat.
+ */
+public abstract class LogicalRepeat
+extends LogicalUnary implements Repeat {
+
+public static final String COL_GROUPING_ID = "GROUPING_ID";
+public static final String GROUPING_PREFIX = "GROUPING_PREFIX_";
+// max num of distinct sets in grouping sets clause
+public static final int MAX_GROUPING_SETS_NUM = 64;
+protected final List groupByExpressions;
+protected final List outputExpressions;
+protected final Set virtualSlotRefs;
+protected final List groupingIdList;
+protected final List virtualGroupingExprs;
+protected final boolean changedOutput;
+protected final boolean isNormalized;
Review Comment:
We should have some comments for these members.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat
+extends PhysicalUnary implements Repeat {
+protected final List groupByExpressions;
+protected final List originGroupByExpressions;
+protected final List outputExpressions;
+protected final List groupingIdList;
+protected final List virtualGroupByExpressions;
Review Comment:
We should add some com
[GitHub] [doris] morningman commented on a diff in pull request #13959: [feature](table-valued-function) Support S3 tvf
morningman commented on code in PR #13959:
URL: https://github.com/apache/doris/pull/13959#discussion_r1013701332
##
fe/fe-core/src/main/java/org/apache/doris/planner/DataGenScanNode.java:
##
@@ -40,15 +40,15 @@
/**
* This scan node is used for table valued function.
Review Comment:
Change the comment
##
be/src/service/internal_service.cpp:
##
@@ -407,6 +410,78 @@ void
PInternalServiceImpl::fetch_data(google::protobuf::RpcController* cntl_base
_exec_env->result_mgr()->fetch_data(request->finst_id(), ctx);
}
+void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController*
controller,
+ const PFetchTableSchemaRequest*
request,
+ PFetchTableSchemaResult* result,
+ google::protobuf::Closure* done)
{
+VLOG_RPC << "fetch table schema";
+brpc::ClosureGuard closure_guard(done);
+TFileScanRange file_scan_range;
+Status st = Status::OK();
+{
+const uint8_t* buf = (const
uint8_t*)(request->file_scan_range().data());
+uint32_t len = request->file_scan_range().size();
+st = deserialize_thrift_msg(buf, &len, false, &file_scan_range);
+if (!st.ok()) {
+LOG(WARNING) << "fetch table schema failed, errmsg=" <<
st.get_error_msg();
+st.to_protobuf(result->mutable_status());
+return;
+}
+}
+
+if (file_scan_range.__isset.ranges == false) {
+st = Status::InternalError("can not get TFileRangeDesc.");
+st.to_protobuf(result->mutable_status());
+return;
+}
+if (file_scan_range.__isset.params == false) {
+st = Status::InternalError("can not get TFileScanRangeParams.");
+st.to_protobuf(result->mutable_status());
+return;
+}
+const TFileRangeDesc& range = file_scan_range.ranges.at(0);
+const TFileScanRangeParams& params = file_scan_range.params;
+// file_slots is no use
+std::vector file_slots;
+std::unique_ptr reader(nullptr);
+std::unique_ptr profile(new
RuntimeProfile("FetchTableSchema"));
+switch (params.format_type) {
+case TFileFormatType::FORMAT_CSV_PLAIN:
+case TFileFormatType::FORMAT_CSV_GZ:
+case TFileFormatType::FORMAT_CSV_BZ2:
+case TFileFormatType::FORMAT_CSV_LZ4FRAME:
+case TFileFormatType::FORMAT_CSV_LZOP:
+case TFileFormatType::FORMAT_CSV_DEFLATE: {
+reader.reset(new vectorized::CsvReader(profile.get(), params, range,
file_slots));
+break;
+}
+default:
+st = Status::InternalError("Not supported file format in fetch table
schema: {}",
+ params.format_type);
+st.to_protobuf(result->mutable_status());
+return;
+}
+std::unordered_map name_to_col_type;
+std::vector col_names;
+std::vector col_types;
+st = reader->get_parsered_schema(&col_names, &col_types);
+if (!st.ok()) {
+LOG(WARNING) << "fetch table schema failed, errmsg=" <<
st.get_error_msg();
+st.to_protobuf(result->mutable_status());
+return;
+}
+result->set_column_nums(col_names.size());
+for (size_t idx = 0; idx < col_names.size(); ++idx) {
+result->add_column_names(col_names[idx]);
+}
+for (size_t idx = 0; idx < col_types.size(); ++idx) {
+PTypeDesc* type_desc = result->add_column_types();
+col_types[idx].to_protobuf(type_desc);
+}
+LOG(INFO) << "complete parse, status: " << st;
Review Comment:
Remove this log, meaningless
##
be/src/service/internal_service.cpp:
##
@@ -407,6 +410,78 @@ void
PInternalServiceImpl::fetch_data(google::protobuf::RpcController* cntl_base
_exec_env->result_mgr()->fetch_data(request->finst_id(), ctx);
}
+void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController*
controller,
+ const PFetchTableSchemaRequest*
request,
+ PFetchTableSchemaResult* result,
+ google::protobuf::Closure* done)
{
+VLOG_RPC << "fetch table schema";
+brpc::ClosureGuard closure_guard(done);
+TFileScanRange file_scan_range;
+Status st = Status::OK();
+{
+const uint8_t* buf = (const
uint8_t*)(request->file_scan_range().data());
+uint32_t len = request->file_scan_range().size();
+st = deserialize_thrift_msg(buf, &len, false, &file_scan_range);
+if (!st.ok()) {
+LOG(WARNING) << "fetch table schema failed, errmsg=" <<
st.get_error_msg();
+st.to_protobuf(result->mutable_status());
+return;
+}
+}
+
+if (file_scan_range.__isset.ranges == false) {
+st = Status::InternalError("can not get TFileRangeDesc.");
+st.to_protobuf(result->mutable_status());
+return;
+
[GitHub] [doris] hello-stephen commented on pull request #13965: [Refactor](join) Refine codes for hashjoin and nljoin
hello-stephen commented on PR #13965: URL: https://github.com/apache/doris/pull/13965#issuecomment-1303077672 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.89 seconds load time: 455 seconds storage size: 17184096966 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104074201_clickbench_pr_39961.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013698522
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat
+extends PhysicalUnary implements Repeat {
+protected final List groupByExpressions;
+protected final List originGroupByExpressions;
+protected final List outputExpressions;
+protected final List groupingIdList;
+protected final List virtualGroupByExpressions;
Review Comment:
We should add some comments for these members. In particular,
`groupByExpressions` and `originGroupByExpressions` and are confusing.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat
+extends PhysicalUnary implements Repeat {
+protected final List groupByExpressions;
+protected final List originGroupByExpressions;
+protected final List outputExpressions;
+protected final List groupingIdList;
+protected final List virtualGroupByExpressions;
Review Comment:
We should add some comments for these members. In particular,
`groupByExpressions` and `originGroupByExpressions` are confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.
[GitHub] [doris] Yulei-Yang commented on a diff in pull request #13914: [feature](function)Support negative index for function split_part
Yulei-Yang commented on code in PR #13914:
URL: https://github.com/apache/doris/pull/13914#discussion_r1013668593
##
be/src/vec/functions/function_string.h:
##
@@ -1278,63 +1278,58 @@ class FunctionSplitPart : public IFunction {
auto str = str_col->get_data_at(i);
if (delimiter.size == 0) {
StringOP::push_empty_string(i, res_chars, res_offsets);
-} else if (delimiter.size == 1) {
-// If delimiter is a char, use memchr to split
-int32_t pre_offset = -1;
-int32_t offset = -1;
-int32_t num = 0;
-while (num < part_number) {
-pre_offset = offset;
-size_t n = str.size - offset - 1;
-const char* pos = reinterpret_cast(
-memchr(str.data + offset + 1, delimiter_str[0],
n));
-if (pos != nullptr) {
-offset = pos - str.data;
-num++;
-} else {
-offset = str.size;
-num = (num == 0) ? 0 : num + 1;
-break;
-}
-}
-
-if (num == part_number) {
-StringOP::push_value_string(
-std::string_view {
-reinterpret_cast(str.data +
pre_offset + 1),
-(size_t)offset - pre_offset - 1},
-i, res_chars, res_offsets);
-} else {
-StringOP::push_null_string(i, res_chars, res_offsets,
null_map_data);
-}
} else {
-// If delimiter is a string, use memmem to split
-int32_t pre_offset = -delimiter.size;
int32_t offset = -delimiter.size;
int32_t num = 0;
-while (num < part_number) {
-pre_offset = offset;
+std::vector find(str.size, -1); //store delimiter position
+while (num < str.size) {
size_t n = str.size - offset - delimiter.size;
-char* pos = reinterpret_cast(memmem(str.data +
offset + delimiter.size,
- n,
delimiter.data, delimiter.size));
-if (pos != nullptr) {
-offset = pos - str.data;
-num++;
+if (delimiter.size == 1) {
Review Comment:
Thank you very much for your review. While I have a question here. If I keep
origin logic, there will be two parts of code which looks similar and seems
duplicated, like line 1282-1284 and 1317-1333。Can you please give me an advice
which issue is more critical? @Gabriel39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #13959: [feature](table-valued-function) Support S3 tvf
BePPPower commented on code in PR #13959:
URL: https://github.com/apache/doris/pull/13959#discussion_r1013716581
##
be/src/service/internal_service.cpp:
##
@@ -407,6 +410,78 @@ void
PInternalServiceImpl::fetch_data(google::protobuf::RpcController* cntl_base
_exec_env->result_mgr()->fetch_data(request->finst_id(), ctx);
}
+void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController*
controller,
+ const PFetchTableSchemaRequest*
request,
+ PFetchTableSchemaResult* result,
+ google::protobuf::Closure* done)
{
+VLOG_RPC << "fetch table schema";
+brpc::ClosureGuard closure_guard(done);
+TFileScanRange file_scan_range;
+Status st = Status::OK();
+{
+const uint8_t* buf = (const
uint8_t*)(request->file_scan_range().data());
+uint32_t len = request->file_scan_range().size();
+st = deserialize_thrift_msg(buf, &len, false, &file_scan_range);
+if (!st.ok()) {
+LOG(WARNING) << "fetch table schema failed, errmsg=" <<
st.get_error_msg();
+st.to_protobuf(result->mutable_status());
+return;
+}
+}
+
+if (file_scan_range.__isset.ranges == false) {
+st = Status::InternalError("can not get TFileRangeDesc.");
+st.to_protobuf(result->mutable_status());
+return;
+}
+if (file_scan_range.__isset.params == false) {
+st = Status::InternalError("can not get TFileScanRangeParams.");
+st.to_protobuf(result->mutable_status());
+return;
+}
+const TFileRangeDesc& range = file_scan_range.ranges.at(0);
+const TFileScanRangeParams& params = file_scan_range.params;
+// file_slots is no use
+std::vector file_slots;
+std::unique_ptr reader(nullptr);
+std::unique_ptr profile(new
RuntimeProfile("FetchTableSchema"));
+switch (params.format_type) {
+case TFileFormatType::FORMAT_CSV_PLAIN:
+case TFileFormatType::FORMAT_CSV_GZ:
+case TFileFormatType::FORMAT_CSV_BZ2:
+case TFileFormatType::FORMAT_CSV_LZ4FRAME:
+case TFileFormatType::FORMAT_CSV_LZOP:
+case TFileFormatType::FORMAT_CSV_DEFLATE: {
Review Comment:
Parquet and ORC will be added after implementing `get_parsered_schema`
method of ParquetReader and OrcReader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #13969: [feature-wip](dlf) prepare to support aliyun dlf
morningman opened a new pull request, #13969: URL: https://github.com/apache/doris/pull/13969 # Proposed changes Issue Number: close #xxx ## Problem summary [What is DLF](https://www.alibabacloud.com/product/datalake-formation) This PR is a preparation for support DLF, with some changes of multi catalog 1. Add RuntimeException for most of hive meta store or es client visit operation. 2. Add DLF related dependencies. 3. Move the checks of es catalog properties to the analysis phase of creating es catalog TODO(in next PR): 1. Refactor the `getSplit` method to support not only hdfs, but s3-compatible object storage. 2. Finish the implementation of supporting DLF ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 5. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 6. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 7. Does it need to update dependencies: - [ ] Yes - [ ] No 8. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13960: [fix](memtracker) Fix DCHECK(!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker))
yiguolei commented on code in PR #13960:
URL: https://github.com/apache/doris/pull/13960#discussion_r1013731494
##
be/src/runtime/load_channel_mgr.h:
##
@@ -85,6 +88,7 @@ class LoadChannelMgr {
std::mutex _lock;
// load id -> load channel
std::unordered_map> _load_channels;
+SpinLock _load_channels_lock;
std::shared_ptr _reduce_memory_channel = nullptr;
Review Comment:
This lock's scope is same as _lock, I think we could just use _lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13942: [chore](be web ui)upgrade jquery version to 3.6.0
yiguolei merged PR #13942: URL: https://github.com/apache/doris/pull/13942 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (06a1efdb01 -> a19e6881c7)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 06a1efdb01 [fix](Nerieds) fix tpch and support trace plan's change event (#13957) add a19e6881c7 [chore](be web ui)upgrade jquery version to 3.6.0 (#13942) No new revisions were added by this update. Summary of changes: be/src/http/web_page_handler.cpp | 2 +- dist/LICENSE-dist.txt| 5 +++-- thirdparty/build-thirdparty.sh | 4 ++-- thirdparty/vars.sh | 6 +++--- 4 files changed, 9 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13970: [Enhancement](function)use extern template to date_time_add
BiteThet opened a new pull request, #13970: URL: https://github.com/apache/doris/pull/13970 # Proposed changes extern template may reduce the build time. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
yiguolei commented on code in PR #13865:
URL: https://github.com/apache/doris/pull/13865#discussion_r1013746447
##
be/src/io/cache/dummy_file_cache.h:
##
@@ -36,7 +36,7 @@ class DummyFileCache final : public FileCache {
Status close() override { return Status::OK(); }
-Status read_at(size_t offset, Slice result, size_t* bytes_read) override {
+Status read_at(size_t offset, Slice result, size_t* bytes_read, const
IOContext* ctx) override {
Review Comment:
use reference not pointer.
Pointer is used as output parameter.
and please add ctx before bytes_read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #13971: [minor](libjvm) allow users to specific java home by configurations
Gabriel39 opened a new pull request, #13971: URL: https://github.com/apache/doris/pull/13971 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13086: [feature](information_schema) add `backends` information_schema table
yiguolei commented on code in PR #13086:
URL: https://github.com/apache/doris/pull/13086#discussion_r1013774521
##
gensrc/thrift/Descriptors.thrift:
##
@@ -102,8 +102,9 @@ enum TSchemaTableType {
SCH_USER_PRIVILEGES,
SCH_VARIABLES,
SCH_VIEWS,
-SCH_INVALID,
-SCH_ROWSETS
+SCH_ROWSETS,
+SCH_BACKENDS,
+SCH_INVALID
Review Comment:
do not change the order of enums.
it will core during rolliing upgrade be because be depend on order or id to
check enum value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13966: [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts
hello-stephen commented on PR #13966: URL: https://github.com/apache/doris/pull/13966#issuecomment-1303170171 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.02 seconds load time: 447 seconds storage size: 17179192833 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104093037_clickbench_pr_40137.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #13971: [minor](libjvm) allow users to specific java home by configurations
morningman commented on PR #13971: URL: https://github.com/apache/doris/pull/13971#issuecomment-1303172495 No need, we can just write `JAVA_HOME=xxx` in be.conf, it will be export as a env var. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #13971: [minor](libjvm) allow users to specific java home by configurations
morningman closed pull request #13971: [minor](libjvm) allow users to specific java home by configurations URL: https://github.com/apache/doris/pull/13971 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree opened a new pull request, #13972: [test](java-udf)add java udf RegressionTest about the currently supported data types
LemonLiTree opened a new pull request, #13972: URL: https://github.com/apache/doris/pull/13972 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #13959: [feature](table-valued-function) Support S3 tvf
morningman commented on PR #13959: URL: https://github.com/apache/doris/pull/13959#issuecomment-1303174238 Need to add some regression test in next PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13967: [fix](load) fix a bug that reduce memory work on hard limit might be …
hello-stephen commented on PR #13967: URL: https://github.com/apache/doris/pull/13967#issuecomment-1303188711 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.88 seconds load time: 451 seconds storage size: 17178690546 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104094941_clickbench_pr_39986.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] k-i-d-d commented on a diff in pull request #13960: [fix](memtracker) Fix DCHECK(!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker))
k-i-d-d commented on code in PR #13960:
URL: https://github.com/apache/doris/pull/13960#discussion_r1013815450
##
be/src/runtime/load_channel_mgr.h:
##
@@ -85,6 +88,7 @@ class LoadChannelMgr {
std::mutex _lock;
// load id -> load channel
std::unordered_map> _load_channels;
+SpinLock _load_channels_lock;
std::shared_ptr _reduce_memory_channel = nullptr;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13960: [fix](memtracker) Fix DCHECK(!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker))
xinyiZzz commented on code in PR #13960:
URL: https://github.com/apache/doris/pull/13960#discussion_r1013818390
##
be/src/runtime/load_channel_mgr.h:
##
@@ -85,6 +88,7 @@ class LoadChannelMgr {
std::mutex _lock;
// load id -> load channel
std::unordered_map> _load_channels;
+SpinLock _load_channels_lock;
std::shared_ptr _reduce_memory_channel = nullptr;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #13966: [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts
morningman commented on PR #13966: URL: https://github.com/apache/doris/pull/13966#issuecomment-1303193284 Please add some comment for this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg opened a new pull request, #13973: [Enhancement](function) add to_bitmap() function with int type
yangzhg opened a new pull request, #13973: URL: https://github.com/apache/doris/pull/13973 # Proposed changes Issue Number: close #xxx ## Problem summary to_bitmap function only support string param only，add to_bitmap() function with int type, this can avoid convert int type to string and then convert string to int ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13086: [feature](information_schema) add `backends` information_schema table
yiguolei commented on code in PR #13086:
URL: https://github.com/apache/doris/pull/13086#discussion_r1013843794
##
be/src/exec/schema_scan_node.cpp:
##
@@ -127,7 +132,8 @@ Status SchemaScanNode::prepare(RuntimeState* state) {
return Status::InternalError("schema scanner get nullptr pointer.");
}
-RETURN_IF_ERROR(_schema_scanner->init(&_scanner_param, _pool));
+RETURN_IF_ERROR(
+_schema_scanner->init(&_scanner_param, _pool,
schema_table->schema_table_type()));
Review Comment:
you could add a member schem table type in SchemaScanner so that init
function could use the member instead of pass in a parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13086: [feature](information_schema) add `backends` information_schema table
yiguolei commented on code in PR #13086:
URL: https://github.com/apache/doris/pull/13086#discussion_r1013844676
##
be/src/vec/exec/vschema_scan_node.cpp:
##
@@ -165,7 +170,8 @@ Status VSchemaScanNode::prepare(RuntimeState* state) {
return Status::InternalError("schema scanner get nullptr pointer.");
}
-RETURN_IF_ERROR(_schema_scanner->init(&_scanner_param, _pool));
+RETURN_IF_ERROR(
+_schema_scanner->init(&_scanner_param, _pool,
schema_table->schema_table_type()));
Review Comment:
same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13086: [feature](information_schema) add `backends` information_schema table
yiguolei commented on code in PR #13086:
URL: https://github.com/apache/doris/pull/13086#discussion_r1013845285
##
be/test/vec/function/function_comparison_test.cpp:
##
@@ -35,7 +35,7 @@ TEST(ComparisonTest, ComparisonFunctionTest) {
SchemaScanner schema_scanner(column_descs, 3);
ObjectPool object_pool;
SchemaScannerParam param;
-schema_scanner.init(¶m, &object_pool);
+schema_scanner.init(¶m, &object_pool, TSchemaTableType::SCH_INVALID);
Review Comment:
these case not need change if you add schema type as member in SchemaScanner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] b19mud opened a new pull request, #151: [Doc] fix spelling errors
b19mud opened a new pull request, #151: URL: https://github.com/apache/doris-website/pull/151 Fix 'expamle' to 'example' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] b19mud opened a new pull request, #13974: [DOC] fixed spelling errors
b19mud opened a new pull request, #13974: URL: https://github.com/apache/doris/pull/13974 # Proposed changes Issue Number: close #xxx ## Problem summary Fix 'expamle' to 'example' ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [x] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13959: [feature](table-valued-function) Support S3 tvf
hello-stephen commented on PR #13959: URL: https://github.com/apache/doris/pull/13959#issuecomment-1303236731 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.44 seconds load time: 432 seconds storage size: 17154655363 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104103500_clickbench_pr_40058.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web opened a new pull request, #13975: [doc](routineload)Common mistakes in adding routine load
caoliang-web opened a new pull request, #13975: URL: https://github.com/apache/doris/pull/13975 # Proposed changes Issue Number: close #xxx ## Problem summary Common mistakes in adding routine load ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #12620: [refactor](cv)wait on condition variable more gently
hello-stephen commented on PR #12620: URL: https://github.com/apache/doris/pull/12620#issuecomment-1303298545 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.07 seconds load time: 446 seconds storage size: 17179983300 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104113358_clickbench_pr_40025.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
yiguolei commented on code in PR #13865:
URL: https://github.com/apache/doris/pull/13865#discussion_r1013911826
##
be/src/olap/reader.h:
##
@@ -101,6 +101,8 @@ class TabletReader {
bool read_orderby_key_reverse = false;
// num of columns for orderby key
size_t read_orderby_key_num_prefix_columns = 0;
+// use local cache when reading remote file.
+bool use_local_file_cache = false;
Review Comment:
This line is useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
yiguolei commented on code in PR #13865:
URL: https://github.com/apache/doris/pull/13865#discussion_r1013912893
##
be/src/olap/rowset/beta_rowset_reader.cpp:
##
@@ -149,6 +149,7 @@ Status BetaRowsetReader::init(RowsetReaderContext*
read_context) {
read_options.record_rowids = read_context->record_rowids;
read_options.read_orderby_key_reverse =
read_context->read_orderby_key_reverse;
read_options.read_orderby_key_columns =
read_context->read_orderby_key_columns;
+read_options.io_ctx.use_local_file_cache =
read_context->use_local_file_cache;
Review Comment:
There is reader type , I think you'd better pass reader type to iO CTX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
yiguolei commented on code in PR #13865:
URL: https://github.com/apache/doris/pull/13865#discussion_r1013914049
##
be/src/olap/rowset/segment_v2/segment.h:
##
@@ -74,7 +74,8 @@ class Segment : public std::enable_shared_from_this {
uint32_t num_rows() const { return _footer.num_rows(); }
-Status new_column_iterator(const TabletColumn& tablet_column,
ColumnIterator** iter);
+Status new_column_iterator(const TabletColumn& tablet_column, const
IOContext& io_ctx,
Review Comment:
Do not modify this method. could modify column iterator's init method and
pass io ctx via init method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #13865: [feature](remote)Only query can use local cache when reading remote files.
yiguolei commented on code in PR #13865: URL: https://github.com/apache/doris/pull/13865#discussion_r1013914789 ## be/src/olap/schema_change.cpp: ## @@ -1942,6 +1942,7 @@ Status SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2& reader_context.is_vec = config::enable_vectorized_alter_table; reader_context.delete_bitmap = &base_tablet->tablet_meta()->delete_bitmap(); reader_context.version = Version(0, end_version); +reader_context.use_local_file_cache = false; Review Comment: I think reader type is better than local file cache because it contains more info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li commented on a diff in pull request #13969: [feature-wip](dlf) prepare to support aliyun dlf
Jibing-Li commented on code in PR #13969:
URL: https://github.com/apache/doris/pull/13969#discussion_r1013905264
##
fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java:
##
@@ -135,28 +140,46 @@ public static String getFormat(String input) throws
DdlException {
}
}
-public static HiveMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
+public static IMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
HiveConf hiveConf = new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, metaStoreUris);
-HiveMetaStoreClient hivemetastoreclient = null;
+return getClient(hiveConf);
+}
+
+public static IMetaStoreClient getClient(HiveConf hiveConf) throws
DdlException {
+hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
+String.valueOf(Config.hive_metastore_client_timeout_second));
+Properties prop = hiveConf.getAllProperties();
+for (String key : prop.stringPropertyNames()) {
+LOG.info("cmy debug key: {}, value: {}", key,
prop.getProperty(key));
+}
+IMetaStoreClient metaStoreClient = null;
+String type = hiveConf.get(HIVE_METASTORE_TYPE);
+LOG.info("cmy debug type: {}", type);
Review Comment:
LOG.debug?
##
fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java:
##
@@ -135,28 +140,46 @@ public static String getFormat(String input) throws
DdlException {
}
}
-public static HiveMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
+public static IMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
HiveConf hiveConf = new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, metaStoreUris);
-HiveMetaStoreClient hivemetastoreclient = null;
+return getClient(hiveConf);
+}
+
+public static IMetaStoreClient getClient(HiveConf hiveConf) throws
DdlException {
+hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
+String.valueOf(Config.hive_metastore_client_timeout_second));
+Properties prop = hiveConf.getAllProperties();
+for (String key : prop.stringPropertyNames()) {
+LOG.info("cmy debug key: {}, value: {}", key,
prop.getProperty(key));
Review Comment:
Set to LOG.debug?
##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java:
##
@@ -74,61 +76,45 @@ public class EsExternalCatalog extends ExternalCatalog {
/**
* Default constructor for EsExternalCatalog.
*/
-public EsExternalCatalog(long catalogId, String name, Map
props) throws DdlException {
+public EsExternalCatalog(long catalogId, String name, Map
props) {
this.id = catalogId;
this.name = name;
this.type = "es";
-validate(props);
+setProperties(props);
this.catalogProperty = new CatalogProperty();
this.catalogProperty.setProperties(props);
}
-private void validate(Map properties) throws DdlException {
-if (properties == null) {
-throw new DdlException(
-"Please set properties of elasticsearch table, " + "they
are: hosts, user, password, index");
-}
-
-if (StringUtils.isBlank(properties.get(PROP_HOSTS))) {
-throw new DdlException("Hosts of ES table is null.");
-}
-nodes = properties.get(PROP_HOSTS).trim().split(",");
-// check protocol
-for (String seed : nodes) {
-if (!seed.startsWith("http")) {
-throw new DdlException("the protocol must be used");
-}
+private void setProperties(Map properties) {
Review Comment:
I think we need to call `setProperties` in `gsonPostProcess` as well to make
sure all properties are set after metadata replay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #13976: [feature](Nereids) support statement having aggregate function in order by list
morrySnow opened a new pull request, #13976: URL: https://github.com/apache/doris/pull/13976 # Proposed changes 1. add a feature that support statement having aggregate function in order by list. such as ```sql SELECT COUNT(*) FROM t GROUP BY c1 ORDER BY COUNT(*) DESC; ``` 2. add clickbench analyze unit tests ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly closed pull request #13873: [feature](nereids) runtime filter prune
englefly closed pull request #13873: [feature](nereids) runtime filter prune URL: https://github.com/apache/doris/pull/13873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13940: [enhancement](test) support tablet repair and balance process in ut
github-actions[bot] commented on PR #13940: URL: https://github.com/apache/doris/pull/13940#issuecomment-1303352887 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13940: [enhancement](test) support tablet repair and balance process in ut
github-actions[bot] commented on PR #13940: URL: https://github.com/apache/doris/pull/13940#issuecomment-1303353033 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lide-reed opened a new issue, #13977: [Bug] BE crash when execute an inserting into select statement.
lide-reed opened a new issue, #13977: URL: https://github.com/apache/doris/issues/13977 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.3-rc02 ### What's Wrong? one BE crash as following stack: WARNING: Logging before InitGoogleLogging() is written to STDERR I1104 20:18:24.117200 672798 env.cpp:46] Env init successfully. *** Check failure stack trace: *** @ 0x56059e0f7a4d google::LogMessage::Fail() @ 0x56059e0f9f89 google::LogMessage::SendToLog() @ 0x56059e0f75b6 google::LogMessage::Flush() @ 0x56059e0fa5f9 google::LogMessageFatal::~LogMessageFatal() @ 0x56059ca08116 doris::vectorized::ColumnNullable::ColumnNullable() @ 0x56059ca0b4d1 doris::vectorized::ColumnNullable::create() @ 0x56059dadf39a doris::vectorized::FunctionIf::execute_for_null_then_else() @ 0x56059dadfd2c doris::vectorized::FunctionIf::execute_impl() @ 0x56059d5f9d4d doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns() @ 0x56059d5fb2c9 doris::vectorized::PreparedFunctionImpl::execute() @ 0x56059d0e2504 doris::vectorized::IFunctionBase::execute() @ 0x56059dbb452d doris::vectorized::VectorizedFnCall::execute() @ 0x56059cac50a7 doris::vectorized::VExprContext::execute() @ 0x56059ceb0595 doris::vectorized::HashJoinNode::_build_output_block() @ 0x56059ceb154e doris::vectorized::HashJoinNode::get_next() @ 0x56059ceb179e doris::vectorized::HashJoinNode::get_next() @ 0x56059ceb179e doris::vectorized::HashJoinNode::get_next() @ 0x56059ceb179e doris::vectorized::HashJoinNode::get_next() @ 0x56059c4be66a doris::PlanFragmentExecutor::get_vectorized_internal() @ 0x56059c4c2c2d doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x56059c4c46df doris::PlanFragmentExecutor::open() @ 0x56059c43233f doris::FragmentExecState::execute() @ 0x56059c436eda doris::FragmentMgr::_exec_actual() @ 0x56059c4401cc std::_Function_handler<>::_M_invoke() @ 0x56059c5fbd1b doris::ThreadPool::dispatch_thread() @ 0x56059c5f531f doris::Thread::supervise_thread() @ 0x7fae62fbfea5 start_thread @ 0x7fae632d29fd __clone @ (nil) (unknown) *** Aborted at 1667564733 (unix time) try "date -d @1667564733" if you are using GNU date *** *** SIGABRT unkown detail explain (@0xa441e) received by PID 672798 (TID 0x7fad217fa700) from PID 672798; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /data/hc2/be/src/common/signal_handler.h:420 1# 0x7FAE6320A400 in /lib64/libc.so.6 2# __GI_raise in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x56059BCE06F2 in /usr/local/service/doris/lib/be/doris_be 5# 0x56059E0F7A4D in /usr/local/service/doris/lib/be/doris_be 6# google::LogMessage::SendToLog() in /usr/local/service/doris/lib/be/doris_be 7# google::LogMessage::Flush() in /usr/local/service/doris/lib/be/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /usr/local/service/doris/lib/be/doris_be 9# doris::vectorized::ColumnNullable::ColumnNullable(COW::mutable_ptr&&, COW::mutable_ptr&&) at /data/hc2/be/src/vec/columns/column_nullable.cpp:44 10# doris::vectorized::ColumnNullable::create(COW::immutable_ptr const&, COW::immutable_ptr const&) at /data/hc2/be/src/vec/columns/column_nullable.h:56 11# doris::vectorized::FunctionIf::execute_for_null_then_else(doris_udf::FunctionContext*, doris::vectorized::Block&, doris::vectorized::ColumnWithTypeAndName const&, doris::vectorized::ColumnWithTypeAndName const&, doris::vectorized::ColumnWithTypeAndName const&, unsigned long, unsigned long, doris::Status&) at /data/hc2/be/src/vec/functions/if.cpp:281 12# doris::vectorized::FunctionIf::execute_impl(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long) at /data/hc2/be/src/vec/functions/if.cpp:447 13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /data/hc2/be/src/vec/functions/function.cpp:251 14# doris::vectorized::PreparedFunctionImpl::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /data/hc2/be/src/vec/functions/function.cpp:274 15# doris::vectorized::IFunctionBase::execute(doris_udf::FunctionContext*, doris::vectorized::Block&, std::vector > const&, unsigned long, unsigned long, bool) at /data/hc2/
[GitHub] [doris] hello-stephen commented on pull request #13969: [feature-wip](dlf) prepare to support aliyun dlf
hello-stephen commented on PR #13969: URL: https://github.com/apache/doris/pull/13969#issuecomment-1303432698 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.73 seconds load time: 447 seconds storage size: 17178443022 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104123020_clickbench_pr_40075.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on pull request #13973: [Enhancement](function) add to_bitmap() function with int type
HappenLee commented on PR #13973: URL: https://github.com/apache/doris/pull/13973#issuecomment-1303466982 Maybe we should only keep `to_bitmap(bigint)` del the `to_bitmap(string)` after next version ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 opened a new pull request, #13978: [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null
zhangstar333 opened a new pull request, #13978: URL: https://github.com/apache/doris/pull/13978 # Proposed changes Issue Number: close #13968 #13964 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on issue #13968: [Bug] sub_bitmap/bitmap_subset_limit return null when the bitmap count is 1
zhangstar333 commented on issue #13968: URL: https://github.com/apache/doris/issues/13968#issuecomment-1303474497 fixed:https://github.com/apache/doris/pull/13978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on issue #13964: [Bug] bitmap_subset_in_range function when there is only one data in the bitmap, the function get the sub-bitmap is null, not itself.
zhangstar333 commented on issue #13964: URL: https://github.com/apache/doris/issues/13964#issuecomment-1303475099 fixed:https://github.com/apache/doris/pull/13978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13970: [Enhancement](function)use extern template to date_time_add
hello-stephen commented on PR #13970: URL: https://github.com/apache/doris/pull/13970#issuecomment-1303485465 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.38 seconds load time: 445 seconds storage size: 17178983521 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104125405_clickbench_pr_40110.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13972: [test](java-udf)add java udf RegressionTest about the currently supported data types
hello-stephen commented on PR #13972: URL: https://github.com/apache/doris/pull/13972#issuecomment-1303559762 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.61 seconds load time: 446 seconds storage size: 17178779139 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104134532_clickbench_pr_40165.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #13979: [fix](join)ColumnNullable need handle const column with nullable const value (#13866)
starocean999 opened a new pull request, #13979: URL: https://github.com/apache/doris/pull/13979 # Proposed changes Issue Number: close (https://github.com/apache/doris/issues/13977) ## Problem summary cherry-pick from https://github.com/apache/doris/pull/13866 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13973: [Enhancement](function) add to_bitmap() function with int type
hello-stephen commented on PR #13973: URL: https://github.com/apache/doris/pull/13973#issuecomment-1303654361 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.84 seconds load time: 452 seconds storage size: 17179876864 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104143401_clickbench_pr_40188.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13961: [fix](nereids) cannot collect decimal column stats
github-actions[bot] commented on PR #13961: URL: https://github.com/apache/doris/pull/13961#issuecomment-1303718713 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13962: [Improvement](javaudf) improve java class loader usage
github-actions[bot] commented on PR #13962: URL: https://github.com/apache/doris/pull/13962#issuecomment-1303741894 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13962: [Improvement](javaudf) improve java class loader usage
github-actions[bot] commented on PR #13962: URL: https://github.com/apache/doris/pull/13962#issuecomment-1303741989 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt opened a new pull request, #13980: [feature](compaction) support vertical compaction
yixiutt opened a new pull request, #13980: URL: https://github.com/apache/doris/pull/13980 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13763: [feature](function)add search functions: multi_search_all_positions & multi_match_any
github-actions[bot] commented on PR #13763: URL: https://github.com/apache/doris/pull/13763#issuecomment-1303790007 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13763: [feature](function)add search functions: multi_search_all_positions & multi_match_any
github-actions[bot] commented on PR #13763: URL: https://github.com/apache/doris/pull/13763#issuecomment-1303790096 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BePPPower commented on a diff in pull request #13086: [feature](information_schema) add `backends` information_schema table
BePPPower commented on code in PR #13086:
URL: https://github.com/apache/doris/pull/13086#discussion_r1014204355
##
gensrc/thrift/Descriptors.thrift:
##
@@ -102,8 +102,9 @@ enum TSchemaTableType {
SCH_USER_PRIVILEGES,
SCH_VARIABLES,
SCH_VIEWS,
-SCH_INVALID,
-SCH_ROWSETS
+SCH_ROWSETS,
+SCH_BACKENDS,
+SCH_INVALID
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13960: [fix](load) Fix load channel mgr lock
github-actions[bot] commented on PR #13960: URL: https://github.com/apache/doris/pull/13960#issuecomment-1303799012 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13966: [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts
github-actions[bot] commented on PR #13966: URL: https://github.com/apache/doris/pull/13966#issuecomment-1303800656 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13966: [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts
github-actions[bot] commented on PR #13966: URL: https://github.com/apache/doris/pull/13966#issuecomment-1303800714 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13976: [feature](Nereids) support statement having aggregate function in order by list
hello-stephen commented on PR #13976: URL: https://github.com/apache/doris/pull/13976#issuecomment-1303835144 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.31 seconds load time: 459 seconds storage size: 17185163279 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104162424_clickbench_pr_40287.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](load) Fix load channel mgr lock (#13960)
This is an automated email from the ASF dual-hosted git repository.
zouxinyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f87be09d69 [fix](load) Fix load channel mgr lock (#13960)
f87be09d69 is described below
commit f87be09d694b1947ccf615efa4fc4d2b12eddb58
Author: Xinyi Zou
AuthorDate: Sat Nov 5 00:48:30 2022 +0800
[fix](load) Fix load channel mgr lock (#13960)
hot fix load channel mgr lock
---
be/src/olap/delta_writer.cpp | 6 +-
be/src/runtime/load_channel_mgr.h| 1 +
be/src/vec/exec/join/vhash_join_node.cpp | 2 +-
3 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index a748f566c1..5ec23ea593 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -404,7 +404,11 @@ Status DeltaWriter::cancel() {
void DeltaWriter::save_mem_consumption_snapshot() {
std::lock_guard l(_lock);
_mem_consumption_snapshot = mem_consumption();
-_memtable_consumption_snapshot = _mem_table->memory_usage();
+if (_mem_table == nullptr) {
+_memtable_consumption_snapshot = 0;
+} else {
+_memtable_consumption_snapshot = _mem_table->memory_usage();
+}
}
int64_t DeltaWriter::get_memtable_consumption_inflush() const {
diff --git a/be/src/runtime/load_channel_mgr.h
b/be/src/runtime/load_channel_mgr.h
index 4fd113cc77..b9182d8a00 100644
--- a/be/src/runtime/load_channel_mgr.h
+++ b/be/src/runtime/load_channel_mgr.h
@@ -61,6 +61,7 @@ public:
void refresh_mem_tracker() {
int64_t mem_usage = 0;
+std::lock_guard l(_lock);
for (auto& kv : _load_channels) {
mem_usage += kv.second->mem_consumption();
}
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp
b/be/src/vec/exec/join/vhash_join_node.cpp
index a478df8744..0b326db114 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -999,7 +999,6 @@ Status HashJoinNode::prepare(RuntimeState* state) {
"");
_mem_tracker = std::make_unique("ExecNode:" +
_runtime_profile->name(),
_runtime_profile.get());
-SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
if (_vconjunct_ctx_ptr) {
RETURN_IF_ERROR((*_vconjunct_ctx_ptr)->prepare(state,
_intermediate_row_desc));
@@ -1008,6 +1007,7 @@ Status HashJoinNode::prepare(RuntimeState* state) {
for (int i = 0; i < _children.size(); ++i) {
RETURN_IF_ERROR(_children[i]->prepare(state));
}
+SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
// Build phase
auto build_phase_profile = runtime_profile()->create_child("BuildPhase",
true, true);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz merged pull request #13960: [fix](load) Fix load channel mgr lock
xinyiZzz merged PR #13960: URL: https://github.com/apache/doris/pull/13960 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13978: [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null
hello-stephen commented on PR #13978: URL: https://github.com/apache/doris/pull/13978#issuecomment-1303894962 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.99 seconds load time: 453 seconds storage size: 17183095979 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104170849_clickbench_pr_40322.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13785: [Feature](Sequence) Support sequence_match and sequence_count functions
hello-stephen commented on PR #13785: URL: https://github.com/apache/doris/pull/13785#issuecomment-1303944569 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.67 seconds load time: 456 seconds storage size: 17177871073 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104174701_clickbench_pr_40362.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13933: [test](delete) Add more deletion test cases
hello-stephen commented on PR #13933: URL: https://github.com/apache/doris/pull/13933#issuecomment-1303962027 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.37 seconds load time: 453 seconds storage size: 17180614520 Bytes https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221104180604_clickbench_pr_40375.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] vkingnew opened a new issue, #13981: WIDTH_BUCKET function support
vkingnew opened a new issue, #13981: URL: https://github.com/apache/doris/issues/13981 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description > function **width_bucket** had supported by oracle and postgresql,hive,sparksql,impala,snowflake and databricks. please add the same name function and have the same feature. please refer to: https://www.postgresql.org/docs/15/functions-math.html https://docs.snowflake.com/en/sql-reference/functions/width_bucket.html ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #9350: remove explicit memtracker from scannode
github-actions[bot] commented on PR #9350: URL: https://github.com/apache/doris/pull/9350#issuecomment-1304350905 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #13966: [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts
morningman merged PR #13966: URL: https://github.com/apache/doris/pull/13966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f87be09d69 -> 04830af039)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f87be09d69 [fix](load) Fix load channel mgr lock (#13960) add 04830af039 [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts (#13966) No new revisions were added by this update. Summary of changes: be/src/exec/tablet_sink.cpp| 46 - be/src/exec/tablet_sink.h | 4 +- be/src/vec/sink/vtablet_sink.cpp | 172 --- be/src/vec/sink/vtablet_sink.h | 4 + be/test/vec/exec/vtablet_sink_test.cpp | 298 +++-- 5 files changed, 330 insertions(+), 194 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-unstable updated (d28010428c -> 04830af039)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.2-unstable in repository https://gitbox.apache.org/repos/asf/doris.git discard d28010428c [fix](memtracker) Fix DCHECK(!std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker)) #13960 add 698541e58d [improvement](exec) add more debug info on fragment exec error (#13899) add f2d84d81e6 [feature-wip][refactor](multi-catalog) Persist external catalog related metadata. (#13746) add e09033276e [fix](runtime-filter) build thread destruct first may cause probe thread coredump (#13911) add efb2596c7a [enhancment](Nereids) enable push down filter through aggregation (#13938) add 9bf20a7b5d [enhancement](Nereids) remove unnecessary int cast (#13881) add dc01fb4085 [enhancement](Nereids) remove unnecessary string cast (#13730) add 948e080b31 [minor](error msg) Fix wrong error message (#13950) add 554f566217 [enhancement](compaction) introduce segment compaction (#12609) (#12866) add 06a1efdb01 [fix](Nerieds) fix tpch and support trace plan's change event (#13957) add a19e6881c7 [chore](be web ui)upgrade jquery version to 3.6.0 (#13942) add f87be09d69 [fix](load) Fix load channel mgr lock (#13960) add 04830af039 [fix](tablet sink) fallback to non-vectorized interface in tablet_sink if is in progress of upgrding from 1.1-lts to 1.2-lts (#13966) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d28010428c) \ N -- N -- N refs/heads/branch-1.2-unstable (04830af039) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: be/src/common/config.h | 11 + be/src/common/status.h | 6 +- be/src/exec/hash_join_node.cpp | 42 +- be/src/exec/tablet_sink.cpp| 46 +- be/src/exec/tablet_sink.h | 4 +- be/src/http/web_page_handler.cpp | 2 +- be/src/olap/compaction.cpp | 2 +- be/src/olap/delta_writer.cpp | 44 +- be/src/olap/delta_writer.h | 3 - be/src/olap/olap_server.cpp| 21 + be/src/olap/rowset/beta_rowset.cpp | 13 + be/src/olap/rowset/beta_rowset.h | 7 + be/src/olap/rowset/beta_rowset_writer.cpp | 654 +++-- be/src/olap/rowset/beta_rowset_writer.h| 72 ++- be/src/olap/rowset/segment_v2/segment_writer.h | 7 +- be/src/olap/schema_change.h| 1 + be/src/olap/storage_engine.cpp | 11 + be/src/olap/storage_engine.h | 15 + be/src/runtime/fragment_mgr.cpp| 7 +- be/src/runtime/load_channel_mgr.h | 1 + be/src/service/doris_main.cpp | 1 - be/src/util/doris_metrics.h| 1 + be/src/util/libjvm_loader.cpp | 4 +- be/src/vec/olap/vgeneric_iterators.cpp | 514 +--- be/src/vec/olap/vgeneric_iterators.h | 261 be/src/vec/sink/vtablet_sink.cpp | 172 -- be/src/vec/sink/vtablet_sink.h | 4 + be/test/CMakeLists.txt | 1 + be/test/olap/segcompaction_test.cpp| 454 ++ be/test/vec/exec/vtablet_sink_test.cpp | 298 ++ dist/LICENSE-dist.txt | 5 +- docs/en/docs/admin-manual/config/be-config.md | 18 + .../maint-monitor/be-olap-error-code.md| 6 +- docs/zh-CN/docs/admin-manual/config/be-config.md | 18 + .../maint-monitor/be-olap-error-code.md| 8 +- .../apache/doris/analysis/FunctionCallExpr.java| 5 +- .../main/java/org/apache/doris/catalog/Column.java | 5 +- .../main/java/org/apache/doris/catalog/Env.java| 2 +- .../org/apache/doris/catalog/RefreshManager.java | 10 + .../doris/catalog/external/EsExternalDatabase.java | 79 ++- .../doris/catalog/external/EsExternalTable.java| 79 ++- .../doris/catalog/external/ExternalDatabase.java | 63 +- .../doris/catalog/external/ExternalTable.java | 81 ++- .../catalog/external/HMSExternalDatabase.java | 81 ++- .../doris/catalo
[GitHub] [doris] github-actions[bot] commented on pull request #13974: [typo](doc) fixed spelling errors
github-actions[bot] commented on PR #13974: URL: https://github.com/apache/doris/pull/13974#issuecomment-1304379788 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13974: [typo](doc) fixed spelling errors
github-actions[bot] commented on PR #13974: URL: https://github.com/apache/doris/pull/13974#issuecomment-1304379794 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13969: [feature-wip](dlf) prepare to support aliyun dlf
morningman commented on code in PR #13969:
URL: https://github.com/apache/doris/pull/13969#discussion_r1014550596
##
fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java:
##
@@ -135,28 +140,46 @@ public static String getFormat(String input) throws
DdlException {
}
}
-public static HiveMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
+public static IMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
HiveConf hiveConf = new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, metaStoreUris);
-HiveMetaStoreClient hivemetastoreclient = null;
+return getClient(hiveConf);
+}
+
+public static IMetaStoreClient getClient(HiveConf hiveConf) throws
DdlException {
+hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
+String.valueOf(Config.hive_metastore_client_timeout_second));
+Properties prop = hiveConf.getAllProperties();
+for (String key : prop.stringPropertyNames()) {
+LOG.info("cmy debug key: {}, value: {}", key,
prop.getProperty(key));
Review Comment:
Removed
##
fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java:
##
@@ -135,28 +140,46 @@ public static String getFormat(String input) throws
DdlException {
}
}
-public static HiveMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
+public static IMetaStoreClient getClient(String metaStoreUris) throws
DdlException {
HiveConf hiveConf = new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, metaStoreUris);
-HiveMetaStoreClient hivemetastoreclient = null;
+return getClient(hiveConf);
+}
+
+public static IMetaStoreClient getClient(HiveConf hiveConf) throws
DdlException {
+hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
+String.valueOf(Config.hive_metastore_client_timeout_second));
+Properties prop = hiveConf.getAllProperties();
+for (String key : prop.stringPropertyNames()) {
+LOG.info("cmy debug key: {}, value: {}", key,
prop.getProperty(key));
+}
+IMetaStoreClient metaStoreClient = null;
+String type = hiveConf.get(HIVE_METASTORE_TYPE);
+LOG.info("cmy debug type: {}", type);
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on pull request #13979: [fix](join)ColumnNullable need handle const column with nullable const value (#13866)
dataalive commented on PR #13979: URL: https://github.com/apache/doris/pull/13979#issuecomment-1304382973 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13969: [feature-wip](dlf) prepare to support aliyun dlf
morningman commented on code in PR #13969:
URL: https://github.com/apache/doris/pull/13969#discussion_r1014552099
##
fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java:
##
@@ -74,61 +76,45 @@ public class EsExternalCatalog extends ExternalCatalog {
/**
* Default constructor for EsExternalCatalog.
*/
-public EsExternalCatalog(long catalogId, String name, Map
props) throws DdlException {
+public EsExternalCatalog(long catalogId, String name, Map
props) {
this.id = catalogId;
this.name = name;
this.type = "es";
-validate(props);
+setProperties(props);
this.catalogProperty = new CatalogProperty();
this.catalogProperty.setProperties(props);
}
-private void validate(Map properties) throws DdlException {
-if (properties == null) {
-throw new DdlException(
-"Please set properties of elasticsearch table, " + "they
are: hosts, user, password, index");
-}
-
-if (StringUtils.isBlank(properties.get(PROP_HOSTS))) {
-throw new DdlException("Hosts of ES table is null.");
-}
-nodes = properties.get(PROP_HOSTS).trim().split(",");
-// check protocol
-for (String seed : nodes) {
-if (!seed.startsWith("http")) {
-throw new DdlException("the protocol must be used");
-}
+private void setProperties(Map properties) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13975: [doc](routineload)Common mistakes in adding routine load
github-actions[bot] commented on PR #13975: URL: https://github.com/apache/doris/pull/13975#issuecomment-1304405748 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13975: [doc](routineload)Common mistakes in adding routine load
github-actions[bot] commented on PR #13975: URL: https://github.com/apache/doris/pull/13975#issuecomment-1304405742 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #13962: [Improvement](javaudf) improve java class loader usage
Gabriel39 merged PR #13962: URL: https://github.com/apache/doris/pull/13962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improvement](javaudf) improve java loader usage (#13962)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2ee7ba79a8 [Improvement](javaudf) improve java loader usage (#13962)
2ee7ba79a8 is described below
commit 2ee7ba79a87ba4daac9eadc7dec9e6677c300e05
Author: Gabriel
AuthorDate: Sat Nov 5 13:20:04 2022 +0800
[Improvement](javaudf) improve java loader usage (#13962)
---
be/src/runtime/user_function_cache.cpp | 1 +
.../apache/doris/analysis/CreateFunctionStmt.java | 81 +++---
2 files changed, 43 insertions(+), 39 deletions(-)
diff --git a/be/src/runtime/user_function_cache.cpp
b/be/src/runtime/user_function_cache.cpp
index 47e688ed5f..b985fec75d 100644
--- a/be/src/runtime/user_function_cache.cpp
+++ b/be/src/runtime/user_function_cache.cpp
@@ -394,6 +394,7 @@ Status
UserFunctionCache::_add_to_classpath(UserFunctionCacheEntry* entry) {
jmethodID url_ctor = env->GetMethodID(class_url, "",
"(Ljava/lang/String;)V");
jobject urlInstance = env->NewObject(class_url, url_ctor,
env->NewStringUTF(path.c_str()));
env->CallVoidMethod(class_loader, method_add_url, urlInstance);
+entry->is_loaded.store(true);
return Status::OK();
#else
return Status::InternalError("No libjvm is found!");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java
index 39fccef1dc..1629990a55 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateFunctionStmt.java
@@ -464,52 +464,55 @@ public class CreateFunctionStmt extends DdlStmt {
private void analyzeJavaUdf(String clazz) throws AnalysisException {
try {
URL[] urls = {new URL("jar:" + userFile + "!/")};
-URLClassLoader cl = URLClassLoader.newInstance(urls);
-Class udfClass = cl.loadClass(clazz);
-
-Method eval = null;
-for (Method m : udfClass.getMethods()) {
-if (!m.getDeclaringClass().equals(udfClass)) {
-continue;
+try (URLClassLoader cl = URLClassLoader.newInstance(urls)) {
+Class udfClass = cl.loadClass(clazz);
+
+Method eval = null;
+for (Method m : udfClass.getMethods()) {
+if (!m.getDeclaringClass().equals(udfClass)) {
+continue;
+}
+String name = m.getName();
+if (EVAL_METHOD_KEY.equals(name) && eval == null) {
+eval = m;
+} else if (EVAL_METHOD_KEY.equals(name)) {
+throw new AnalysisException(String.format(
+"UDF class '%s' has multiple methods with name
'%s' ", udfClass.getCanonicalName(),
+EVAL_METHOD_KEY));
+}
}
-String name = m.getName();
-if (EVAL_METHOD_KEY.equals(name) && eval == null) {
-eval = m;
-} else if (EVAL_METHOD_KEY.equals(name)) {
+if (eval == null) {
throw new AnalysisException(String.format(
-"UDF class '%s' has multiple methods with name
'%s' ", udfClass.getCanonicalName(),
-EVAL_METHOD_KEY));
+"No method '%s' in class '%s'!", EVAL_METHOD_KEY,
udfClass.getCanonicalName()));
+}
+if (Modifier.isStatic(eval.getModifiers())) {
+throw new AnalysisException(
+String.format("Method '%s' in class '%s' should be
non-static", eval.getName(),
+udfClass.getCanonicalName()));
+}
+if (!Modifier.isPublic(eval.getModifiers())) {
+throw new AnalysisException(
+String.format("Method '%s' in class '%s' should be
public", eval.getName(),
+udfClass.getCanonicalName()));
+}
+if (eval.getParameters().length !=
argsDef.getArgTypes().length) {
+throw new AnalysisException(
+String.format("The number of parameters for method
'%s' in class '%s' should be %d",
+eval.getName(),
udfClass.getCanonicalName(), argsDef.getArgTypes().length));
}
-}
-if (eval == null) {
-throw new AnalysisException(String.format(
-"No method '%s' in class '%s'!", EVAL_METHOD_KEY,
udfClass.getCanonicalName()));
-}
-
[GitHub] [doris] Yukang-Lian opened a new issue, #13982: [Feature] group_uniq_array function
Yukang-Lian opened a new issue, #13982: URL: https://github.com/apache/doris/issues/13982 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description **syntax**: groupUniqArray(x) or groupUniqArray(max_size)(x) **function**: Creates an array from different argument values. ```sql SELECT * FROM arrays_test ┌─s───┬─arr─┐ │ alex│ [1,2] │ │ hello │ [1,2] │ │ World │ [3,4,5] │ │ Goodbye │ [] │ └─┴─┘ SELECT groupUniqArray(arr) from arrays_test ┌─groupUniqArray(arr)─┐ │ [[],[1,2],[3,4,5]] │ └─┘ ``` ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on issue #13981: WIDTH_BUCKET function support
Yukang-Lian commented on issue #13981: URL: https://github.com/apache/doris/issues/13981#issuecomment-1304419306 Yeah, thanks for your advice. We will support it in the future. I think maybe in one month or two. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
