[GitHub] [doris] ByteYue commented on pull request #23741: [chore](ColdHotSeparation) refresh case
ByteYue commented on PR #23741: URL: https://github.com/apache/doris/pull/23741#issuecomment-1702264423 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mongo360 commented on pull request #23679: [Fix](stream load) stearm load record use valid txn info when two txn with same label
mongo360 commented on PR #23679: URL: https://github.com/apache/doris/pull/23679#issuecomment-1702267470 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] echo-hhj commented on pull request #23609: [typo](docs)Changes to expressions
echo-hhj commented on PR #23609: URL: https://github.com/apache/doris/pull/23609#issuecomment-1702246483 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #23715: [fix](Nereids) When col stats is Unknow, not expression should return the stats with selectivity of 1
XieJiann commented on code in PR #23715:
URL: https://github.com/apache/doris/pull/23715#discussion_r1312632942
##
fe/fe-core/src/main/java/org/apache/doris/statistics/Statistics.java:
##
@@ -131,6 +131,11 @@ public Statistics addColumnStats(Expression expression,
ColumnStatistic columnSt
return this;
}
+public boolean containUnknownChildStats() {
+return expressionToColumnStats.values().stream()
+.allMatch(e -> e.isUnKnown);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
yiguolei commented on code in PR #23616:
URL: https://github.com/apache/doris/pull/23616#discussion_r1312631408
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowConvertLSCStmt.java:
##
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+public class ShowConvertLSCStmt extends ShowStmt {
+
+private final String dbName;
+
+public ShowConvertLSCStmt(String dbName) {
+this.dbName = dbName;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws UserException {
+super.analyze(analyzer);
+
+if
(!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)
+&&
!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
+PrivPredicate.OPERATOR)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN/OPERATOR");
+}
+}
+
+@Override
+public ShowResultSetMetaData getMetaData() {
+ShowResultSetMetaData.Builder builder =
ShowResultSetMetaData.builder();
+Column tableNameColumn = new Column("table name",
ScalarType.createStringType());
Review Comment:
should also have database column??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #23746: [improvement](log) optimize template function log for performance
yiguolei commented on PR #23746: URL: https://github.com/apache/doris/pull/23746#issuecomment-1702261725 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (5b2360e836 -> eaf2a6a80e)
This is an automated email from the ASF dual-hosted git repository. ashingau pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5b2360e836 [opt](planner) speed up computeColumnsFilter on ScanNode (#23742) add eaf2a6a80e [fix](date) return right date value even if out of the range of date dictionary(#23664) No new revisions were added by this update. Summary of changes: be/src/service/doris_main.cpp | 1 - be/src/vec/exec/format/orc/vorc_reader.h | 8 ++--- .../format/parquet/fix_length_dict_decoder.hpp | 7 ++-- .../format/parquet/fix_length_plain_decoder.cpp| 9 ++--- be/src/vec/runtime/vdatetime_value.cpp | 39 ++ be/src/vec/runtime/vdatetime_value.h | 23 +++-- be/test/vec/exec/parquet/parquet_thrift_test.cpp | 1 - .../external_table_p2/hive/test_complex_types.out | 3 ++ .../hive/test_complex_types.groovy | 2 ++ 9 files changed, 68 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
yiguolei commented on code in PR #23616:
URL: https://github.com/apache/doris/pull/23616#discussion_r1312632931
##
fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnIdFlushDaemon.java:
##
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.alter.AlterLightSchChangeHelper;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.util.MasterDaemon;
+
+import com.google.common.collect.Maps;
+import org.apache.doris.persist.AlterLightSchemaChangeInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+
+/**
+ * note(tsy): this class is temporary, make table before 1.2 to enable light
schema change
+ */
+public class ColumnIdFlushDaemon extends MasterDaemon {
+
+private static final Logger LOG =
LogManager.getLogger(ColumnIdFlushDaemon.class);
+
+/**
+ * db name -> (tbl name -> status)
+ */
+private final Map> resultCollector;
+
+private final ReadWriteLock rwLock;
+
+private final BiConsumer flushFunc;
+
+public ColumnIdFlushDaemon() {
+super("colum-id-flusher", TimeUnit.HOURS.toMillis(1));
+resultCollector = Maps.newHashMap();
+rwLock = new ReentrantReadWriteLock();
+if (Config.enable_convert_light_weight_schema_change) {
+flushFunc = this::doFlush;
+} else {
+flushFunc = (db, table) -> record(db.getFullName(),
table.getName(), FlushStatus.init());
+}
+}
+
+@Override
+protected void runAfterCatalogReady() {
+flush();
+}
+
+private void flush() {
+List dbs = Env.getCurrentEnv().getInternalCatalog().getDbs();
+rwLock.writeLock().lock();
+try {
+for (Database db : dbs) {
+db.getTables()
+.stream()
+.filter(table -> table instanceof OlapTable)
+.map(table -> (OlapTable) table)
+.filter(olapTable ->
!olapTable.getTableProperty().getUseSchemaLightChange())
+.forEach(table -> flushFunc.accept(db, table));
+}
+} finally {
+rwLock.writeLock().unlock();
+}
+}
+
+private void doFlush(Database db, OlapTable table) {
Review Comment:
I think this should only run on master FE. Or the FE process will crash when
call logAlterLightSchemaChange
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang opened a new pull request, #23746: [improvement](log) optimize template function log for performance
xiaokang opened a new pull request, #23746: URL: https://github.com/apache/doris/pull/23746 ## Proposed changes change log level to debug and use format in template function log for performance. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23733: [fix](clone) fix cannot further repair clone replica which miss version data
dataroaring merged PR #23733: URL: https://github.com/apache/doris/pull/23733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [fix](clone) fix cannot further repair clone replica which miss version data (#23733)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new ea64eeb5e3 [fix](clone) fix cannot further repair clone replica which
miss version data (#23733)
ea64eeb5e3 is described below
commit ea64eeb5e300aefd78b028ade45d70a611fde6bb
Author: yujun
AuthorDate: Fri Sep 1 15:04:29 2023 +0800
[fix](clone) fix cannot further repair clone replica which miss version
data (#23733)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java | 4
1 file changed, 4 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
index 23852cf398..177d7ab2a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
@@ -437,6 +437,10 @@ public class Tablet extends MetaObject implements Writable
{
if (replica.getLastFailedVersion() > 0 || replica.getVersion() <
visibleVersion) {
// this replica is alive but version incomplete
+if (replica.needFurtherRepair() && needFurtherRepairReplica ==
null
+&& backend.isScheduleAvailable()) {
+needFurtherRepairReplica = replica;
+}
continue;
}
aliveAndVersionComplete++;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23731: [fix](tablet clone) fix tablet sched ctx toString cause null exeption
dataroaring merged PR #23731: URL: https://github.com/apache/doris/pull/23731 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](tablet clone) fix tablet sched ctx toString cause null exeption (#23731)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b843b79ddc [fix](tablet clone) fix tablet sched ctx toString cause
null exeption (#23731)
b843b79ddc is described below
commit b843b79ddca7a448b2b2da142a56096f35f71f90
Author: yujun
AuthorDate: Fri Sep 1 15:05:28 2023 +0800
[fix](tablet clone) fix tablet sched ctx toString cause null exeption
(#23731)
---
.../src/main/java/org/apache/doris/clone/TabletSchedCtx.java | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
index 4a3a1a6eb2..5b1ad25e68 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
@@ -1191,8 +1191,16 @@ public class TabletSchedCtx implements
Comparable {
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
-sb.append("tablet id: ").append(tabletId).append(", status:
").append(tabletStatus.name());
-sb.append(", state: ").append(state.name()).append(", type:
").append(type.name());
+sb.append("tablet id: ").append(tabletId);
+if (tabletStatus != null) {
+sb.append(", status: ").append(tabletStatus.name());
+}
+if (state != null) {
+sb.append(", state: ").append(state.name());
+}
+if (type != null) {
+sb.append(", type: ").append(type.name());
+}
if (type == Type.BALANCE && balanceType != null) {
sb.append(", balance: ").append(balanceType.name());
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23732: [fix](clone) fix cannot further repair clone replica which miss version data
dataroaring merged PR #23732: URL: https://github.com/apache/doris/pull/23732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](clone) fix cannot further repair clone replica which miss version data (#23732)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new c1ab29298b [fix](clone) fix cannot further repair clone replica which
miss version data (#23732)
c1ab29298b is described below
commit c1ab29298b23c79c2b0308c1c6a6011ed7c2399e
Author: yujun
AuthorDate: Fri Sep 1 15:05:40 2023 +0800
[fix](clone) fix cannot further repair clone replica which miss version
data (#23732)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java | 4
1 file changed, 4 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
index 003097baa2..1ba726cffc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java
@@ -451,6 +451,10 @@ public class Tablet extends MetaObject implements Writable
{
if (replica.getLastFailedVersion() > 0 || replica.getVersion() <
visibleVersion) {
// this replica is alive but version incomplete
+if (replica.needFurtherRepair() && needFurtherRepairReplica ==
null
+&& backend.isScheduleAvailable()) {
+needFurtherRepairReplica = replica;
+}
continue;
}
aliveAndVersionComplete++;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23732: [fix](clone) fix cannot further repair clone replica which miss version data
github-actions[bot] commented on PR #23732: URL: https://github.com/apache/doris/pull/23732#issuecomment-1702272786 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23732: [fix](clone) fix cannot further repair clone replica which miss version data
github-actions[bot] commented on PR #23732: URL: https://github.com/apache/doris/pull/23732#issuecomment-1702272853 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 02/02: [opt](planner) speed up computeColumnsFilter on ScanNode (#23742)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c981d109bcf1c842f3f0e8ecbeb3f471da9e09e2
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Sep 1 14:22:17 2023 +0800
[opt](planner) speed up computeColumnsFilter on ScanNode (#23742)
computeColumnsFilter compute filter on all table base schema's column.
However, it table is very wide, such as 5000 columns. It will take a
long time. This PR compare conjuncts size and columns size. If conjuncts
size is small than columns size, then collect slots from conjuncts to
avoid traverse all columns.
---
.../planner/BackendPartitionedSchemaScanNode.java | 2 +-
.../org/apache/doris/planner/OlapScanNode.java | 4 +-
.../java/org/apache/doris/planner/ScanNode.java| 56 +++---
.../doris/planner/external/ExternalScanNode.java | 4 +-
.../doris/planner/external/FileQueryScanNode.java | 2 +-
.../doris/planner/external/hudi/HudiScanNode.java | 2 +-
.../planner/external/paimon/PaimonScanNode.java| 2 +-
7 files changed, 47 insertions(+), 25 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java
index 6b2e278a01..53f48c60e8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java
@@ -78,7 +78,7 @@ public class BackendPartitionedSchemaScanNode extends
SchemaScanNode {
@Override
public void init(Analyzer analyzer) throws UserException {
super.init(analyzer);
-computeColumnFilter();
+computeColumnsFilter();
computePartitionInfo();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
index 7fdf7eda2d..806000c637 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
@@ -536,7 +536,7 @@ public class OlapScanNode extends ScanNode {
// lazy evaluation, since stmt is a prepared statment
isFromPrepareStmt = analyzer.getPrepareStmt() != null;
if (!isFromPrepareStmt) {
-computeColumnFilter();
+computeColumnsFilter();
computePartitionInfo();
}
computeTupleState(analyzer);
@@ -1116,7 +1116,7 @@ public class OlapScanNode extends ScanNode {
// Lazy evaluation
selectedIndexId = olapTable.getBaseIndexId();
// Only key columns
-computeColumnFilter(olapTable.getBaseSchemaKeyColumns());
+computeColumnsFilter(olapTable.getBaseSchemaKeyColumns());
computePartitionInfo();
scanBackendIds.clear();
scanTabletIds.clear();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
index 84b95c5a06..ef42ae1e4e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/ScanNode.java
@@ -154,32 +154,54 @@ public abstract class ScanNode extends PlanNode {
Set requiredByProjectSlotIdSet) throws UserException {
}
+private void computeColumnFilter(Column column, SlotDescriptor slotDesc) {
+// Set `columnFilters` all the time because `DistributionPruner` also
use this.
+// Maybe we could use `columnNameToRange` for `DistributionPruner` and
+// only create `columnFilters` when
`partition_prune_algorithm_version` is 1.
+PartitionColumnFilter keyFilter = createPartitionFilter(slotDesc,
conjuncts);
+if (null != keyFilter) {
+columnFilters.put(column.getName(), keyFilter);
+}
+
+ColumnRange columnRange = createColumnRange(slotDesc, conjuncts);
+if (columnRange != null) {
+columnNameToRange.put(column.getName(), columnRange);
+}
+}
+
// TODO(ML): move it into PrunerOptimizer
-public void computeColumnFilter(List columns) {
-for (Column column : columns) {
-SlotDescriptor slotDesc = desc.getColumnSlot(column.getName());
-if (null == slotDesc) {
-continue;
+public void computeColumnsFilter(List columns) {
+if (columns.size() > conjuncts.size()) {
+Set slotRefs = Sets.newHashSet();
+for (Expr conjunct : conjuncts) {
+conjunct.collect(SlotRef.class, slotRefs);
}
-// Set `columnFilters` all the time because `DistributionPruner`
also use this.
-// Maybe we could use `columnNameToRang
[GitHub] [doris] morrySnow closed pull request #23718: [fix](function) fix bug that array_max&min( array ) function.
morrySnow closed pull request #23718: [fix](function) fix bug that array_max&min( array ) function. URL: https://github.com/apache/doris/pull/23718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaka11chen commented on a diff in pull request #23745: [opt](hudi) reduce the memory usage of avro reader
kaka11chen commented on code in PR #23745:
URL: https://github.com/apache/doris/pull/23745#discussion_r1312638010
##
fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java:
##
@@ -84,53 +144,70 @@ public HudiJniScanner(int fetchSize, Map
params) {
@Override
public void open() throws IOException {
-Thread.currentThread().setContextClassLoader(classLoader);
-initTableInfo(split.requiredTypes(), split.requiredFields(),
predicates, fetchSize);
-long startTime = System.nanoTime();
-// RecordReader will use ProcessBuilder to start a hotspot process,
which may be stuck,
-// so use another process to kill this stuck process.
-// TODO(gaoxin): better way to solve the stuck process?
-AtomicBoolean isKilled = new AtomicBoolean(false);
-ScheduledExecutorService executorService =
Executors.newScheduledThreadPool(1);
-executorService.scheduleAtFixedRate(() -> {
-if (!isKilled.get()) {
-synchronized (HudiJniScanner.class) {
-List pids = Utils.getChildProcessIds(
-Utils.getCurrentProcId());
-for (long pid : pids) {
-String cmd = Utils.getCommandLine(pid);
-if (cmd != null &&
cmd.contains("org.openjdk.jol.vm.sa.AttachMain")) {
-Utils.killProcess(pid);
-isKilled.set(true);
-LOG.info("Kill hotspot debugger process " + pid);
+Future avroFuture = avroReadPool.submit(() -> {
+Thread.currentThread().setContextClassLoader(classLoader);
+initTableInfo(split.requiredTypes(), split.requiredFields(),
predicates, fetchSize);
+long startTime = System.nanoTime();
+// RecordReader will use ProcessBuilder to start a hotspot
process, which may be stuck,
+// so use another process to kill this stuck process.
+// TODO(gaoxin): better way to solve the stuck process?
+AtomicBoolean isKilled = new AtomicBoolean(false);
+ScheduledExecutorService executorService =
Executors.newScheduledThreadPool(1);
+executorService.scheduleAtFixedRate(() -> {
+if (!isKilled.get()) {
+synchronized (HudiJniScanner.class) {
+List pids = Utils.getChildProcessIds(
+Utils.getCurrentProcId());
+for (long pid : pids) {
+String cmd = Utils.getCommandLine(pid);
+if (cmd != null &&
cmd.contains("org.openjdk.jol.vm.sa.AttachMain")) {
+Utils.killProcess(pid);
+isKilled.set(true);
+LOG.info("Kill hotspot debugger process " +
pid);
+}
}
}
}
+}, 100, 1000, TimeUnit.MILLISECONDS);
+
+cleanResolverLock.readLock().lock();
+try {
+lastUpdateTime.set(System.currentTimeMillis());
+if (ugi != null) {
+recordIterator = ugi.doAs(
+(PrivilegedExceptionAction>)
() -> new MORSnapshotSplitReader(
+
split).buildScanIterator(split.requiredFields(), new Filter[0]));
+} else {
+recordIterator = new MORSnapshotSplitReader(split)
+.buildScanIterator(split.requiredFields(), new
Filter[0]);
+}
+} catch (Exception e) {
+LOG.error("Failed to open hudi scanner, split params:\n" +
debugString, e);
+throw new RuntimeException(e.getMessage(), e);
+} finally {
+cleanResolverLock.readLock().unlock();
}
-}, 100, 1000, TimeUnit.MILLISECONDS);
-try {
-if (ugi != null) {
-recordIterator = ugi.doAs(
-(PrivilegedExceptionAction>) ()
-> new MORSnapshotSplitReader(
-
split).buildScanIterator(split.requiredFields(), new Filter[0]));
-} else {
-recordIterator = new MORSnapshotSplitReader(split)
-.buildScanIterator(split.requiredFields(), new
Filter[0]);
+isKilled.set(true);
+executorService.shutdownNow();
Review Comment:
use style?
```
shutdown()
waitTermination(timeout)
if timed out, shutdownNow()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please con
[GitHub] [doris] github-actions[bot] commented on pull request #23729: [fix](tablet clone) fix backend not chose disk based on path hash
github-actions[bot] commented on PR #23729: URL: https://github.com/apache/doris/pull/23729#issuecomment-1702274968 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23729: [fix](tablet clone) fix backend not chose disk based on path hash
github-actions[bot] commented on PR #23729: URL: https://github.com/apache/doris/pull/23729#issuecomment-1702275029 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc commented on issue #23747: COUNT_ BY_ ENUM function not supported
ixzc commented on issue #23747: URL: https://github.com/apache/doris/issues/23747#issuecomment-1702275623 we will fix it later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23729: [fix](tablet clone) fix backend not chose disk based on path hash
dataroaring merged PR #23729: URL: https://github.com/apache/doris/pull/23729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](tablet clone) fix clone backend chose wrong disk (#23729)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 91c5640cae [fix](tablet clone) fix clone backend chose wrong disk
(#23729)
91c5640cae is described below
commit 91c5640cae529fcfd14eb15c247c04f2f869f365
Author: yujun
AuthorDate: Fri Sep 1 15:12:35 2023 +0800
[fix](tablet clone) fix clone backend chose wrong disk (#23729)
---
be/src/olap/storage_engine.cpp | 20
be/src/olap/storage_engine.h | 4 ++--
be/src/olap/task/engine_clone_task.cpp | 3 ++-
3 files changed, 20 insertions(+), 7 deletions(-)
diff --git a/be/src/olap/storage_engine.cpp b/be/src/olap/storage_engine.cpp
index b25e0862fc..d968c6010a 100644
--- a/be/src/olap/storage_engine.cpp
+++ b/be/src/olap/storage_engine.cpp
@@ -1043,7 +1043,7 @@ Status StorageEngine::create_tablet(const
TCreateTabletReq& request, RuntimeProf
return _tablet_manager->create_tablet(request, stores, profile);
}
-Status StorageEngine::obtain_shard_path(TStorageMedium::type storage_medium,
+Status StorageEngine::obtain_shard_path(TStorageMedium::type storage_medium,
int64_t path_hash,
std::string* shard_path, DataDir**
store) {
LOG(INFO) << "begin to process obtain root path. storage_medium=" <<
storage_medium;
@@ -1058,18 +1058,30 @@ Status
StorageEngine::obtain_shard_path(TStorageMedium::type storage_medium,
"no available disk can be used to create tablet.");
}
+*store = nullptr;
+if (path_hash != -1) {
+for (auto data_dir : stores) {
+if (data_dir->path_hash() == path_hash) {
+*store = data_dir;
+break;
+}
+}
+}
+if (*store == nullptr) {
+*store = stores[0];
+}
+
Status res = Status::OK();
uint64_t shard = 0;
-res = stores[0]->get_shard(&shard);
+res = (*store)->get_shard(&shard);
if (!res.ok()) {
LOG(WARNING) << "fail to get root path shard. res=" << res;
return res;
}
std::stringstream root_path_stream;
-root_path_stream << stores[0]->path() << "/" << DATA_PREFIX << "/" <<
shard;
+root_path_stream << (*store)->path() << "/" << DATA_PREFIX << "/" << shard;
*shard_path = root_path_stream.str();
-*store = stores[0];
LOG(INFO) << "success to process obtain root path. path=" << shard_path;
return res;
diff --git a/be/src/olap/storage_engine.h b/be/src/olap/storage_engine.h
index 822f0a458b..b81dcabab5 100644
--- a/be/src/olap/storage_engine.h
+++ b/be/src/olap/storage_engine.h
@@ -122,8 +122,8 @@ public:
//
// @param [out] shard_path choose an available root_path to clone new
tablet
// @return error code
-Status obtain_shard_path(TStorageMedium::type storage_medium, std::string*
shared_path,
- DataDir** store);
+Status obtain_shard_path(TStorageMedium::type storage_medium, int64_t
path_hash,
+ std::string* shared_path, DataDir** store);
// Load new tablet to make it effective.
//
diff --git a/be/src/olap/task/engine_clone_task.cpp
b/be/src/olap/task/engine_clone_task.cpp
index 19143e1079..9b03375756 100644
--- a/be/src/olap/task/engine_clone_task.cpp
+++ b/be/src/olap/task/engine_clone_task.cpp
@@ -179,7 +179,8 @@ Status EngineCloneTask::_do_clone() {
string local_shard_root_path;
DataDir* store = nullptr;
RETURN_IF_ERROR(StorageEngine::instance()->obtain_shard_path(
-_clone_req.storage_medium, &local_shard_root_path, &store));
+_clone_req.storage_medium, _clone_req.dest_path_hash,
&local_shard_root_path,
+&store));
auto tablet_dir = fmt::format("{}/{}/{}", local_shard_root_path,
_clone_req.tablet_id,
_clone_req.schema_hash);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17644: [fix](cooldown) Fix potential data loss when clone task's dst tablet is cooldown replica
github-actions[bot] commented on PR #17644: URL: https://github.com/apache/doris/pull/17644#issuecomment-1702280851 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring closed pull request #23143: [fix](colocate table) fix backend no delete colocate tablets data
dataroaring closed pull request #23143: [fix](colocate table) fix backend no delete colocate tablets data URL: https://github.com/apache/doris/pull/23143 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #23143: [fix](colocate table) fix backend no delete colocate tablets data
dataroaring commented on PR #23143: URL: https://github.com/apache/doris/pull/23143#issuecomment-1702281237 We dont main 1.1 any more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann opened a new pull request, #23748: [fix](Nereids) fix stats inject in or_expansion.groovy
XieJiann opened a new pull request, #23748: URL: https://github.com/apache/doris/pull/23748 ## Proposed changes make stats injection run first ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on pull request #23748: [fix](Nereids) fix stats inject in or_expansion.groovy
XieJiann commented on PR #23748: URL: https://github.com/apache/doris/pull/23748#issuecomment-1702283077 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on pull request #23715: [fix](Nereids) When col stats is Unknow, not expression should return the stats with selectivity of 1
XieJiann commented on PR #23715: URL: https://github.com/apache/doris/pull/23715#issuecomment-1702246834 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
yiguolei commented on code in PR #23616:
URL: https://github.com/apache/doris/pull/23616#discussion_r1312631103
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowConvertLSCStmt.java:
##
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+public class ShowConvertLSCStmt extends ShowStmt {
+
+private final String dbName;
+
+public ShowConvertLSCStmt(String dbName) {
+this.dbName = dbName;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws UserException {
+super.analyze(analyzer);
+
+if
(!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)
+&&
!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
+PrivPredicate.OPERATOR)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN/OPERATOR");
+}
+}
+
+@Override
+public ShowResultSetMetaData getMetaData() {
+ShowResultSetMetaData.Builder builder =
ShowResultSetMetaData.builder();
+Column tableNameColumn = new Column("table name",
ScalarType.createStringType());
+Column statusColum = new Column("status",
ScalarType.createStringType());
+builder.addColumn(tableNameColumn);
+builder.addColumn(statusColum);
+return builder.build();
+}
+
+@Override
+public RedirectStatus getRedirectStatus() {
+if (ConnectContext.get().getSessionVariable().getForwardToMaster()) {
+return RedirectStatus.FORWARD_NO_SYNC;
+} else {
Review Comment:
Is this command need forward to master? If not the show result is empty???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on pull request #23737: [enhancement](Nereids) make stats unchanged
XieJiann commented on PR #23737: URL: https://github.com/apache/doris/pull/23737#issuecomment-1702285346 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #23580: [Bug](materialized-view) fix mv not match because cast and alias name
BiteThet commented on PR #23580: URL: https://github.com/apache/doris/pull/23580#issuecomment-1702286595 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on a diff in pull request #23715: [fix](Nereids) When col stats is Unknow, not expression should return the stats with selectivity of 1
XieJiann commented on code in PR #23715:
URL: https://github.com/apache/doris/pull/23715#discussion_r1312634488
##
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##
@@ -348,8 +348,16 @@ A not in (1, 2, 3, 100):
return estimated;
}
+// Right Now, we just assume the selectivity is 1 when stats is Unknown
+private Statistics handleUnknownCase(EstimationContext context) {
+return context.statistics;
+}
+
@Override
public Statistics visitNot(Not not, EstimationContext context) {
+if (context.statistics.containUnknownChildStats()) {
+return handleUnknownCase(context);
Review Comment:
It helps to understand the behavior that the selectivity is 1 when there is
unknown input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #23580: [Bug](materialized-view) fix mv not match because cast and alias name
BiteThet commented on PR #23580: URL: https://github.com/apache/doris/pull/23580#issuecomment-1702287752 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated (c981d109bc -> 364c17bd94)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from c981d109bc [opt](planner) speed up computeColumnsFilter on ScanNode (#23742) add 364c17bd94 [improvement](log) optimize template function log for performance (#23746) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/catalog/FunctionSet.java | 20 1 file changed, 12 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
TangSiyang2001 commented on PR #23616: URL: https://github.com/apache/doris/pull/23616#issuecomment-1702289607 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23064: [improvement](colocate table) forbit change colocate table's replica allocation
github-actions[bot] commented on PR #23064: URL: https://github.com/apache/doris/pull/23064#issuecomment-1702290347 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23064: [improvement](colocate table) forbit change colocate table's replica allocation
github-actions[bot] commented on PR #23064: URL: https://github.com/apache/doris/pull/23064#issuecomment-1702290401 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on a diff in pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
TangSiyang2001 commented on code in PR #23616:
URL: https://github.com/apache/doris/pull/23616#discussion_r1312666728
##
fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnIdFlushDaemon.java:
##
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.alter.AlterLightSchChangeHelper;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.util.MasterDaemon;
+
+import com.google.common.collect.Maps;
+import org.apache.doris.persist.AlterLightSchemaChangeInfo;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+
+/**
+ * note(tsy): this class is temporary, make table before 1.2 to enable light
schema change
+ */
+public class ColumnIdFlushDaemon extends MasterDaemon {
+
+private static final Logger LOG =
LogManager.getLogger(ColumnIdFlushDaemon.class);
+
+/**
+ * db name -> (tbl name -> status)
+ */
+private final Map> resultCollector;
+
+private final ReadWriteLock rwLock;
+
+private final BiConsumer flushFunc;
+
+public ColumnIdFlushDaemon() {
+super("colum-id-flusher", TimeUnit.HOURS.toMillis(1));
+resultCollector = Maps.newHashMap();
+rwLock = new ReentrantReadWriteLock();
+if (Config.enable_convert_light_weight_schema_change) {
+flushFunc = this::doFlush;
+} else {
+flushFunc = (db, table) -> record(db.getFullName(),
table.getName(), FlushStatus.init());
+}
+}
+
+@Override
+protected void runAfterCatalogReady() {
+flush();
+}
+
+private void flush() {
+List dbs = Env.getCurrentEnv().getInternalCatalog().getDbs();
+rwLock.writeLock().lock();
+try {
+for (Database db : dbs) {
+db.getTables()
+.stream()
+.filter(table -> table instanceof OlapTable)
+.map(table -> (OlapTable) table)
+.filter(olapTable ->
!olapTable.getTableProperty().getUseSchemaLightChange())
+.forEach(table -> flushFunc.accept(db, table));
+}
+} finally {
+rwLock.writeLock().unlock();
+}
+}
+
+private void doFlush(Database db, OlapTable table) {
Review Comment:
It has been.
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowConvertLSCStmt.java:
##
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+public class ShowConvertLSCStmt extends ShowStmt {
+
+private final String dbName;
+
+public ShowConvertLSCStmt(String dbName) {
+this.dbName = dbName;
+}
+
+@Override
+public void analyze(Analyz
[GitHub] [doris] see-you-at-Zihuatanejo opened a new issue, #23749: [Bug] When using the JSONB data type, if there are integers less than -128 in the JSON data, the stored data is incorrect(使用JSONB数据类
see-you-at-Zihuatanejo opened a new issue, #23749:
URL: https://github.com/apache/doris/issues/23749
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Version
1.2.4
### What's Wrong?
When using the JSONB data type, if there are integers less than -128 in the
JSON data, the stored data is incorrect (it seems like an overflow?). However,
if it is a floating-point number or a positive integer, it will not occur.
使用JSONB数据类型时，若JSON数据中出现小于-128的整数，则保存的数据错误（感觉像是溢出了？）假如是浮点数、正整数则不会
```sql
CREATE TABLE test2.test_json (
id INT,
name JSONB
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
INSERT INTO test2.test_json (id, name)
VALUES
(0, '{"value":-129}'),
(1, '{"value":-129.0}'),
(2, '{"value":-128}'),
(3, '{"value":0}'),
(4, '{"value":200}');
SELECT * FROM test2.test_json
ORDER BY id;
```
### What You Expected?
When using the JSONB data type, the data is stored correctly when saving
negative integers.
使用JSONB数据类型时，保存负整数时，保存的数据是正确的
### How to Reproduce?
```sql
CREATE TABLE test2.test_json (
id INT,
name JSONB
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
INSERT INTO test2.test_json (id, name)
VALUES
(0, '{"value":-129}'),
(1, '{"value":-129.0}'),
(2, '{"value":-128}'),
(3, '{"value":0}'),
(4, '{"value":200}');
SELECT * FROM test2.test_json
ORDER BY id;
```
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on a diff in pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
TangSiyang2001 commented on code in PR #23616:
URL: https://github.com/apache/doris/pull/23616#discussion_r1312667092
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowConvertLSCStmt.java:
##
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.UserException;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+public class ShowConvertLSCStmt extends ShowStmt {
+
+private final String dbName;
+
+public ShowConvertLSCStmt(String dbName) {
+this.dbName = dbName;
+}
+
+@Override
+public void analyze(Analyzer analyzer) throws UserException {
+super.analyze(analyzer);
+
+if
(!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
PrivPredicate.ADMIN)
+&&
!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
+PrivPredicate.OPERATOR)) {
+
ErrorReport.reportAnalysisException(ErrorCode.ERR_SPECIFIC_ACCESS_DENIED_ERROR,
"ADMIN/OPERATOR");
+}
+}
+
+@Override
+public ShowResultSetMetaData getMetaData() {
+ShowResultSetMetaData.Builder builder =
ShowResultSetMetaData.builder();
+Column tableNameColumn = new Column("table name",
ScalarType.createStringType());
+Column statusColum = new Column("status",
ScalarType.createStringType());
+builder.addColumn(tableNameColumn);
+builder.addColumn(statusColum);
+return builder.build();
+}
+
+@Override
+public RedirectStatus getRedirectStatus() {
+if (ConnectContext.get().getSessionVariable().getForwardToMaster()) {
+return RedirectStatus.FORWARD_NO_SYNC;
+} else {
Review Comment:
modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23580: [Bug](materialized-view) fix mv not match because cast and alias name
github-actions[bot] commented on PR #23580: URL: https://github.com/apache/doris/pull/23580#issuecomment-1702294239 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23573: [fix](tablet sheduler) change sched period back to 1s
github-actions[bot] commented on PR #23573: URL: https://github.com/apache/doris/pull/23573#issuecomment-1702294751 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17644: [fix](cooldown) Fix potential data loss when clone task's dst tablet is cooldown replica
dataroaring merged PR #17644: URL: https://github.com/apache/doris/pull/17644 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](cooldown) Fix potential data loss when clone task's dst tablet is cooldown replica (#17644)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9d2fc78bd5 [fix](cooldown) Fix potential data loss when clone task's
dst tablet is cooldown replica (#17644)
9d2fc78bd5 is described below
commit 9d2fc78bd5d268a37a5cc61472bdb2c6cc559c27
Author: plat1ko
AuthorDate: Fri Sep 1 15:27:52 2023 +0800
[fix](cooldown) Fix potential data loss when clone task's dst tablet is
cooldown replica (#17644)
Co-authored-by: Yongqiang YANG
<98214048+dataroar...@users.noreply.github.com>
Co-authored-by: Kang
---
be/src/olap/tablet.cpp | 3 ++-
be/src/olap/task/engine_clone_task.cpp | 17 -
.../java/org/apache/doris/clone/TabletSchedCtx.java | 3 +++
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index 45d94f857f..afae3fa7e2 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -2144,7 +2144,8 @@ Status Tablet::_read_cooldown_meta(const
std::shared_ptr&
RETURN_IF_ERROR(tablet_meta_reader->read_at(0, {buf.get(), file_size},
&bytes_read));
tablet_meta_reader->close();
if (!tablet_meta_pb->ParseFromArray(buf.get(), file_size)) {
-return Status::InternalError("malformed tablet meta");
+return Status::InternalError("malformed tablet meta, path={}/{}",
fs->root_path().native(),
+ remote_meta_path);
}
return Status::OK();
}
diff --git a/be/src/olap/task/engine_clone_task.cpp
b/be/src/olap/task/engine_clone_task.cpp
index 9b03375756..75e17e433e 100644
--- a/be/src/olap/task/engine_clone_task.cpp
+++ b/be/src/olap/task/engine_clone_task.cpp
@@ -735,7 +735,22 @@ Status EngineCloneTask::_finish_full_clone(Tablet* tablet,
RETURN_IF_ERROR(tablet->create_rowset(rs_meta, &rs));
to_add.push_back(std::move(rs));
}
-
tablet->tablet_meta()->set_cooldown_meta_id(cloned_tablet_meta->cooldown_meta_id());
+{
+std::shared_lock cooldown_conf_rlock(tablet->get_cooldown_conf_lock());
+if (tablet->cooldown_conf_unlocked().first == tablet->replica_id()) {
+// If this replica is cooldown replica, MUST generate a new
`cooldown_meta_id` to avoid use `cooldown_meta_id`
+// generated in old cooldown term which may lead to such situation:
+// Replica A is cooldown replica, cooldown_meta_id=2,
+// Replica B: cooldown_replica=A, cooldown_meta_id=1
+// Replica A: full clone Replica A, cooldown_meta_id=1, but remote
cooldown_meta is still with cooldown_meta_id=2
+// After tablet report. FE finds all replicas' cooldowned data is
consistent
+// Replica A: confirm_unused_remote_files, delete some cooldowned
data of cooldown_meta_id=2
+// Replica B: follow_cooldown_data, cooldown_meta_id=2, data lost
+tablet->tablet_meta()->set_cooldown_meta_id(UniqueId::gen_uid());
+} else {
+
tablet->tablet_meta()->set_cooldown_meta_id(cloned_tablet_meta->cooldown_meta_id());
+}
+}
if (tablet->enable_unique_key_merge_on_write()) {
tablet->tablet_meta()->delete_bitmap() =
cloned_tablet_meta->delete_bitmap();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
index 5b1ad25e68..f3c03b3e1e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
@@ -925,6 +925,8 @@ public class TabletSchedCtx implements
Comparable {
committedVersion, /* use committed version as last failed
version */
-1 /* last success version */);
+LOG.info("create clone task to make new replica, tabletId={},
replicaId={}", tabletId,
+cloneReplica.getId());
// addReplica() method will add this replica to tablet inverted
index too.
tablet.addReplica(cloneReplica);
replicaId = cloneReplica.getId();
@@ -950,6 +952,7 @@ public class TabletSchedCtx implements
Comparable {
replicaId, schemaHash, Lists.newArrayList(tSrcBe),
storageMedium,
visibleVersion, (int) (taskTimeoutMs / 1000));
cloneTask.setPathHash(srcPathHash, destPathHash);
+LOG.info("create clone task to repair replica, tabletId={},
replicaId={}", tabletId, replicaId);
this.state = State.RUNNING;
return cloneTask;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](tablet sheduler) change sched period back to 1s (#23573)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e3886bcf2a [fix](tablet sheduler) change sched period back to 1s
(#23573)
e3886bcf2a is described below
commit e3886bcf2a296c6ab3279c6cd25a825d57b8ebe9
Author: yujun
AuthorDate: Fri Sep 1 15:29:59 2023 +0800
[fix](tablet sheduler) change sched period back to 1s (#23573)
This reverts commit 285bf978442fdff65fda5264ff40bd8291954ef2.
* change tablet sched peroid back to 1s
---
fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
index 789315c33f..d3e699b88e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
@@ -67,7 +67,7 @@ public class FeConstants {
public static String null_string = "\\N";
public static long tablet_checker_interval_ms = 20 * 1000L;
-public static long tablet_schedule_interval_ms = 100L;
+public static long tablet_schedule_interval_ms = 1000L;
public static String csv = "csv";
public static String csv_with_names = "csv_with_names";
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #23573: [fix](tablet sheduler) change sched period back to 1s
dataroaring merged PR #23573: URL: https://github.com/apache/doris/pull/23573 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23723: [chroe](ci) add script for create_issue_comment
github-actions[bot] commented on PR #23723: URL: https://github.com/apache/doris/pull/23723#issuecomment-1702300650 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Mryange opened a new pull request, #23750: [pipelineX](refactor) add repeat node in pipelineX
Mryange opened a new pull request, #23750: URL: https://github.com/apache/doris/pull/23750 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Mryange commented on pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
Mryange commented on PR #23750: URL: https://github.com/apache/doris/pull/23750#issuecomment-1702301147 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #23563: [fix](Nereids): resolve stats range to DateTime
jackwener commented on PR #23563: URL: https://github.com/apache/doris/pull/23563#issuecomment-1702302331 run buildfeut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
github-actions[bot] commented on PR #23750: URL: https://github.com/apache/doris/pull/23750#issuecomment-1702310329 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #23751: [Improvement](test)Add property to support manually use auto analyzer to analyze db.
Jibing-Li opened a new pull request, #23751: URL: https://github.com/apache/doris/pull/23751 This pr is mainly for test, make it easier to add regression test case for auto analyzer. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #23723: [chroe](ci) add script for create_issue_comment
Gabriel39 merged PR #23723: URL: https://github.com/apache/doris/pull/23723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [chroe](ci) add script for create_issue_comment (#23723)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fe5feae480 [chroe](ci) add script for create_issue_comment (#23723)
fe5feae480 is described below
commit fe5feae48011c4ddd11fb91742088bb25276a3f9
Author: Dongyang Li
AuthorDate: Fri Sep 1 15:46:25 2023 +0800
[chroe](ci) add script for create_issue_comment (#23723)
Co-authored-by: stephen
---
.../common/update-or-create-an-issue-comment.sh| 96 ++
1 file changed, 96 insertions(+)
diff --git
a/regression-test/pipeline/common/update-or-create-an-issue-comment.sh
b/regression-test/pipeline/common/update-or-create-an-issue-comment.sh
new file mode 100755
index 00..ce2c5bcbf2
--- /dev/null
+++ b/regression-test/pipeline/common/update-or-create-an-issue-comment.sh
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+create_issue_comment() {
+if [[ -z "$1" ]] || [[ -z "$2" ]]; then
+echo 'Usage: update_or_create_issue_comment
' && return 1
+fi
+if [[ -z ${github_token} ]]; then
+echo "require env: github_token" && return 1
+fi
+ISSUE_NUMBER="$1"
+COMMENT_BODY="$2"
+
+if curl -s \
+-X POST \
+-H "Accept: application/vnd.github+json" \
+-H "Authorization: Bearer ${github_token}" \
+
https://api.github.com/repos/apache/doris/issues/"${ISSUE_NUMBER}"/comments \
+-d "{\"body\": \"${COMMENT_BODY}\"}"; then
+echo -e "\033[32m Create issue(${ISSUE_NUMBER}) comment SUCCESS...
\033[0m" && return 0
+else
+echo -e "\033[31m Create issue(${ISSUE_NUMBER}) comment FAIL...
\033[0m" && return 1
+fi
+}
+
+update_or_create_issue_comment() {
+if [[ -z "$1" ]] || [[ -z "$2" ]]; then
+echo 'Usage: update_or_create_issue_comment
' && return 1
+fi
+if [[ -z ${github_token} ]]; then
+echo "require env: github_token" && return 1
+fi
+
+ISSUE_NUMBER="$1"
+COMMENT_BODY="$2"
+COMMENT_USER='"selectdb-robot"'
+
+# Refer to:
https://docs.github.com/en/rest/issues/comments#create-an-issue-comment
+
+file_name='comments_file'
+if ! curl -s \
+-H "Accept: application/vnd.github+json" \
+-H "Authorization: Bearer ${github_token}" \
+
https://api.github.com/repos/apache/doris/issues/"${ISSUE_NUMBER}"/comments \
+>"${file_name}"; then
+echo -e "\033[31m List issue(${ISSUE_NUMBER}) comments FAIL...
\033[0m" && return 1
+fi
+
+comments_count=$(jq '.[] | length' "${file_name}" | wc -l)
+for ((i = 1; i <= comments_count; ++i)); do
+comment_body=$(jq ".[-${i}].body" "${file_name}")
+comment_user=$(jq ".[-${i}].user.login" "${file_name}")
+if [[ "${comment_user}" == "${COMMENT_USER}" ]] &&
+[[ "${comment_body}" == *"${COMMENT_BODY:0:18}"* ]]; then
+echo "Similar comment already exists, will update it..."
+comment_id=$(jq ".[-${i}].id" "${file_name}")
+if curl -s \
+-X PATCH \
+-H "Accept: application/vnd.github+json" \
+-H "Authorization: Bearer ${github_token}" \
+
https://api.github.com/repos/apache/doris/issues/comments/"${comment_id}"; \
+-d "{\"body\":\"${COMMENT_BODY}\"}"; then
+echo -e "\033[32m Update issue(${ISSUE_NUMBER}) comment
SUCCESS... \033[0m" && return 0
+else
+echo -e "\033[31m Update issue(${ISSUE_NUMBER}) comment
FAIL... \033[0m" && return 1
+fi
+fi
+done
+
+echo "No similar comment exists, will create a new one..."
+if curl -s \
+-X POST \
+-H "Accept: application/vnd.github+json" \
+-H "Authorization: Bearer ${github_token}" \
+
https://api.github.com/repos/apache/doris/issues/"${ISSUE_NUMBER}"/comments \
+-d "{\"body\": \"${COMMENT_BODY}\"}"; then
+echo -e "\033[32m Create issue(${ISSUE_NUMBER}) comment SUCCESS...
\033[0m" && return 0
+else
+echo -e "\033
[GitHub] [doris] Gabriel39 commented on a diff in pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
Gabriel39 commented on code in PR #23750:
URL: https://github.com/apache/doris/pull/23750#discussion_r1312690184
##
be/src/pipeline/exec/repeat_operator.h:
##
@@ -45,5 +45,56 @@ class RepeatOperator final : public
StatefulOperator {
Status close(RuntimeState* state) override;
};
+class RepeatLocalState final : public PipelineXLocalState {
+public:
+ENABLE_FACTORY_CREATOR(RepeatLocalState);
+using Base = PipelineXLocalState;
+RepeatLocalState(RuntimeState* state, OperatorXBase* parent);
+
+Status init(RuntimeState* state, LocalStateInfo& info) override;
+Status close(RuntimeState* state) override;
+
+private:
+friend class RepeatOperatorX;
+std::unique_ptr _child_block;
+SourceState _child_source_state;
+bool _child_eos;
+int _repeat_id_idx;
+std::unique_ptr _intermediate_block {};
+};
+class RepeatOperatorX final : public OperatorXBase {
+public:
+RepeatOperatorX(ObjectPool* pool, const TPlanNode& tnode, const
DescriptorTbl& descs);
+Status setup_local_state(RuntimeState* state, LocalStateInfo& info)
override;
+Status get_block(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) override;
+Status init(const TPlanNode& tnode, RuntimeState* state) override;
+
+Status prepare(RuntimeState* state) override;
+Status open(RuntimeState* state) override;
+
+private:
+friend class RepeatLocalState;
+Status get_repeated_block(vectorized::Block* child_block, int
repeat_id_idx,
+ vectorized::Block* output_block);
+bool need_more_input_data(RuntimeState* state) const;
+
+Status pull(RuntimeState* state, vectorized::Block* output_block,
SourceState& source_state);
+Status push(RuntimeState* state, vectorized::Block* input_block,
SourceState& source_state);
+// Slot id set used to indicate those slots need to set to null.
+std::vector> _slot_id_set_list;
+// all slot id
+std::set _all_slot_ids;
+// An integer bitmap list, it indicates the bit position of the exprs not
null.
+std::vector _repeat_id_list;
+std::vector> _grouping_list;
+TupleId _output_tuple_id;
+const TupleDescriptor* _output_tuple_desc;
+
+std::vector _output_slots;
+
+vectorized::VExprContextSPtrs _expr_ctxs;
Review Comment:
Also need this in local state
##
be/src/pipeline/exec/repeat_operator.h:
##
@@ -45,5 +45,56 @@ class RepeatOperator final : public
StatefulOperator {
Status close(RuntimeState* state) override;
};
+class RepeatLocalState final : public PipelineXLocalState {
+public:
+ENABLE_FACTORY_CREATOR(RepeatLocalState);
+using Base = PipelineXLocalState;
+RepeatLocalState(RuntimeState* state, OperatorXBase* parent);
+
+Status init(RuntimeState* state, LocalStateInfo& info) override;
+Status close(RuntimeState* state) override;
+
+private:
+friend class RepeatOperatorX;
+std::unique_ptr _child_block;
+SourceState _child_source_state;
+bool _child_eos;
+int _repeat_id_idx;
+std::unique_ptr _intermediate_block {};
+};
+class RepeatOperatorX final : public OperatorXBase {
Review Comment:
```suggestion
class RepeatOperatorX final : public OperatorX {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
TangSiyang2001 commented on PR #23616: URL: https://github.com/apache/doris/pull/23616#issuecomment-1702324802 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc commented on issue #23747: COUNT_ BY_ ENUM function not supported
ixzc commented on issue #23747: URL: https://github.com/apache/doris/issues/23747#issuecomment-1702325996 https://github.com/apache/doris/pull/22071 . The pr has not been merged into 2.0.0 version and the version 2.0.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li commented on pull request #23751: [Improvement](test)Add property to support manually use auto analyzer to analyze db.
Jibing-Li commented on PR #23751: URL: https://github.com/apache/doris/pull/23751#issuecomment-1702327584 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23609: [typo](docs)Changes to expressions
github-actions[bot] commented on PR #23609: URL: https://github.com/apache/doris/pull/23609#issuecomment-1702329690 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] shuke987 commented on pull request #23738: [branch-2.0](fix) compile error when compiler opens -Werror and -Wunused-result
shuke987 commented on PR #23738: URL: https://github.com/apache/doris/pull/23738#issuecomment-1702329847 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] KassieZ opened a new pull request, #300: Fix 404 on home page
KassieZ opened a new pull request, #300: URL: https://github.com/apache/doris-website/pull/300 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yujun777 commented on pull request #23064: [improvement](colocate table) forbit change colocate table's replica allocation
yujun777 commented on PR #23064: URL: https://github.com/apache/doris/pull/23064#issuecomment-1702331459 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] luzhijing merged pull request #300: Fix 404 on home page
luzhijing merged PR #300: URL: https://github.com/apache/doris-website/pull/300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: Fix 404 on home page (#300)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 44134671cae Fix 404 on home page (#300)
44134671cae is described below
commit 44134671cae5955de1604a675b463dc9cfb954d7
Author: KassieZ <139741991+kass...@users.noreply.github.com>
AuthorDate: Fri Sep 1 15:57:55 2023 +0800
Fix 404 on home page (#300)
---
src/pages/index.tsx | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/pages/index.tsx b/src/pages/index.tsx
index de3c2129198..23914505e43 100644
--- a/src/pages/index.tsx
+++ b/src/pages/index.tsx
@@ -175,7 +175,7 @@ export default function Home(): JSX.Element {
}
footer={
More
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23715: [fix](Nereids) When col stats is Unknow, not expression should return the stats with selectivity of 1
hello-stephen commented on PR #23715: URL: https://github.com/apache/doris/pull/23715#issuecomment-1702333624 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.37 seconds stream load tsv: 544 seconds loaded 74807831229 Bytes, about 131 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162321766 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23622: [enhancement](stats) Split period from auto collector
hello-stephen commented on PR #23622: URL: https://github.com/apache/doris/pull/23622#issuecomment-1702335018 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.99 seconds stream load tsv: 546 seconds loaded 74807831229 Bytes, about 130 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17161996768 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23745: [opt](hudi) reduce the memory usage of avro reader
hello-stephen commented on PR #23745: URL: https://github.com/apache/doris/pull/23745#issuecomment-1702335080 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.33 seconds stream load tsv: 546 seconds loaded 74807831229 Bytes, about 130 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17161906758 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23746: [improvement](log) optimize template function log for performance
hello-stephen commented on PR #23746: URL: https://github.com/apache/doris/pull/23746#issuecomment-1702335169 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.59 seconds stream load tsv: 544 seconds loaded 74807831229 Bytes, about 131 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162197181 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23746: [improvement](log) optimize template function log for performance
hello-stephen commented on PR #23746: URL: https://github.com/apache/doris/pull/23746#issuecomment-1702335233 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.56 seconds stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17162330294 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23580: [Bug](materialized-view) fix mv not match because cast and alias name
hello-stephen commented on PR #23580: URL: https://github.com/apache/doris/pull/23580#issuecomment-1702337461 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.93 seconds stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17161882259 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23352: [feature](csv_serde)1.append csv serde for serialize to csv and deserialize from csv. 2.let csvReader use csv serde not text_converter
hello-stephen commented on PR #23352: URL: https://github.com/apache/doris/pull/23352#issuecomment-1702338505 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.7 seconds stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162182957 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] zzzxl1993 opened a new pull request, #115: [Optimize](chinese) Optimize chinese tokenizer index process
zzzxl1993 opened a new pull request, #115: URL: https://github.com/apache/doris-thirdparty/pull/115 1. chinese tokenzier use sDocument -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
hello-stephen commented on PR #23750: URL: https://github.com/apache/doris/pull/23750#issuecomment-1702343253 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 50.24 seconds stream load tsv: 538 seconds loaded 74807831229 Bytes, about 132 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162085143 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gojumprope commented on issue #23079: [Bug] JDBC SQL Server unable to filter cast(column as datetime)
gojumprope commented on issue #23079: URL: https://github.com/apache/doris/issues/23079#issuecomment-1702343638 Thanks, it is working when enable_nereids_planner is set to true -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gojumprope closed issue #23079: [Bug] JDBC SQL Server unable to filter cast(column as datetime)
gojumprope closed issue #23079: [Bug] JDBC SQL Server unable to filter cast(column as datetime) URL: https://github.com/apache/doris/issues/23079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zzzzzzzs opened a new pull request, #23752: [Enhancement](Load) stream tvf support json
zzzs opened a new pull request, #23752:
URL: https://github.com/apache/doris/pull/23752
## Proposed changes
Issue Number: close #https://github.com/apache/doris/issues/23678
## Further comments
stream tvf support json
```
[{"id":1, "name":"ftw", "age":18}]
[{"id":2, "name":"xxx", "age":17}]
[{"id":3, "name":"yyy", "age":19}]
```
example:
``` curl
curl -v --location-trusted -u root: -H "sql: insert into test.t1(c1, c2)
select id, name from http_stream(\"format\" = \"json\", \"strip_outer_array\" =
\"true\", \"read_json_by_line\" = \"true\")" -T /root/json_file.json
http://127.0.0.1:8030/api/_http_stream
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zzzzzzzs commented on pull request #23752: [Enhancement](Load) stream tvf support json
zzzs commented on PR #23752: URL: https://github.com/apache/doris/pull/23752#issuecomment-1702345139 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] gnehil opened a new pull request, #135: [fix] streaming write execution plan error
gnehil opened a new pull request, #135:
URL: https://github.com/apache/doris-spark-connector/pull/135
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
When writing doris through structured streaming, the following exception
will occur:
```java
23/09/01 14:13:40 ERROR MicroBatchExecution: Query [id =
c1d9450a-43ab-46fd-9826-10e0eb8ab1bc, runId =
21bd5cfe-84a1-487f-a2e7-ce27e2813691] terminated with error
org.apache.spark.sql.AnalysisException: Queries with streaming sources must
be executed with writeStream.start();
StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10,
offset#11L, timestamp#12, timestampType#13],
org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@56bd0218,
KafkaV2[Subscribe[spark_streaming_test]], {"spark_streaming_test":{"0":13}},
{"spark_streaming_test":{"0":14}}
at
org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.throwError(UnsupportedOperationChecker.scala:421)
at
org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.$anonfun$checkForBatch$1(UnsupportedOperationChecker.scala:38)
at
org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.$anonfun$checkForBatch$1$adapted(UnsupportedOperationChecker.scala:36)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:184)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:183)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:183)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:183)
at scala.collection.immutable.List.foreach(List.scala:392)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:183)
at
org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.checkForBatch(UnsupportedOperationChecker.scala:36)
at
org.apache.spark.sql.execution.QueryExecution.assertSupported(QueryExecution.scala:67)
at
org.apache.spark.sql.execution.QueryExecution.$anonfun$withCachedData$1(QueryExecution.scala:78)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at
org.apache.spark.sql.execution.QueryExecution.withCachedData$lzycompute(QueryExecution.scala:76)
at
org.apache.spark.sql.execution.QueryExecution.withCachedData(QueryExecution.scala:76)
at
org.apache.spark.sql.execution.QueryExecution.$anonfun$optimizedPlan$1(QueryExecution.scala:87)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at
org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at
org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:84)
at
org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:84)
at
org.apache.spark.sql.execution.QueryExecution.assertOptimized(QueryExecution.scala:95)
at
org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:113)
at
org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:110)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)
at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3241)
at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3239)
at org.apache.doris.spark.writer.DorisWriter.write(DorisWriter.scala:62)
at
org.apache.doris.spark.sql.DorisStreamLoadSink.addBatch(DorisStreamLoadSink.scala:37)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at
org.apache.spark.sql.execution
[GitHub] [doris] zclllyybb commented on pull request #23236: [Feature-WIP](partitions) Support auto partition
zclllyybb commented on PR #23236: URL: https://github.com/apache/doris/pull/23236#issuecomment-1702365549 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Mryange commented on pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
Mryange commented on PR #23750: URL: https://github.com/apache/doris/pull/23750#issuecomment-1702365812 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] waywtdcc commented on pull request #23594: [Enhancement](Column) Support int default value to int type column
waywtdcc commented on PR #23594: URL: https://github.com/apache/doris/pull/23594#issuecomment-1702368092 > You should add new tests, including create, alter, insert into, stream load, and not modify the old ones. We need to ensure compatibility with the old behaviors Thanks for your review. I made a commit, is that right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23616: [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically
hello-stephen commented on PR #23616: URL: https://github.com/apache/doris/pull/23616#issuecomment-1702368443 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.38 seconds stream load tsv: 534 seconds loaded 74807831229 Bytes, about 133 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 28.5 seconds inserted 1000 Rows, about 350K ops/s storage size: 17161956787 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23751: [Improvement](test)Add property to support manually use auto analyzer to analyze db.
hello-stephen commented on PR #23751: URL: https://github.com/apache/doris/pull/23751#issuecomment-170237 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.76 seconds stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17161839955 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23064: [improvement](colocate table) forbit change colocate table's replica allocation
hello-stephen commented on PR #23064: URL: https://github.com/apache/doris/pull/23064#issuecomment-1702370258 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.76 seconds stream load tsv: 536 seconds loaded 74807831229 Bytes, about 133 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17161970067 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] shuke987 commented on pull request #23738: [branch-2.0](fix) compile error when compiler opens -Werror and -Wunused-result
shuke987 commented on PR #23738: URL: https://github.com/apache/doris/pull/23738#issuecomment-1702373852 run compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23750: [pipelineX](refactor) add repeat node in pipelineX
github-actions[bot] commented on PR #23750: URL: https://github.com/apache/doris/pull/23750#issuecomment-1702374462 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TimothyDing closed issue #23482: [Bug] 使用官方提供的docker-compose文件部署后BE起不来
TimothyDing closed issue #23482: [Bug] 使用官方提供的docker-compose文件部署后BE起不来 URL: https://github.com/apache/doris/issues/23482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TimothyDing commented on issue #23482: [Bug] 使用官方提供的docker-compose文件部署后BE起不来
TimothyDing commented on issue #23482: URL: https://github.com/apache/doris/issues/23482#issuecomment-1702377616 @ixzc Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23021: [Feature](Multi-Catalog) support query doris bitmap column in external jdbc catalog
github-actions[bot] commented on PR #23021: URL: https://github.com/apache/doris/pull/23021#issuecomment-1702379355 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BePPPower commented on pull request #23635: [fix](Export) Concatenation the outfile sql for Export
BePPPower commented on PR #23635: URL: https://github.com/apache/doris/pull/23635#issuecomment-1702379826 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] GoGoWen commented on pull request #23021: [Feature](Multi-Catalog) support query doris bitmap column in external jdbc catalog
GoGoWen commented on PR #23021: URL: https://github.com/apache/doris/pull/23021#issuecomment-1702380612 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23752: [Enhancement](Load) stream tvf support json
hello-stephen commented on PR #23752: URL: https://github.com/apache/doris/pull/23752#issuecomment-1702380991 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.29 seconds stream load tsv: 538 seconds loaded 74807831229 Bytes, about 132 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162058540 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #23645: [Branch-2.0](Full compaction) Fix full compaction regressison test (#23487)
Yukang-Lian commented on PR #23645: URL: https://github.com/apache/doris/pull/23645#issuecomment-1702383555 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23613: [Fix](Planner) fix create table as select failed
github-actions[bot] commented on PR #23613: URL: https://github.com/apache/doris/pull/23613#issuecomment-1702384129 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23613: [Fix](Planner) fix create table as select failed
github-actions[bot] commented on PR #23613: URL: https://github.com/apache/doris/pull/23613#issuecomment-1702384068 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhiqiang-hhhh closed pull request #23716: Revert "[Improvement](pipeline) Cancel outdated query if original fe restarts"
zhiqiang- closed pull request #23716: Revert "[Improvement](pipeline) Cancel outdated query if original fe restarts" URL: https://github.com/apache/doris/pull/23716 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hubgeter opened a new pull request, #23753: [feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with my
hubgeter opened a new pull request, #23753: URL: https://github.com/apache/doris/pull/23753 add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---+--+--+---+-+---+ | Field | Type | Null | Key | Default | Extra | +---+--+--+---+-+---+ | CATALOG_ID| BIGINT | Yes | false | NULL| | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL| | | DATABASE_ID | BIGINT | Yes | false | NULL| | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL| | | TABLE_ID | BIGINT | Yes | false | NULL| | | TABLE_NAME| VARCHAR(64) | Yes | false | NULL| | +---+--+--+---+-+---+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *** 1. row *** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *** 1. row *** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *** 1. row *** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *** 1. row *** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *** 1. row *** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *** 1. row *** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ``` ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
