[GitHub] [doris] hello-stephen commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
hello-stephen commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657062921 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.01 seconds stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17170491643 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657065167 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #21911: [Feature](Export) Supports concurrently export of table data
morningman commented on code in PR #21911:
URL: https://github.com/apache/doris/pull/21911#discussion_r1278521853
##
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java:
##
@@ -266,17 +278,101 @@ private void generateQueryStmt() {
}
}
-List tableRefList = Lists.newArrayList();
-tableRefList.add(this.tableRef);
-FromClause fromClause = new FromClause(tableRefList);
+ArrayList> tableRefListPerQuery =
splitTablets(stmt);
+LOG.info("Export task is split into {} outfile statements.",
tableRefListPerQuery.size());
-SelectStmt selectStmt = new SelectStmt(list, fromClause,
this.whereExpr, null,
-null, null, LimitElement.NO_LIMIT);
-// generate outfile clause
-OutFileClause outfile = new OutFileClause(this.exportPath,
this.format, convertOutfileProperties());
-selectStmt.setOutFileClause(outfile);
-selectStmt.setOrigStmt(new OriginStatement(selectStmt.toSql(), 0));
-selectStmtList.add(selectStmt);
+if (LOG.isDebugEnabled()) {
+for (int i = 0; i < tableRefListPerQuery.size(); i++) {
+LOG.debug("Outfile clause {} is responsible for tables: {}", i,
+
tableRefListPerQuery.get(i).get(0).getSampleTabletIds());
+}
+}
+
+for (ArrayList tableRefList : tableRefListPerQuery) {
+FromClause fromClause = new FromClause(tableRefList);
+// generate outfile clause
+OutFileClause outfile = new OutFileClause(this.exportPath,
this.format, convertOutfileProperties());
+SelectStmt selectStmt = new SelectStmt(list, fromClause,
this.whereExpr, null,
+null, null, LimitElement.NO_LIMIT);
+selectStmt.setOutFileClause(outfile);
+selectStmt.setOrigStmt(new OriginStatement(selectStmt.toSql(), 0));
+selectStmtList.add(selectStmt);
+}
+stmtExecutorList = Arrays.asList(new
StmtExecutor[selectStmtList.size()]);
+if (LOG.isDebugEnabled()) {
+for (int i = 0; i < selectStmtList.size(); i++) {
+LOG.debug("Outfile clause {} is: {}", i,
selectStmtList.get(i).toSql());
+}
+}
+}
+
+private ArrayList> splitTablets(ExportStmt stmt)
throws UserException {
+// get tablets
+Database db =
Env.getCurrentEnv().getInternalCatalog().getDbOrAnalysisException(stmt.getTblName().getDb());
+OlapTable table =
db.getOlapTableOrAnalysisException(stmt.getTblName().getTbl());
+List tabletIdList = Lists.newArrayList();
+table.readLock();
+try {
+Collection partitions = new ArrayList();
+// get partitions
+// user specifies partitions, already checked in ExportStmt
+if (this.partitionNames != null) {
+if (partitionNames.size() >
Config.maximum_number_of_export_partitions) {
+throw new UserException("The partitions number of this
export job is larger than the maximum number"
++ " of partitions allowed by a export job");
+}
+for (String partName : this.partitionNames) {
+partitions.add(table.getPartition(partName));
+}
+} else {
+if (table.getPartitions().size() >
Config.maximum_number_of_export_partitions) {
+throw new UserException("The partitions number of this
export job is larger than the maximum number"
++ " of partitions allowed by a export job");
+}
+partitions = table.getPartitions();
+}
+
+// get tablets
+for (Partition partition : partitions) {
+partition.getVisibleVersion();
Review Comment:
Unused code?
##
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java:
##
@@ -266,17 +278,101 @@ private void generateQueryStmt() {
}
}
-List tableRefList = Lists.newArrayList();
-tableRefList.add(this.tableRef);
-FromClause fromClause = new FromClause(tableRefList);
+ArrayList> tableRefListPerQuery =
splitTablets(stmt);
+LOG.info("Export task is split into {} outfile statements.",
tableRefListPerQuery.size());
-SelectStmt selectStmt = new SelectStmt(list, fromClause,
this.whereExpr, null,
-null, null, LimitElement.NO_LIMIT);
-// generate outfile clause
-OutFileClause outfile = new OutFileClause(this.exportPath,
this.format, convertOutfileProperties());
-selectStmt.setOutFileClause(outfile);
-selectStmt.setOrigStmt(new OriginStatement(selectStmt.toSql(), 0));
-selectStmtList.add(selectStmt);
+if (LOG.isDebugEnabled()) {
+for (int i = 0; i < tableRefListPerQuery.size(); i++) {
+
[GitHub] [doris] github-actions[bot] commented on pull request #22360: [Opt](parquet) opt the performance of date parquet and orc convert to date
github-actions[bot] commented on PR #22360: URL: https://github.com/apache/doris/pull/22360#issuecomment-1657066045 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22360: [Opt](parquet) opt the performance of date parquet and orc convert to date
github-actions[bot] commented on PR #22360: URL: https://github.com/apache/doris/pull/22360#issuecomment-1657066053 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
github-actions[bot] commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657066170 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on pull request #22323: [fix](ipv6)Remove restrictions from IPv4 when add backend
zddr commented on PR #22323: URL: https://github.com/apache/doris/pull/22323#issuecomment-1657069044 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #22318: [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column s
morningman commented on code in PR #22318:
URL: https://github.com/apache/doris/pull/22318#discussion_r1278525100
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -624,7 +626,7 @@ public class SessionVariable implements Serializable,
Writable {
public boolean extractWideRangeExpr = true;
@VariableMgr.VarAttr(name = ENABLE_NEREIDS_DML)
-public boolean enableNereidsDML = false;
+public boolean enableNereidsDML = true;
Review Comment:
Why changing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Opt](parquet) opt the performance of date convertion (#22360)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4077338284 [Opt](parquet) opt the performance of date convertion
(#22360)
4077338284 is described below
commit 4077338284e677617902a34986e734d2d1af742a
Author: HappenLee
AuthorDate: Sun Jul 30 15:54:13 2023 +0800
[Opt](parquet) opt the performance of date convertion (#22360)
before：
```
mysql> select count(l_commitdate) from lineitem;
+-+
| count(l_commitdate) |
+-+
| 600037902 |
+-+
1 row in set (1.61 sec)
```
after:
```
mysql> select count(l_commitdate) from lineitem;
+-+
| count(l_commitdate) |
+-+
| 600037902 |
+-+
1 row in set (0.86 sec)
```
---
be/src/vec/exec/format/parquet/decoder.cpp | 6 ++
be/src/vec/exec/format/parquet/decoder.h| 1 +
be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp | 7 ---
be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp | 6 --
be/src/vec/runtime/vdatetime_value.h| 5 ++---
5 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/decoder.cpp
b/be/src/vec/exec/format/parquet/decoder.cpp
index a1fc3706fd..539fc04a10 100644
--- a/be/src/vec/exec/format/parquet/decoder.cpp
+++ b/be/src/vec/exec/format/parquet/decoder.cpp
@@ -177,5 +177,11 @@ void Decoder::init(FieldSchema* field_schema,
cctz::time_zone* ctz) {
_decode_params->scale_to_nano_factor = 1000;
}
}
+
+if (_decode_params->ctz) {
+VecDateTimeValue t;
+t.from_unixtime(0, *_decode_params->ctz);
+_decode_params->offset_days = doris::calc_daynr(t.year(), t.month(),
t.day());
+}
}
} // namespace doris::vectorized
diff --git a/be/src/vec/exec/format/parquet/decoder.h
b/be/src/vec/exec/format/parquet/decoder.h
index aacb3730ad..6c1030818c 100644
--- a/be/src/vec/exec/format/parquet/decoder.h
+++ b/be/src/vec/exec/format/parquet/decoder.h
@@ -71,6 +71,7 @@ struct DecodeParams {
static const cctz::time_zone utc0;
// schema.logicalType.TIMESTAMP.isAdjustedToUTC == true, we should set the
time zone
cctz::time_zone* ctz = nullptr;
+size_t offset_days = 0;
int64_t second_mask = 1;
int64_t scale_to_nano_factor = 1;
DecimalScaleParams decimal_scale;
diff --git a/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
b/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
index 887797636e..817b5e7f96 100644
--- a/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
+++ b/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
@@ -216,15 +216,16 @@ protected:
size_t data_index = column_data.size();
column_data.resize(data_index + select_vector.num_values() -
select_vector.num_filtered());
size_t dict_index = 0;
+
ColumnSelectVector::DataReadType read_type;
while (size_t run_length =
select_vector.get_next_run(&read_type)) {
switch (read_type) {
case ColumnSelectVector::CONTENT: {
for (size_t i = 0; i < run_length; ++i) {
-int64_t date_value = _dict_items[_indexes[dict_index++]];
+int64_t date_value =
+_dict_items[_indexes[dict_index++]] +
_decode_params->offset_days;
auto& v =
reinterpret_cast(column_data[data_index++]);
-v.from_unixtime(date_value * 24 * 60 * 60,
-*_decode_params->ctz); // day to seconds
+v.get_date_from_daynr(date_value);
if constexpr (std::is_same_v) {
// we should cast to date if using date v1.
v.cast_to_date();
diff --git a/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
b/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
index 72d362fe61..940e70db79 100644
--- a/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
+++ b/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
@@ -248,14 +248,16 @@ Status
FixLengthPlainDecoder::_decode_date(MutableColumnPtr& doris_column,
size_t data_index = column_data.size();
column_data.resize(data_index + select_vector.num_values() -
select_vector.num_filtered());
ColumnSelectVector::DataReadType read_type;
+
while (size_t run_length =
select_vector.get_next_run(&read_type)) {
switch (read_type) {
case ColumnSelectVector::CONTENT: {
for (size_t i = 0; i < run_length; ++i) {
char* buf_start = _data->data + _offset;
-
[GitHub] [doris] morningman merged pull request #22360: [Opt](parquet) opt the performance of date parquet and orc convert to date
morningman merged PR #22360: URL: https://github.com/apache/doris/pull/22360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #22371: [refactor](reader) move reader from vec/exec/scan to vec/exec/format
morningman opened a new pull request, #22371: URL: https://github.com/apache/doris/pull/22371 ## Proposed changes This `readers` should be in `vec/exec/format` ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22323: [fix](ipv6)Remove restrictions from IPv4 when add backend
github-actions[bot] commented on PR #22323: URL: https://github.com/apache/doris/pull/22323#issuecomment-1657071602 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
hello-stephen commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657072605 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.66 seconds stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 36.5 seconds inserted 1000 Rows, about 273K ops/s storage size: 17167360166 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22371: [refactor](reader) move reader from vec/exec/scan to vec/exec/format
github-actions[bot] commented on PR #22371: URL: https://github.com/apache/doris/pull/22371#issuecomment-1657072816 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22323: [fix](ipv6)Remove restrictions from IPv4 when add backend
hello-stephen commented on PR #22323: URL: https://github.com/apache/doris/pull/22323#issuecomment-1657077711 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.63 seconds stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17162710159 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #22364: [FIX](decimal)fix decimal precision
amorynan commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657083151 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22364: [FIX](decimal)fix decimal precision
github-actions[bot] commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657084233 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22364: [FIX](decimal)fix decimal precision
hello-stephen commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657096730 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.22 seconds stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162483555 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657097417 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] imchao9 commented on issue #20541: [Bug] Version 2.0.0 Alpha1 failed to start under JDK11.
imchao9 commented on issue #20541: URL: https://github.com/apache/doris/issues/20541#issuecomment-1657098180 same question -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #22372: [enhancement](exchange) ignore overcrowded in brpc used by exchange
dataroaring opened a new pull request, #22372: URL: https://github.com/apache/doris/pull/22372 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
github-actions[bot] commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657098983 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22372: [enhancement](exchange) ignore overcrowded in brpc used by exchange
github-actions[bot] commented on PR #22372: URL: https://github.com/apache/doris/pull/22372#issuecomment-1657100424 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
BiteThet commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657100793 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22372: [enhancement](exchange) ignore overcrowded in brpc used by exchange
github-actions[bot] commented on PR #22372: URL: https://github.com/apache/doris/pull/22372#issuecomment-1657101353 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
github-actions[bot] commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657101991 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
hello-stephen commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657105314 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.84 seconds stream load tsv: 540 seconds loaded 74807831229 Bytes, about 132 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 30.0 seconds inserted 1000 Rows, about 333K ops/s storage size: 17162320155 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangtianyi2004 opened a new pull request, #22373: Update standard-deployment.md
wangtianyi2004 opened a new pull request, #22373: URL: https://github.com/apache/doris/pull/22373 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangtianyi2004 opened a new pull request, #22374: Update standard-deployment.md
wangtianyi2004 opened a new pull request, #22374: URL: https://github.com/apache/doris/pull/22374 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan opened a new pull request, #22375: [FIX](complex-type)fix complex type nested col_const
amorynan opened a new pull request, #22375: URL: https://github.com/apache/doris/pull/22375 ## Proposed changes Issue Number: close #xxx for array/map/struct in mysql_writer unpack_if_const only unpack self column not nested , so col_const should not used in nested column. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
amorynan commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657113888 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
github-actions[bot] commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657115114 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #22376: [Improvement](multi catalog) Support Iceberg, Paimon and MaxCompute table in nereids. (#22338)
Jibing-Li opened a new pull request, #22376: URL: https://github.com/apache/doris/pull/22376 Add Iceberg external table, Paimon external table and Max compute external table support in Nereids getLogicalPlan function to support these tables in nereids planner. This is a cherry-pick of commit: https://github.com/apache/doris/pull/22338 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22371: [refactor](reader) move reader from vec/exec/scan to vec/exec/format
github-actions[bot] commented on PR #22371: URL: https://github.com/apache/doris/pull/22371#issuecomment-1657118924 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] lide-reed commented on issue #22324: [Bug] BE core at je_arena_dalloc_promoted
lide-reed commented on issue #22324: URL: https://github.com/apache/doris/issues/22324#issuecomment-1657120283 It seems fixed by https://github.com/apache/doris/pull/21486 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
hello-stephen commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657122286 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.92 seconds stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 39 seconds loaded 2358488459 Bytes, about 57 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17162305740 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657123314 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
github-actions[bot] commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657124753 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
xiaokang commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657124983 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaka11chen commented on pull request #22318: [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema.
kaka11chen commented on PR #22318: URL: https://github.com/apache/doris/pull/22318#issuecomment-1657125710 > Please also update the hive catalog doc. ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaka11chen commented on pull request #22318: [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema.
kaka11chen commented on PR #22318: URL: https://github.com/apache/doris/pull/22318#issuecomment-1657125765 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22318: [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column sc
github-actions[bot] commented on PR #22318: URL: https://github.com/apache/doris/pull/22318#issuecomment-1657126828 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau commented on pull request #22331: [fix](complex_type) throw error when reading complex types in broker/stream load
AshinGau commented on PR #22331: URL: https://github.com/apache/doris/pull/22331#issuecomment-1657127713 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22331: [fix](complex_type) throw error when reading complex types in broker/stream load
github-actions[bot] commented on PR #22331: URL: https://github.com/apache/doris/pull/22331#issuecomment-1657129104 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #22367: [opt](datetime) opt datetime parsing performance
zclllyybb commented on PR #22367: URL: https://github.com/apache/doris/pull/22367#issuecomment-1657130093 run build all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #22367: [opt](datetime) opt datetime parsing performance
zclllyybb commented on PR #22367: URL: https://github.com/apache/doris/pull/22367#issuecomment-1657130161 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #22372: [enhancement](exchange) ignore overcrowded in brpc used by exchange
dataroaring commented on PR #22372: URL: https://github.com/apache/doris/pull/22372#issuecomment-1657130988 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22367: [opt](datetime) opt datetime parsing performance
github-actions[bot] commented on PR #22367: URL: https://github.com/apache/doris/pull/22367#issuecomment-1657131833 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
hello-stephen commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657131916 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.74 seconds stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17164983479 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
BiteThet commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657132181 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22318: [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema.
hello-stephen commented on PR #22318: URL: https://github.com/apache/doris/pull/22318#issuecomment-1657136441 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.83 seconds stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17165849408 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
github-actions[bot] commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657143144 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #22249: [enhance](ColdHeatSeparation) carry use path style info along with cold heat separation to support using minio
dataroaring merged PR #22249: URL: https://github.com/apache/doris/pull/22249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](ColdHeatSeparation) carry use path style info along with cold heat separation to support using minio (#22249)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 06e4061b94 [enhance](ColdHeatSeparation) carry use path style info
along with cold heat separation to support using minio (#22249)
06e4061b94 is described below
commit 06e4061b94281ee53d2a01ba105219ab5074698f
Author: AlexYue
AuthorDate: Sun Jul 30 21:03:33 2023 +0800
[enhance](ColdHeatSeparation) carry use path style info along with cold
heat separation to support using minio (#22249)
---
be/src/agent/task_worker_pool.cpp | 3 +++
.../apache/doris/datasource/property/constants/S3Properties.java| 6 ++
gensrc/thrift/AgentService.thrift | 1 +
3 files changed, 10 insertions(+)
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index f40ff6ef4c..6fa6c88197 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -1116,6 +1116,9 @@ void
TaskWorkerPool::_push_storage_policy_worker_thread_callback() {
s3_conf.connect_timeout_ms =
resource.s3_storage_param.conn_timeout_ms;
s3_conf.max_connections = resource.s3_storage_param.max_conn;
s3_conf.request_timeout_ms =
resource.s3_storage_param.request_timeout_ms;
+// When using cold heat separation in minio, user might use ip
address directly,
+// which needs enable use_virtual_addressing to true
+s3_conf.use_virtual_addressing =
resource.s3_storage_param.use_path_style;
std::shared_ptr fs;
if (existed_resource.fs == nullptr) {
st = io::S3FileSystem::create(s3_conf,
std::to_string(resource.id), &fs);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
index 13fb6ee336..3d3a003466 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/constants/S3Properties.java
@@ -22,6 +22,7 @@ import org.apache.doris.common.DdlException;
import org.apache.doris.datasource.credentials.CloudCredential;
import org.apache.doris.datasource.credentials.CloudCredentialWithEndpoint;
import org.apache.doris.datasource.credentials.DataLakeAWSCredentialsProvider;
+import org.apache.doris.datasource.property.PropertyConverter;
import org.apache.doris.thrift.TS3StorageParam;
import com.amazonaws.auth.EnvironmentVariableCredentialsProvider;
@@ -241,6 +242,9 @@ public class S3Properties extends BaseProperties {
if (properties.containsKey(S3Properties.Env.BUCKET)) {
properties.putIfAbsent(S3Properties.BUCKET,
properties.get(S3Properties.Env.BUCKET));
}
+if (properties.containsKey(PropertyConverter.USE_PATH_STYLE)) {
+properties.putIfAbsent(PropertyConverter.USE_PATH_STYLE,
properties.get(PropertyConverter.USE_PATH_STYLE));
+}
}
public static TS3StorageParam getS3TStorageParam(Map
properties) {
@@ -261,6 +265,8 @@ public class S3Properties extends BaseProperties {
String connTimeoutMs =
properties.get(S3Properties.CONNECTION_TIMEOUT_MS);
s3Info.setMaxConn(Integer.parseInt(connTimeoutMs == null
? S3Properties.Env.DEFAULT_CONNECTION_TIMEOUT_MS :
connTimeoutMs));
+String usePathStyle =
properties.getOrDefault(PropertyConverter.USE_PATH_STYLE, "false");
+s3Info.setUsePathStyle(Boolean.parseBoolean(usePathStyle));
return s3Info;
}
}
diff --git a/gensrc/thrift/AgentService.thrift
b/gensrc/thrift/AgentService.thrift
index 13bee01da8..c1d25d5e66 100644
--- a/gensrc/thrift/AgentService.thrift
+++ b/gensrc/thrift/AgentService.thrift
@@ -70,6 +70,7 @@ struct TS3StorageParam {
7: optional i32 conn_timeout_ms = 1000
8: optional string root_path
9: optional string bucket
+10: optional bool use_path_style = false
}
struct TStoragePolicy {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #22377: [fix](show-table-status) fix hive view NPE and external meta cache refresh issue
morningman opened a new pull request, #22377: URL: https://github.com/apache/doris/pull/22377 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22105: [enhance](S3) add s3 bvar metrics for all s3 operation
github-actions[bot] commented on PR #22105: URL: https://github.com/apache/doris/pull/22105#issuecomment-1657155132 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #22105: [enhance](S3) add s3 bvar metrics for all s3 operation
dataroaring merged PR #22105: URL: https://github.com/apache/doris/pull/22105 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](S3) add s3 bvar metrics for all s3 operation (#22105)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 63a9a886f5 [enhance](S3) add s3 bvar metrics for all s3 operation
(#22105)
63a9a886f5 is described below
commit 63a9a886f551b7b7baee11e73eb70194c7bdb949
Author: AlexYue
AuthorDate: Sun Jul 30 21:09:17 2023 +0800
[enhance](S3) add s3 bvar metrics for all s3 operation (#22105)
---
be/src/io/fs/s3_file_reader.cpp | 2 ++
be/src/io/fs/s3_file_system.cpp | 11 +++
be/src/io/fs/s3_file_writer.cpp | 6 ++
be/src/util/s3_util.cpp | 13 +
be/src/util/s3_util.h | 16
5 files changed, 48 insertions(+)
diff --git a/be/src/io/fs/s3_file_reader.cpp b/be/src/io/fs/s3_file_reader.cpp
index 2403b2497e..950f9cff17 100644
--- a/be/src/io/fs/s3_file_reader.cpp
+++ b/be/src/io/fs/s3_file_reader.cpp
@@ -33,6 +33,7 @@
#include "common/compiler_util.h" // IWYU pragma: keep
#include "io/fs/s3_common.h"
#include "util/doris_metrics.h"
+#include "util/s3_util.h"
namespace doris {
namespace io {
@@ -95,6 +96,7 @@ Status S3FileReader::read_at_impl(size_t offset, Slice
result, size_t* bytes_rea
return Status::InternalError("init s3 client error");
}
auto outcome = client->GetObject(request);
+s3_bvar::s3_get_total << 1;
if (!outcome.IsSuccess()) {
return Status::IOError("failed to read from {}: {}", _path.native(),
outcome.GetError().GetMessage());
diff --git a/be/src/io/fs/s3_file_system.cpp b/be/src/io/fs/s3_file_system.cpp
index ca4fd0bda8..8bde016aae 100644
--- a/be/src/io/fs/s3_file_system.cpp
+++ b/be/src/io/fs/s3_file_system.cpp
@@ -161,6 +161,7 @@ Status S3FileSystem::delete_file_impl(const Path& file) {
request.WithBucket(_s3_conf.bucket).WithKey(key);
auto outcome = client->DeleteObject(request);
+s3_bvar::s3_delete_total << 1;
if (outcome.IsSuccess() ||
outcome.GetError().GetResponseCode() ==
Aws::Http::HttpResponseCode::NOT_FOUND) {
return Status::OK();
@@ -184,6 +185,7 @@ Status S3FileSystem::delete_directory_impl(const Path& dir)
{
bool is_trucated = false;
do {
auto outcome = client->ListObjectsV2(request);
+s3_bvar::s3_list_total << 1;
if (!outcome.IsSuccess()) {
return Status::IOError("failed to list objects when delete dir {}:
{}", dir.native(),
error_msg(prefix, outcome));
@@ -199,6 +201,7 @@ Status S3FileSystem::delete_directory_impl(const Path& dir)
{
del.WithObjects(std::move(objects)).SetQuiet(true);
delete_request.SetDelete(std::move(del));
auto delete_outcome = client->DeleteObjects(delete_request);
+s3_bvar::s3_delete_total << 1;
if (!delete_outcome.IsSuccess()) {
return Status::IOError("failed to delete dir {}: {}",
dir.native(),
error_msg(prefix, delete_outcome));
@@ -243,6 +246,7 @@ Status S3FileSystem::batch_delete_impl(const
std::vector& remote_files) {
del.WithObjects(std::move(objects)).SetQuiet(true);
delete_request.SetDelete(std::move(del));
auto delete_outcome = client->DeleteObjects(delete_request);
+s3_bvar::s3_delete_total << 1;
if (UNLIKELY(!delete_outcome.IsSuccess())) {
return Status::IOError(
"failed to delete objects: {}",
@@ -268,6 +272,7 @@ Status S3FileSystem::exists_impl(const Path& path, bool*
res) const {
request.WithBucket(_s3_conf.bucket).WithKey(key);
auto outcome = client->HeadObject(request);
+s3_bvar::s3_head_total << 1;
if (outcome.IsSuccess()) {
*res = true;
} else if (outcome.GetError().GetResponseCode() ==
Aws::Http::HttpResponseCode::NOT_FOUND) {
@@ -288,6 +293,7 @@ Status S3FileSystem::file_size_impl(const Path& file,
int64_t* file_size) const
request.WithBucket(_s3_conf.bucket).WithKey(key);
auto outcome = client->HeadObject(request);
+s3_bvar::s3_head_total << 1;
if (outcome.IsSuccess()) {
*file_size = outcome.GetResult().GetContentLength();
} else {
@@ -314,6 +320,7 @@ Status S3FileSystem::list_impl(const Path& dir, bool
only_file, std::vectorListObjectsV2(request);
+s3_bvar::s3_list_total << 1;
if (!outcome.IsSuccess()) {
return Status::IOError("failed to list {}: {}", dir.native(),
error_msg(prefix, outcome));
@@ -430,6 +437,7 @@ Status S3FileSystem::direct_upload_impl(const Path&
remote_file, const std::stri
remote_file.native());
}
Aws::S3::Model::PutObjectOutcome response = _client->PutObject(request);
+s3_bvar::s3_put_total << 1;
if (response.IsSuccess()) {
[GitHub] [doris] xiaokang merged pull request #22376: [Improvement](multi catalog) Support Iceberg, Paimon and MaxCompute table in nereids. (#22338)
xiaokang merged PR #22376: URL: https://github.com/apache/doris/pull/22376 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [Improvement](multi catalog) Support Iceberg, Paimon and MaxCompute table in nereids. (#22338) (#22376)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 6644eb40a2 [Improvement](multi catalog) Support Iceberg, Paimon and
MaxCompute table in nereids. (#22338) (#22376)
6644eb40a2 is described below
commit 6644eb40a27cb00dfd5ae7e040c48a6cddd1714e
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Sun Jul 30 21:13:54 2023 +0800
[Improvement](multi catalog) Support Iceberg, Paimon and MaxCompute table
in nereids. (#22338) (#22376)
---
.../glue/translator/PhysicalPlanTranslator.java| 4 ++
.../doris/nereids/rules/analysis/BindRelation.java | 8 ++-
.../doris/planner/external/MaxComputeScanNode.java | 4 ++
.../test_external_catalog_icebergv2_nereids.out| 74 +++
.../test_external_catalog_icebergv2_nereids.groovy | 84 ++
5 files changed, 172 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 8f9b336585..4b460e944d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -48,6 +48,7 @@ import org.apache.doris.catalog.external.ExternalTable;
import org.apache.doris.catalog.external.HMSExternalTable;
import org.apache.doris.catalog.external.IcebergExternalTable;
import org.apache.doris.catalog.external.JdbcExternalTable;
+import org.apache.doris.catalog.external.MaxComputeExternalTable;
import org.apache.doris.catalog.external.PaimonExternalTable;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.Util;
@@ -157,6 +158,7 @@ import org.apache.doris.planner.SortNode;
import org.apache.doris.planner.TableFunctionNode;
import org.apache.doris.planner.UnionNode;
import org.apache.doris.planner.external.HiveScanNode;
+import org.apache.doris.planner.external.MaxComputeScanNode;
import org.apache.doris.planner.external.hudi.HudiScanNode;
import org.apache.doris.planner.external.iceberg.IcebergScanNode;
import org.apache.doris.planner.external.jdbc.JdbcScanNode;
@@ -403,6 +405,8 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor(tableQualifier,
hiveViewPlan);
}
}
-return new LogicalFileScan(RelationUtil.newRelationId(),
-(HMSExternalTable) table, ImmutableList.of(dbName));
+return new LogicalFileScan(RelationUtil.newRelationId(),
(HMSExternalTable) table, tableQualifier);
+case ICEBERG_EXTERNAL_TABLE:
+case PAIMON_EXTERNAL_TABLE:
+case MAX_COMPUTE_EXTERNAL_TABLE:
+return new LogicalFileScan(RelationUtil.newRelationId(),
(ExternalTable) table, tableQualifier);
case SCHEMA:
return new LogicalSchemaScan(RelationUtil.newRelationId(),
table, ImmutableList.of(dbName));
case JDBC_EXTERNAL_TABLE:
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java
index 11e9bafd86..1030a67a30 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/MaxComputeScanNode.java
@@ -44,6 +44,10 @@ public class MaxComputeScanNode extends FileQueryScanNode {
private final MaxComputeExternalCatalog catalog;
public static final int MIN_SPLIT_SIZE = 4096;
+public MaxComputeScanNode(PlanNodeId id, TupleDescriptor desc, boolean
needCheckColumnPriv) {
+this(id, desc, "MCScanNode", StatisticalType.MAX_COMPUTE_SCAN_NODE,
needCheckColumnPriv);
+}
+
public MaxComputeScanNode(PlanNodeId id, TupleDescriptor desc, String
planNodeName,
StatisticalType statisticalType, boolean
needCheckColumnPriv) {
super(id, desc, planNodeName, statisticalType, needCheckColumnPriv);
diff --git
a/regression-test/data/external_table_emr_p2/iceberg/test_external_catalog_icebergv2_nereids.out
b/regression-test/data/external_table_emr_p2/iceberg/test_external_catalog_icebergv2_nereids.out
new file mode 100644
index 00..a6e68d2f62
--- /dev/null
+++
b/regression-test/data/external_table_emr_p2/iceberg/test_external_catalog_icebergv2_nereids.out
@@ -0,0 +1,74 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !q01 --
+149988
+
+-- !q02 --
+1
+3
+4
+7
+
+-- !q03 --
+8242263
+
+-- !q04 --
+0
+
+-- !q05 --
+1 Customer#1 IVhzIApeRb ot,c,E
[GitHub] [doris] xiaokang merged pull request #22281: [fix](cast) fix wrong result of casting empty string to array date
xiaokang merged PR #22281: URL: https://github.com/apache/doris/pull/22281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](cast) fix wrong result of casting empty string to array date (#22281)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 79289e32dc [fix](cast) fix wrong result of casting empty string to
array date (#22281)
79289e32dc is described below
commit 79289e32dc0e7e0091301d676209fd0cd822483a
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Sun Jul 30 21:15:03 2023 +0800
[fix](cast) fix wrong result of casting empty string to array date (#22281)
---
be/src/vec/data_types/data_type_time_v2.cpp | 12
be/src/vec/runtime/vdatetime_value.cpp | 6 ++
.../cast_function/test_cast_string_to_array.out | 17 -
.../cast_function/test_cast_string_to_array.groovy | 9 +
4 files changed, 31 insertions(+), 13 deletions(-)
diff --git a/be/src/vec/data_types/data_type_time_v2.cpp
b/be/src/vec/data_types/data_type_time_v2.cpp
index 06275d029c..c4b397f6e7 100644
--- a/be/src/vec/data_types/data_type_time_v2.cpp
+++ b/be/src/vec/data_types/data_type_time_v2.cpp
@@ -67,12 +67,6 @@ void DataTypeDateV2::to_string(const IColumn& column, size_t
row_num, BufferWrit
UInt32 int_val = assert_cast(*ptr).get_element(row_num);
DateV2Value val = binary_cast>(int_val);
-// if this is an invalid date, write nothing(instead of -00-00) to
output string, or else
-// it will cause problem for null DataTypeDateV2 value in cast function,
-// e.g. cast(cast(null_date as char) as date)
-if (!val.is_valid_date()) {
-return;
-}
char buf[64];
char* pos = val.to_string(buf);
@@ -150,12 +144,6 @@ void DataTypeDateTimeV2::to_string(const IColumn& column,
size_t row_num,
UInt64 int_val = assert_cast(*ptr).get_element(row_num);
DateV2Value val =
binary_cast>(int_val);
-// if this is an invalid date, write nothing(instead of -00-00) to
output string, or else
-// it will cause problem for null DataTypeDateV2 value in cast function,
-// e.g. cast(cast(null_date as char) as date)
-if (!val.is_valid_date()) {
-return;
-}
char buf[64];
char* pos = val.to_string(buf, _scale);
diff --git a/be/src/vec/runtime/vdatetime_value.cpp
b/be/src/vec/runtime/vdatetime_value.cpp
index a8db4d6f66..8aaa51a376 100644
--- a/be/src/vec/runtime/vdatetime_value.cpp
+++ b/be/src/vec/runtime/vdatetime_value.cpp
@@ -2406,6 +2406,12 @@ bool DateV2Value::from_date_format_str(const char*
format, int format_len, co
template
int32_t DateV2Value::to_buffer(char* buffer, int scale) const {
+// if this is an invalid date, write nothing(instead of -00-00) to
output string, or else
+// it will cause problem for null DataTypeDateV2 value in cast function,
+// e.g. cast(cast(null_date as char) as date)
+if (!is_valid_date()) {
+return 0;
+}
char* start = buffer;
uint32_t temp;
// Year
diff --git
a/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
b/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
index 1c2a470c90..7a6acbd024 100644
---
a/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
+++
b/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
@@ -30,5 +30,20 @@
[1.340, 2.010, 0.000, 0.000, 0.000]
-- !sql --
-[2022-09-01, -00-00, -00-00]
+[2022-09-01, , ]
+
+-- !sql --
+["2022-09-01", "", ""]
+
+-- !sql --
+[2022-09-01, NULL, NULL]
+
+-- !sql --
+[2022-09-01, , ]
+
+-- !sql --
+["2022-09-01", "", ""]
+
+-- !sql --
+[2022-09-01, NULL, NULL]
diff --git
a/regression-test/suites/query_p0/sql_functions/cast_function/test_cast_string_to_array.groovy
b/regression-test/suites/query_p0/sql_functions/cast_function/test_cast_string_to_array.groovy
index d9c5ee3546..b246fb5534 100644
---
a/regression-test/suites/query_p0/sql_functions/cast_function/test_cast_string_to_array.groovy
+++
b/regression-test/suites/query_p0/sql_functions/cast_function/test_cast_string_to_array.groovy
@@ -41,5 +41,14 @@ suite("test_cast_string_to_array") {
qt_sql """ select cast ("[1,2,3,,,]" as array) """
qt_sql """ select cast ("[a,b,c,,,]" as array) """
qt_sql """ select cast ("[1.34,2.01,,,]" as array) """
+
+sql """ ADMIN SET FRONTEND CONFIG ("enable_date_conversion" = "false"); """
+qt_sql """ select cast ("[2022-09-01,,]" as array) """
+qt_sql """ select cast ("[2022-09-01,,]" as array) """
+qt_sql """ select cast(cast ("[2022-09-01,,]" as array) as
array) """
+
+sql """ ADMIN SET FRONTEND CONFIG ("enable_date_conversion" = "true"); """
qt_sql """ select cast ("[2022-09-01,,]" as array) """
+qt_sql """ select cast ("[2022-09-01,,]" as array) """
+qt_sql """ select cast(cast ("[2022-09-01
[GitHub] [doris] hello-stephen commented on pull request #22367: [opt](datetime) opt datetime parsing performance
hello-stephen commented on PR #22367: URL: https://github.com/apache/doris/pull/22367#issuecomment-1657168762 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.89 seconds stream load tsv: 540 seconds loaded 74807831229 Bytes, about 132 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17169168166 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22372: [enhancement](exchange) ignore overcrowded in brpc used by exchange
hello-stephen commented on PR #22372: URL: https://github.com/apache/doris/pull/22372#issuecomment-1657169978 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.59 seconds stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162112020 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21725: [refactor](load) create segment creator abstraction in rowset writer
github-actions[bot] commented on PR #21725: URL: https://github.com/apache/doris/pull/21725#issuecomment-1657172189 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21725: [refactor](load) create segment creator abstraction in rowset writer
github-actions[bot] commented on PR #21725: URL: https://github.com/apache/doris/pull/21725#issuecomment-1657172176 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [Opt](parquet) opt the performance of date convertion (#22360)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 8d5d49362d [Opt](parquet) opt the performance of date convertion
(#22360)
8d5d49362d is described below
commit 8d5d49362df9b8445b090d095172f49abb217177
Author: HappenLee
AuthorDate: Sun Jul 30 15:54:13 2023 +0800
[Opt](parquet) opt the performance of date convertion (#22360)
before：
```
mysql> select count(l_commitdate) from lineitem;
+-+
| count(l_commitdate) |
+-+
| 600037902 |
+-+
1 row in set (1.61 sec)
```
after:
```
mysql> select count(l_commitdate) from lineitem;
+-+
| count(l_commitdate) |
+-+
| 600037902 |
+-+
1 row in set (0.86 sec)
```
---
be/src/vec/exec/format/parquet/decoder.cpp | 6 ++
be/src/vec/exec/format/parquet/decoder.h| 1 +
be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp | 7 ---
be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp | 6 --
be/src/vec/runtime/vdatetime_value.h| 5 ++---
5 files changed, 17 insertions(+), 8 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/decoder.cpp
b/be/src/vec/exec/format/parquet/decoder.cpp
index a1fc3706fd..539fc04a10 100644
--- a/be/src/vec/exec/format/parquet/decoder.cpp
+++ b/be/src/vec/exec/format/parquet/decoder.cpp
@@ -177,5 +177,11 @@ void Decoder::init(FieldSchema* field_schema,
cctz::time_zone* ctz) {
_decode_params->scale_to_nano_factor = 1000;
}
}
+
+if (_decode_params->ctz) {
+VecDateTimeValue t;
+t.from_unixtime(0, *_decode_params->ctz);
+_decode_params->offset_days = doris::calc_daynr(t.year(), t.month(),
t.day());
+}
}
} // namespace doris::vectorized
diff --git a/be/src/vec/exec/format/parquet/decoder.h
b/be/src/vec/exec/format/parquet/decoder.h
index aacb3730ad..6c1030818c 100644
--- a/be/src/vec/exec/format/parquet/decoder.h
+++ b/be/src/vec/exec/format/parquet/decoder.h
@@ -71,6 +71,7 @@ struct DecodeParams {
static const cctz::time_zone utc0;
// schema.logicalType.TIMESTAMP.isAdjustedToUTC == true, we should set the
time zone
cctz::time_zone* ctz = nullptr;
+size_t offset_days = 0;
int64_t second_mask = 1;
int64_t scale_to_nano_factor = 1;
DecimalScaleParams decimal_scale;
diff --git a/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
b/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
index 887797636e..817b5e7f96 100644
--- a/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
+++ b/be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp
@@ -216,15 +216,16 @@ protected:
size_t data_index = column_data.size();
column_data.resize(data_index + select_vector.num_values() -
select_vector.num_filtered());
size_t dict_index = 0;
+
ColumnSelectVector::DataReadType read_type;
while (size_t run_length =
select_vector.get_next_run(&read_type)) {
switch (read_type) {
case ColumnSelectVector::CONTENT: {
for (size_t i = 0; i < run_length; ++i) {
-int64_t date_value = _dict_items[_indexes[dict_index++]];
+int64_t date_value =
+_dict_items[_indexes[dict_index++]] +
_decode_params->offset_days;
auto& v =
reinterpret_cast(column_data[data_index++]);
-v.from_unixtime(date_value * 24 * 60 * 60,
-*_decode_params->ctz); // day to seconds
+v.get_date_from_daynr(date_value);
if constexpr (std::is_same_v) {
// we should cast to date if using date v1.
v.cast_to_date();
diff --git a/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
b/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
index 72d362fe61..940e70db79 100644
--- a/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
+++ b/be/src/vec/exec/format/parquet/fix_length_plain_decoder.cpp
@@ -248,14 +248,16 @@ Status
FixLengthPlainDecoder::_decode_date(MutableColumnPtr& doris_column,
size_t data_index = column_data.size();
column_data.resize(data_index + select_vector.num_values() -
select_vector.num_filtered());
ColumnSelectVector::DataReadType read_type;
+
while (size_t run_length =
select_vector.get_next_run(&read_type)) {
switch (read_type) {
case ColumnSelectVector::CONTENT: {
for (size_t i = 0; i < run_length; ++i) {
char* buf_start = _data->data + _offset;
-
[GitHub] [doris] yujun777 opened a new pull request, #22378: [fix](tablet clone) fix not add colocate replica and print some logs
yujun777 opened a new pull request, #22378: URL: https://github.com/apache/doris/pull/22378 ## Proposed changes 1. When be reports its tablets to FE, if this be is one of the tablet's colocate backends, then add back it. 2. When chose a dest replica for a version incomplete replica, choose the colocate backend first. 3. Add some logs too. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yujun777 commented on pull request #22378: [fix](tablet clone) fix not add colocate replica and print some logs
yujun777 commented on PR #22378: URL: https://github.com/apache/doris/pull/22378#issuecomment-1657172342 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
github-actions[bot] commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172563 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
github-actions[bot] commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172558 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657172716 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
hello-stephen commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657173806 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.06 seconds stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17167764621 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #21634: [Refactor](load) refactor memtable flush actively
dataroaring merged PR #21634: URL: https://github.com/apache/doris/pull/21634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](load) refactor memtable flush actively (#21634)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ee754307bb [refactor](load) refactor memtable flush actively (#21634)
ee754307bb is described below
commit ee754307bb62f509a929098421039a2d9acae579
Author: HHoflittlefish777 <77738092+hhoflittlefish...@users.noreply.github.com>
AuthorDate: Sun Jul 30 21:31:54 2023 +0800
[refactor](load) refactor memtable flush actively (#21634)
---
be/src/common/config.cpp | 4 +-
be/src/common/config.h | 4 +-
be/src/common/daemon.cpp | 18 +-
be/src/common/daemon.h | 4 +-
be/src/olap/delta_writer.cpp | 6 +-
be/src/olap/delta_writer.h | 2 +
be/src/olap/memtable.cpp | 5 +-
be/src/olap/memtable_memory_limiter.cpp | 222 +++
be/src/olap/memtable_memory_limiter.h| 71
be/src/runtime/exec_env.h| 3 +
be/src/runtime/exec_env_init.cpp | 4 +
be/src/runtime/load_channel.cpp | 13 +-
be/src/runtime/load_channel.h| 64 ++-
be/src/runtime/load_channel_mgr.cpp | 215 ++
be/src/runtime/load_channel_mgr.h| 43 ++---
be/src/runtime/memory/mem_tracker_limiter.cpp| 4 +-
be/src/runtime/tablets_channel.cpp | 85 -
be/src/runtime/tablets_channel.h | 9 +-
be/src/util/doris_metrics.h | 1 +
be/test/olap/memtable_memory_limiter_test.cpp| 182 +++
docs/en/docs/admin-manual/config/be-config.md| 5 +
docs/zh-CN/docs/admin-manual/config/be-config.md | 5 +
22 files changed, 574 insertions(+), 395 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index bbf7c3c6b0..74af2f754a 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -551,8 +551,8 @@ DEFINE_mInt32(memory_maintenance_sleep_time_ms, "100");
// After minor gc, no minor gc during sleep, but full gc is possible.
DEFINE_mInt32(memory_gc_sleep_time_ms, "1000");
-// Sleep time in milliseconds between load channel memory refresh iterations
-DEFINE_mInt64(load_channel_memory_refresh_sleep_time_ms, "100");
+// Sleep time in milliseconds between memtbale flush mgr refresh iterations
+DEFINE_mInt64(memtable_mem_tracker_refresh_interval_ms, "100");
// Alignment
DEFINE_Int32(memory_max_alignment, "16");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 1f196d0904..c6e54d45b7 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -592,8 +592,8 @@ DECLARE_mInt32(memory_maintenance_sleep_time_ms);
// After minor gc, no minor gc during sleep, but full gc is possible.
DECLARE_mInt32(memory_gc_sleep_time_ms);
-// Sleep time in milliseconds between load channel memory refresh iterations
-DECLARE_mInt64(load_channel_memory_refresh_sleep_time_ms);
+// Sleep time in milliseconds between memtbale flush mgr memory refresh
iterations
+DECLARE_mInt64(memtable_mem_tracker_refresh_interval_ms);
// Alignment
DECLARE_Int32(memory_max_alignment);
diff --git a/be/src/common/daemon.cpp b/be/src/common/daemon.cpp
index cfff81088c..e1d6e97bc1 100644
--- a/be/src/common/daemon.cpp
+++ b/be/src/common/daemon.cpp
@@ -40,12 +40,12 @@
#include "common/config.h"
#include "common/logging.h"
#include "common/status.h"
+#include "olap/memtable_memory_limiter.h"
#include "olap/options.h"
#include "olap/storage_engine.h"
#include "olap/tablet_manager.h"
#include "runtime/block_spill_manager.h"
#include "runtime/exec_env.h"
-#include "runtime/load_channel_mgr.h"
#include "runtime/memory/mem_tracker.h"
#include "runtime/memory/mem_tracker_limiter.h"
#include "runtime/task_group/task_group_manager.h"
@@ -276,14 +276,14 @@ void Daemon::memory_gc_thread() {
}
}
-void Daemon::load_channel_tracker_refresh_thread() {
+void Daemon::memtable_memory_limiter_tracker_refresh_thread() {
// Refresh the memory statistics of the load channel tracker more
frequently,
// which helps to accurately control the memory of LoadChannelMgr.
while (!_stop_background_threads_latch.wait_for(
-
std::chrono::milliseconds(config::load_channel_memory_refresh_sleep_time_ms)) &&
+
std::chrono::milliseconds(config::memtable_mem_tracker_refresh_interval_ms)) &&
!k_doris_exit) {
if (ExecEnv::GetInstance()->initialized()) {
-
doris::ExecEnv::GetInstance()->load_channel_mgr()->refresh_mem_tracker();
+
doris::ExecEnv::GetInstance()->memtable_memory_limiter()->refresh_mem_tracker();
}
}
}
@@ -464,9 +464,9 @@ void Daemon::start() {
[GitHub] [doris] kaka11chen opened a new pull request, #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `experimental_
kaka11chen opened a new pull request, #22379: URL: https://github.com/apache/doris/pull/22379 ## Proposed changes ### Issue Sometimes external table file cache statistics in profile are lost when disable `experimental_enable_pipeline_engine`. Because file cache statistics updated to the `RuntimeProfile` in the scanner `close()`, but in some case it will called after`stop_report_thread()` and` send_report(true)` in the end of `PlanFragmentExecutor::open()`. For example: a simple sql without aggregation `select * from table limit 10`. And in this case, scanner `close()` will be called in the `PlanFragmentExecutor::close()`. BTW, some operation will call `_children[0]->close(state) ` in the `open()` functions. So if `_children` is scan node, it will call `close()`. File cache statistics: ``` - FileCache: 0ns - BytesScannedFromCache: 599.00 B - BytesScannedFromRemote: 0.00 - BytesWriteIntoCache: 0.00 - LocalIOUseTimer: 5.771us - NumLocalIOTotal: 1 - NumRemoteIOTotal: 0 - NumSkipCacheIOTotal: 0 - RemoteIOUseTimer: 0ns - WriteCacheIOUseTimer: 0ns ``` ### Resolution This PR use a workaround resolution to update file cache statistics to the `RuntimeProfile` in the `FileScanner::_update_counters_before_close()` which like `NewOlapScanner`. Because I don't know whether we can put `stop_report_thread()` and` send_report(true)` in `PlanFragmentExecutor::close()`. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaka11chen commented on pull request #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `experimental_e
kaka11chen commented on PR #22379: URL: https://github.com/apache/doris/pull/22379#issuecomment-1657178168 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22331: [fix](complex_type) throw error when reading complex types in broker/stream load
hello-stephen commented on PR #22331: URL: https://github.com/apache/doris/pull/22331#issuecomment-1657178298 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.03 seconds stream load tsv: 505 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17162185145 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `exper
github-actions[bot] commented on PR #22379: URL: https://github.com/apache/doris/pull/22379#issuecomment-1657179951 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22378: [fix](tablet clone) fix not add colocate replica and print some logs
hello-stephen commented on PR #22378: URL: https://github.com/apache/doris/pull/22378#issuecomment-1657179987 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.34 seconds stream load tsv: 507 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17167544308 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
hello-stephen commented on PR #22159: URL: https://github.com/apache/doris/pull/22159#issuecomment-1657181347 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.5 seconds stream load tsv: 502 seconds loaded 74807831229 Bytes, about 142 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17169210993 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #22364: [FIX](decimal)fix decimal precision
amorynan commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657184047 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22370: [typo](docs) add show collation doc description and example
github-actions[bot] commented on PR #22370: URL: https://github.com/apache/doris/pull/22370#issuecomment-1657184894 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22370: [typo](docs) add show collation doc description and example
github-actions[bot] commented on PR #22370: URL: https://github.com/apache/doris/pull/22370#issuecomment-1657184904 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `experimenta
hello-stephen commented on PR #22379: URL: https://github.com/apache/doris/pull/22379#issuecomment-1657185405 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.21 seconds stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17167842943 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22364: [FIX](decimal)fix decimal precision
github-actions[bot] commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657185543 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #22347: [feature](hive)append support for struct and map column type on textfile format of hive table
morningman commented on PR #22347: URL: https://github.com/apache/doris/pull/22347#issuecomment-1657186263 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
amorynan commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657186492 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #22342: [fix](multi-catalog)compatible with hdfs HA empty prefix
morningman merged PR #22342: URL: https://github.com/apache/doris/pull/22342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](multi-catalog)compatible with hdfs HA empty prefix (#22342)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f87f29e1ab [fix](multi-catalog)compatible with hdfs HA empty prefix
(#22342)
f87f29e1ab is described below
commit f87f29e1abf7c5d6fd20ce25f3e6b3f1e61938c2
Author: slothever <18522955+w...@users.noreply.github.com>
AuthorDate: Sun Jul 30 22:21:14 2023 +0800
[fix](multi-catalog)compatible with hdfs HA empty prefix (#22342)
compatible with hdfs HA empty prefix
for example: ’hdfs:///‘ will be replaced to ’hdfs://ha-nameservice/‘
---
.../org/apache/doris/catalog/HdfsResource.java | 1 +
.../java/org/apache/doris/common/util/S3Util.java | 35 +++---
.../doris/datasource/hive/HiveMetaStoreCache.java | 7 +++--
.../planner/external/iceberg/IcebergScanNode.java | 7 +++--
.../planner/external/iceberg/IcebergSplit.java | 6 +++-
5 files changed, 45 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsResource.java
index cdfb169590..2b50ec63b6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsResource.java
@@ -53,6 +53,7 @@ public class HdfsResource extends Resource {
public static String HADOOP_SHORT_CIRCUIT = "dfs.client.read.shortcircuit";
public static String HADOOP_SOCKET_PATH = "dfs.domain.socket.path";
public static String DSF_NAMESERVICES = "dfs.nameservices";
+public static final String HDFS_PREFIX = "hdfs://";
@SerializedName(value = "properties")
private Map properties;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/S3Util.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/S3Util.java
index 64c897c306..a47d838537 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/S3Util.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/S3Util.java
@@ -17,9 +17,11 @@
package org.apache.doris.common.util;
+import org.apache.doris.catalog.HdfsResource;
import org.apache.doris.common.FeConstants;
import org.apache.doris.datasource.credentials.CloudCredential;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.fs.Path;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -38,7 +40,9 @@ import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import java.net.URI;
+import java.net.URISyntaxException;
import java.time.Duration;
+import java.util.Map;
public class S3Util {
private static final Logger LOG = LogManager.getLogger(S3Util.class);
@@ -63,7 +67,7 @@ public class S3Util {
* @param location origin location
* @return metadata location path. just convert when storage is compatible
with s3 client.
*/
-public static String convertToS3IfNecessary(String location) {
+public static String convertToS3IfNecessary(String location, Map props) {
LOG.debug("try convert location to s3 prefix: " + location);
if (isObjStorageUseS3Client(location)) {
int pos = location.indexOf("://");
@@ -72,7 +76,30 @@ public class S3Util {
}
return "s3" + location.substring(pos);
}
-return location;
+return normalizedLocation(location, props);
+}
+
+private static String normalizedLocation(String location, Map props) {
+try {
+URI normalizedUri = new URI(location);
+if (StringUtils.isEmpty(normalizedUri.getHost()) &&
location.startsWith(HdfsResource.HDFS_PREFIX)) {
+// Need add hdfs host to location
+String host = props.get(HdfsResource.DSF_NAMESERVICES);
+if (StringUtils.isNotEmpty(host)) {
+// Replace 'hdfs://' to 'hdfs://name_service', for
example: hdfs:///abc to hdfs://name_service/abc
+return location.replace(HdfsResource.HDFS_PREFIX,
HdfsResource.HDFS_PREFIX + host);
+} else {
+// If no hadoop HA config
+if (location.startsWith(HdfsResource.HDFS_PREFIX + '/')) {
+// Do not support hdfs:///location
+throw new RuntimeException("Invalid location with
empty host: " + location);
+}
+}
+}
+return location;
+} catch (URISyntaxException e) {
+throw new RuntimeException(e.getMessage(), e);
+}
}
/**
@@ -80,7 +107,7 @@ public class S3Util {
* @param location origin split path
* @return BE scan range path
*/
-public static Path toScanRangeLocation(String location) {
+publi
[GitHub] [doris] morningman commented on a diff in pull request #22272: [fix](rest)check response code when get image
morningman commented on code in PR #22272:
URL: https://github.com/apache/doris/pull/22272#discussion_r1278260342
##
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##
@@ -49,6 +49,7 @@
*/
public class HttpUtils {
static final int REQUEST_SUCCESS_CODE = 0;
+static final int DEFAULT_TIME_OUT = 2000;
Review Comment:
Add unit of this time, eg, `DEFAULT_TIME_OUT_MS`
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -1642,13 +1640,19 @@ private void getNewImage(HostInfo helperNode) throws
IOException {
try {
String hostPort =
NetUtils.getHostPortInAccessibleFormat(helperNode.getHost(), Config.http_port);
String infoUrl = "http://"; + hostPort + "/info";
-StorageInfo info = getStorageInfo(infoUrl);
+ResponseBody responseBody = MetaHelper
+.doGet(infoUrl, HTTP_TIMEOUT_SECOND * 1000,
StorageInfo.class);
Review Comment:
Is this should be `StorageInfoV2`?
##
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##
@@ -74,12 +75,16 @@ static String concatUrl(Pair ipPort,
String path, Map headers) throws
IOException {
+public static String doGet(String url, Map headers, int
timeout) throws IOException {
Review Comment:
```suggestion
public static String doGet(String url, Map headers, int
timeoutMs) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22375: [FIX](complex-type)fix complex type nested col_const
github-actions[bot] commented on PR #22375: URL: https://github.com/apache/doris/pull/22375#issuecomment-1657188125 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] fsilent opened a new issue, #22380: [Bug] add md5 check to prevent image files from being damaged when network error packets occur
fsilent opened a new issue, #22380: URL: https://github.com/apache/doris/issues/22380 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master, 1.2-LTS ### What's Wrong? image files damaged when network error packets occur ### What You Expected? master fe send image file to none master correctly, if has network error should not show execute success ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22364: [FIX](decimal)fix decimal precision
hello-stephen commented on PR #22364: URL: https://github.com/apache/doris/pull/22364#issuecomment-1657191360 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.24 seconds stream load tsv: 504 seconds loaded 74807831229 Bytes, about 141 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162531758 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `exper
github-actions[bot] commented on PR #22379: URL: https://github.com/apache/doris/pull/22379#issuecomment-1657192030 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22379: [Fix](multi_catalog) Fix the problem that sometimes external table file cache statistics in profile are lost when disable `exper
github-actions[bot] commented on PR #22379: URL: https://github.com/apache/doris/pull/22379#issuecomment-1657192041 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #22323: [fix](ipv6)Remove restrictions from IPv4 when add backend
morningman merged PR #22323: URL: https://github.com/apache/doris/pull/22323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](ipv6)Remove restrictions from IPv4 when add backend (#22323)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0e7f63f5f6 [fix](ipv6)Remove restrictions from IPv4 when add backend
(#22323)
0e7f63f5f6 is described below
commit 0e7f63f5f67587a5fc99c39cd3ec518a4b57050c
Author: zhangdong <493738...@qq.com>
AuthorDate: Sun Jul 30 22:47:24 2023 +0800
[fix](ipv6)Remove restrictions from IPv4 when add backend (#22323)
When adding be, it is required to have only one colon, otherwise an error
will be reported. However, ipv6 has many colons
```
String[] pair = hostPort.split(":");
if (pair.length != 2) {
throw new AnalysisException("Invalid host port: " + hostPort);
}
```
---
.../org/apache/doris/common/util/NetUtils.java | 14 +-
.../org/apache/doris/system/SystemInfoService.java | 5
.../apache/doris/system/SystemInfoServiceTest.java | 32 ++
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/NetUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/NetUtils.java
index 3470cea203..334dd11564 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/NetUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/NetUtils.java
@@ -137,16 +137,16 @@ public class NetUtils {
}
public static SystemInfoService.HostInfo
resolveHostInfoFromHostPort(String hostPort) throws AnalysisException {
+String[] pair;
if (hostPort.charAt(0) == '[') {
-String[] pair = hostPort.substring(1).split("]:");
-return new SystemInfoService.HostInfo(pair[0],
Integer.valueOf(pair[1]));
+pair = hostPort.substring(1).split("]:");
} else {
-String[] pair = hostPort.split(":");
-if (pair.length != 2) {
-throw new AnalysisException("invalid host port: " + hostPort);
-}
-return new SystemInfoService.HostInfo(pair[0],
Integer.valueOf(pair[1]));
+pair = hostPort.split(":");
+}
+if (pair.length != 2) {
+throw new AnalysisException("invalid host port: " + hostPort);
}
+return new SystemInfoService.HostInfo(pair[0],
Integer.valueOf(pair[1]));
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
b/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
index 0f4639a13f..6523f9f0f0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
@@ -718,11 +718,6 @@ public class SystemInfoService {
throw new AnalysisException("Invalid host port: " + hostPort);
}
-String[] pair = hostPort.split(":");
-if (pair.length != 2) {
-throw new AnalysisException("Invalid host port: " + hostPort);
-}
-
HostInfo hostInfo = NetUtils.resolveHostInfoFromHostPort(hostPort);
String host = hostInfo.getHost();
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
b/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
index 172fbd5594..d207e0ce2a 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/system/SystemInfoServiceTest.java
@@ -20,9 +20,11 @@ package org.apache.doris.system;
import org.apache.doris.catalog.DiskInfo;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.ReplicaAllocation;
+import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.meta.MetaContext;
import org.apache.doris.resource.Tag;
+import org.apache.doris.system.SystemInfoService.HostInfo;
import org.apache.doris.thrift.TStorageMedium;
import com.google.common.collect.ImmutableMap;
@@ -57,6 +59,35 @@ public class SystemInfoServiceTest {
infoService.addBackend(backend);
}
+@Test
+public void testGetHostAndPort() {
+String ipv4 = "192.168.1.2:9050";
+String ipv6 = "[fe80::5054:ff:fec9:dee0]:9050";
+String ipv6Error = "fe80::5054:ff:fec9:dee0:9050";
+try {
+HostInfo hostAndPort = SystemInfoService.getHostAndPort(ipv4);
+Assert.assertEquals("192.168.1.2", hostAndPort.getHost());
+Assert.assertEquals(9050, hostAndPort.getPort());
+} catch (AnalysisException e) {
+e.printStackTrace();
+Assert.fail();
+}
+try {
+HostInfo hostAndPort = SystemInfoService.getHostAndPort(ipv6);
+Assert.assertEquals("fe80::5054:ff:fec9:dee0",
hostAndPort.getHost());
+As
[GitHub] [doris] kaka11chen opened a new pull request, #22381: [Opt](orc-reader) Opt the performance of date conversion in orc reader.
kaka11chen opened a new pull request, #22381: URL: https://github.com/apache/doris/pull/22381 ## Proposed changes Opt the performance of date conversion in orc reader. ``` mysql> select count(l_commitdate) from lineitem; +-+ | count(l_commitdate) | +-+ | 600037902 | +-+ 1 row in set (3.60 sec) mysql> select count(l_commitdate) from lineitem; +-+ | count(l_commitdate) | +-+ | 600037902 | +-+ 1 row in set (2.01 sec) ``` Similar with https://github.com/apache/doris/pull/22360 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kaka11chen commented on pull request #22381: [Opt](orc-reader) Opt the performance of date conversion in orc reader.
kaka11chen commented on PR #22381: URL: https://github.com/apache/doris/pull/22381#issuecomment-1657192506 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
