[GitHub] [doris] lexluo09 opened a new issue, #16984: [Bug] lateral view explode_split appear incorrect results
lexluo09 opened a new issue, #16984: URL: https://github.com/apache/doris/issues/16984 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris 1.1.3 ### What's Wrong? SELECT a663_single as a663,COUNT(1) FROM ( SELECT a663_single,a1 FROM ( SELECT a663,a1 FROM table_a WHERE (dayno >= '2023-02-12') AND (dayno <= '2023-02-12') AND ( ( id in ( select id from table_a WHERE ( ( ( a955 = '是' )) AND ( ( a444 = '' and a444 IS NOT NULL) ) ) AND (dayno= '2023-02-12') )) ) AND NOT ( ( a663 is null ) ) ) tab1 lateral view explode_split(a663, '#') tmp1 as a663_single ) tab2 GROUP BY a663_single LIMIT 1000  ### What You Expected?  ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 closed pull request #16893: [fix](docs)fix FE initialize failure caused by docs grouping.md
TangSiyang2001 closed pull request #16893: [fix](docs)fix FE initialize failure caused by docs grouping.md URL: https://github.com/apache/doris/pull/16893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #16985: [fix](fe)bucket shuffle join is not recognized if the first table is a subquery
starocean999 opened a new pull request, #16985: URL: https://github.com/apache/doris/pull/16985 # Proposed changes Issue Number: close #xxx ## Problem summary consider sql select * from (select * from test_1) a inner join (select * from test_2) b on **a.id** = b.id inner join (select * from test_3) c on **a.id** = c.id Because a.id is from a subquery, to find its source table, need use function getSrcSlotRef(). ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] catpineapple commented on pull request #16759: [fix](planner)fix multi partition support datetime column
catpineapple commented on PR #16759: URL: https://github.com/apache/doris/pull/16759#issuecomment-1438019088 run clickbench -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #16666: [ehancement](planner) Support filter the output of set operation node
Kikyou1997 commented on PR #1: URL: https://github.com/apache/doris/pull/1#issuecomment-1438021887 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
TangSiyang2001 commented on PR #15339: URL: https://github.com/apache/doris/pull/15339#issuecomment-1438023384 Regression .out file has not been modified, waiting for ipv6 bug in `master` fixed and then regression test is able to run on the local env. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
github-actions[bot] commented on PR #15339: URL: https://github.com/apache/doris/pull/15339#issuecomment-1438024780 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
platoneko commented on PR #16803: URL: https://github.com/apache/doris/pull/16803#issuecomment-1438025120 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on pull request #16889: [fix](replica) Fix inconsistent replica id between BE and FE in corner case of tablet rebalance
platoneko commented on PR #16889: URL: https://github.com/apache/doris/pull/16889#issuecomment-1438025644 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
freemandealer commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438027521 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16941: (feature)[DOE]Support array for Doris on ES
github-actions[bot] commented on PR #16941: URL: https://github.com/apache/doris/pull/16941#issuecomment-1438029077 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] qidaye commented on a diff in pull request #16941: (feature)[DOE]Support array for Doris on ES
qidaye commented on code in PR #16941:
URL: https://github.com/apache/doris/pull/16941#discussion_r1112707454
##
be/src/exec/es/es_scroll_parser.cpp:
##
@@ -519,7 +521,178 @@ Status ScrollParser::fill_columns(const TupleDescriptor*
tuple_desc,
}
break;
}
+case TYPE_ARRAY: {
+vectorized::Array array;
+const auto& sub_type =
tuple_desc->slots()[i]->type().children[0].type;
+for (auto& sub_col : col.GetArray()) {
+switch (sub_type) {
+case TYPE_CHAR:
+case TYPE_VARCHAR:
+case TYPE_STRING: {
+std::string val;
+if (pure_doc_value) {
+if (!sub_col[0].IsString()) {
+val = json_value_to_string(sub_col[0]);
+} else {
+val = sub_col[0].GetString();
+}
+} else {
+RETURN_ERROR_IF_COL_IS_ARRAY(sub_col, type);
+if (!sub_col.IsString()) {
+val = json_value_to_string(sub_col);
+} else {
+val = sub_col.GetString();
+}
+}
+array.push_back(val);
+break;
+}
+case TYPE_TINYINT: {
+int8_t val;
+RETURN_IF_ERROR(get_int_value(sub_col, sub_type,
&val, pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_SMALLINT: {
+int16_t val;
+RETURN_IF_ERROR(
+get_int_value(sub_col, sub_type, &val,
pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_INT: {
+int32 val;
+RETURN_IF_ERROR(get_int_value(sub_col, sub_type,
&val, pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_BIGINT: {
+int64_t val;
+RETURN_IF_ERROR(
+get_int_value(sub_col, sub_type, &val,
pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_LARGEINT: {
+__int128 val;
+RETURN_IF_ERROR(
+get_int_value<__int128>(sub_col, sub_type, &val,
pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_FLOAT: {
+float val;
+RETURN_IF_ERROR(
+get_float_value(sub_col, sub_type, &val,
pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_DOUBLE: {
+double val;
+RETURN_IF_ERROR(
+get_float_value(sub_col, sub_type, &val,
pure_doc_value));
+array.push_back(val);
+break;
+}
+case TYPE_BOOLEAN: {
+if (sub_col.IsBool()) {
+array.push_back(sub_col.GetBool());
+break;
+}
+
+if (sub_col.IsNumber()) {
+array.push_back(sub_col.GetInt());
+break;
+}
+
+bool is_nested_str = false;
+if (pure_doc_value && sub_col.IsArray() &&
sub_col[0].IsBool()) {
+array.push_back(sub_col[0].GetBool());
+break;
+} else if (pure_doc_value && sub_col.IsArray() &&
sub_col[0].IsString()) {
+is_nested_str = true;
+} else if (pure_doc_value && sub_col.IsArray()) {
+return Status::InternalError(ERROR_INVALID_COL_DATA,
"BOOLEAN");
+}
+
+const rapidjson::Value& str_col = is_nested_str ?
sub_col[0] : sub_col;
+
+const std::string& val = str_col.GetString();
+size_t val_size = str_col.GetStringLength();
+StringParser::ParseResult result;
+bool b = StringParser::string_to_bool(val.c_str(),
val_size, &result);
+RETURN_ERROR_IF_PARSING_FAILED(result, str_col, type);
+array.push_back(b);
+break;
+}
+// date/datetime v2 is the default type for catalog table,
+// see https://github.com/apache/doris/pull/16304
+
[GitHub] [doris] github-actions[bot] commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
github-actions[bot] commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438030229 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
github-actions[bot] commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1438031827 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #16953: [Function](vec) use const column to opt function current_time()
HappenLee merged PR #16953: URL: https://github.com/apache/doris/pull/16953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (cc839aead7 -> f37da6e789)
This is an automated email from the ASF dual-hosted git repository. lihaopeng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cc839aead7 [fix](Nereids) fix signatures of some window functions (#16871) add f37da6e789 [Function](vec) use const column to opt function current_time() (#16953) No new revisions were added by this update. Summary of changes: .../vec/functions/function_date_or_datetime_computation.h | 14 +- 1 file changed, 5 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16731: [WIP] segcompaction feat. verticalcompaction
github-actions[bot] commented on code in PR #16731:
URL: https://github.com/apache/doris/pull/16731#discussion_r1112715214
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -116,17 +118,17 @@ Status BetaRowsetWriter::add_block(const
vectorized::Block* block) {
return _add_block(block, &_segment_writer);
}
-vectorized::VMergeIterator* BetaRowsetWriter::_get_segcompaction_reader(
-SegCompactionCandidatesSharedPtr segments, std::shared_ptr
schema,
-OlapReaderStatistics* stat, uint64_t* merged_row_stat) {
+std::unique_ptr
BetaRowsetWriter::_get_segcompaction_reader(
Review Comment:
warning: method '_get_segcompaction_reader' can be made const
[readability-make-member-function-const]
be/src/olap/rowset/beta_rowset_writer.cpp:122:
```diff
- std::shared_ptr schema, OlapReaderStatistics* stat,
uint64_t* merged_row_stat, vectorized::RowSourcesBuffer& row_sources_buf, bool
is_key, std::vector& return_columns) {
+ std::shared_ptr schema, OlapReaderStatistics* stat,
uint64_t* merged_row_stat, vectorized::RowSourcesBuffer& row_sources_buf, bool
is_key, std::vector& return_columns) const { const
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16978: [fix](struct-type) fix be core when load array orc file
github-actions[bot] commented on PR #16978: URL: https://github.com/apache/doris/pull/16978#issuecomment-1438046076 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1438049315 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
xinyiZzz commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438049197 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #16987: [improvement](memory) Support GC segment cache, when memory insufficient
xinyiZzz opened a new pull request, #16987: URL: https://github.com/apache/doris/pull/16987 # Proposed changes Issue Number: close #xxx ## Problem summary 1. fix segment cache memory tracker statistics 2. support GC ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16978: [fix](struct-type) fix be core when load array orc file
github-actions[bot] commented on PR #16978: URL: https://github.com/apache/doris/pull/16978#issuecomment-1438049269 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
github-actions[bot] commented on code in PR #16987:
URL: https://github.com/apache/doris/pull/16987#discussion_r1112731759
##
be/src/olap/rowset/segment_v2/segment.h:
##
@@ -116,6 +116,8 @@ class Segment : public
std::enable_shared_from_this {
io::FileReaderSPtr file_reader() { return _file_reader; }
+int64_t meta_mem_usage() { return _meta_mem_usage; }
Review Comment:
warning: method 'meta_mem_usage' can be made const
[readability-make-member-function-const]
```suggestion
int64_t meta_mem_usage() const { return _meta_mem_usage; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
zhannngchen commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1438060898 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16935: [Fix](function)fix datatime-diff function's overflow
hello-stephen commented on PR #16935: URL: https://github.com/apache/doris/pull/16935#issuecomment-1438075013 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.01 seconds stream load tsv: 473 seconds loaded 74807831229 Bytes, about 150 MB/s stream load json: 38 seconds loaded 2358488459 Bytes, about 59 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230221084944_clickbench_pr_100910.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16607: [fix](engine_clone_task) disable compact when tablet version discontinuous to avoid engine clone task failure
github-actions[bot] commented on PR #16607: URL: https://github.com/apache/doris/pull/16607#issuecomment-1438076264 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16854: [fix](planner) fix bug in agg on constant column (#16442)
morrySnow merged PR #16854: URL: https://github.com/apache/doris/pull/16854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [fix](planner) fix bug in agg on constant column (#16442) (#16854)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a commit to branch branch-1.2-lts in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-1.2-lts by this push: new dbb6579e88 [fix](planner) fix bug in agg on constant column (#16442) (#16854) dbb6579e88 is described below commit dbb6579e88cd4f4dded739f4b3a7759d51ea8796 Author: minghong AuthorDate: Tue Feb 21 16:55:31 2023 +0800 [fix](planner) fix bug in agg on constant column (#16442) (#16854) For performance reason, we want to remove constant column from groupingExprs. For example: `select sum(T.A) from T group by T.B, 'xyz'` is equivalent to `select sum(T.A) from T group by T.B` We can remove constant column `abc` from groupingExprs. But there is an exception when all groupingExpr are constant For example: sql1: `select 'abc' from t group by 'abc'` is not equivalent to sql2: `select 'abc' from t` sql3: `select 'abc', sum(a) from t group by 'abc'` is not equivalent to sql4: `select 1, sum(a) from t` (when t is empty, sql3 returns 0 tuple, sql4 return 1 tuple) We need to keep some constant columns if all groupingExpr are constant. Consider sql5 `select a from (select "abc" as a, 'def' as b) T group by b, a;` if the constant column `a` is in select list, this column should not be removed. sql5 is transformed to sql6 `select a from (select "abc" as a, 'def' as b) T group by a;` (cherry picked from master #16442) (cherry picked from commit a2b9b9edd72d70fed4cd76c571ee5b736493e3a7) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #16340: [feature](mysql) Support secure MySQL connection to FE
Yukang-Lian commented on PR #16340: URL: https://github.com/apache/doris/pull/16340#issuecomment-1438089197 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
xinyiZzz commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438101582 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
github-actions[bot] commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438110255 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W commented on pull request #16977: [fix](alter inverted index) fix write edit log in replaymodifyTableAddOrDropInvertedIndices function
Tanya-W commented on PR #16977: URL: https://github.com/apache/doris/pull/16977#issuecomment-1438144228 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #16844: [refactor](Nereids) let type coercion same with legacy planner
morrySnow commented on PR #16844: URL: https://github.com/apache/doris/pull/16844#issuecomment-1438154517 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #16890: [fix](planner)create view init bug
morrySnow commented on PR #16890: URL: https://github.com/apache/doris/pull/16890#issuecomment-1438155266 run compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16958: [BugFix](Jdbc Catalog) Fix null pointer exception in JdbcExecutor
github-actions[bot] commented on PR #16958: URL: https://github.com/apache/doris/pull/16958#issuecomment-1438164014 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] JackDrogon opened a new issue, #16988: [Enhancement] improve fe meta serialization compatibility
JackDrogon opened a new issue, #16988: URL: https://github.com/apache/doris/issues/16988 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Doris' fe implements serialization itself in many places, mixing sequential writing of data and Gson. This solution makes it extremely difficult to guarantee the backward and forward compatibility of metadata after adding features. ### Solution Replace the whole serialization scheme of metadata with Gson and Protobuffer (Protostuff). This requires a near-irreversible upgrade (if only a few minor releases apart you can consider generating multiple serialization sequences at the same time to preserve the previous serialization). ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16384: [regression-test](fuzzy) fuzzy session variable batch_size
yiguolei merged PR #16384: URL: https://github.com/apache/doris/pull/16384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression-test](fuzzy) fuzzy session variable batch_size (#16384)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ed05f3b480 [regression-test](fuzzy) fuzzy session variable batch_size
(#16384)
ed05f3b480 is described below
commit ed05f3b4804788c6364a1030a20e61d747d8c07e
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Tue Feb 21 17:53:19 2023 +0800
[regression-test](fuzzy) fuzzy session variable batch_size (#16384)
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 8
.../src/main/java/org/apache/doris/qe/SessionVariable.java | 10 +-
regression-test/pipeline/p0/conf/fe.conf | 1 +
regression-test/pipeline/p1/conf/fe.conf | 1 +
4 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 1263c80593..a9925f2c52 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1910,6 +1910,14 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static long external_cache_expire_time_minutes_after_access = 24 *
60; // 1 day
+/**
+ * Github workflow test type, for setting some session variables
+ * only for certain test type. E.g. only settting batch_size to small
+ * value for p0.
+ */
+@ConfField(mutable = true, masterOnly = false)
+public static String fuzzy_test_type = "";
+
/**
* Set session variables randomly to check more issues in github workflow
*/
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index d5a6c7e0f7..dba1d1d65f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -406,7 +406,7 @@ public class SessionVariable implements Serializable,
Writable {
public int codegenLevel = 0;
// 4096 minus 16 + 16 bytes padding that in padding pod array
-@VariableMgr.VarAttr(name = BATCH_SIZE)
+@VariableMgr.VarAttr(name = BATCH_SIZE, fuzzy = true)
public int batchSize = 4064;
@VariableMgr.VarAttr(name = DISABLE_STREAMING_PREAGGREGATIONS, fuzzy =
true)
@@ -750,6 +750,14 @@ public class SessionVariable implements Serializable,
Writable {
// this.enableTwoPhaseReadOpt = true;
}
+if (Config.fuzzy_test_type.equals("p0")) {
+if (Config.pull_request_id % 2 == 1) {
+this.batchSize = 4064;
+} else {
+this.batchSize = 50;
+}
+}
+
// set random 1, 10, 100, 1000, 1
this.topnOptLimitThreshold = 10 ^ (random.nextInt(5));
}
diff --git a/regression-test/pipeline/p0/conf/fe.conf
b/regression-test/pipeline/p0/conf/fe.conf
index d979744153..538b18bf4b 100644
--- a/regression-test/pipeline/p0/conf/fe.conf
+++ b/regression-test/pipeline/p0/conf/fe.conf
@@ -68,4 +68,5 @@ sys_log_verbose_modules = org.apache.doris
enable_outfile_to_local = true
tablet_create_timeout_second=20
remote_fragment_exec_timeout_ms=6
+fuzzy_test_type=p0
use_fuzzy_session_variable=true
diff --git a/regression-test/pipeline/p1/conf/fe.conf
b/regression-test/pipeline/p1/conf/fe.conf
index 8a6749c90d..6f737da817 100644
--- a/regression-test/pipeline/p1/conf/fe.conf
+++ b/regression-test/pipeline/p1/conf/fe.conf
@@ -69,4 +69,5 @@ priority_networks=172.19.0.0/24
enable_outfile_to_local = true
tablet_create_timeout_second=20
remote_fragment_exec_timeout_ms=6
+fuzzy_test_type=p1
use_fuzzy_session_variable=true
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on pull request #16889: [fix](replica) Fix inconsistent replica id between BE and FE in corner case of tablet rebalance
platoneko commented on PR #16889: URL: https://github.com/apache/doris/pull/16889#issuecomment-1438189360 build clickbench -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
github-actions[bot] commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438197610 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #16989: [fix](memory) fix memory GC details and join probe catch bad_alloc
xinyiZzz opened a new pull request, #16989: URL: https://github.com/apache/doris/pull/16989 # Proposed changes Issue Number: close #xxx ## Problem summary 1. Fix Redhat 4.x OS `/proc/meminfo` has no `MemAvailable`, disable `MemAvailable` to control memory. 2. `vm_rss_str` and `mem_available_str` recorded when gc is triggered, to avoid memory changes during gc and cause inaccurate logs. 3. join probe catch bad_alloc, this may apply for 64G memory at a time, avoid OOM. 4. Modify document `doris_be_all_segments_num` and `doris_be_all_rowsets_num` names. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on pull request #16563: [fix](fold_constant) add defer in fold constant to close
zhangstar333 commented on PR #16563: URL: https://github.com/apache/doris/pull/16563#issuecomment-1438206557 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
xinyiZzz commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438207305 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #16989: [fix](memory) fix memory GC details and join probe catch bad_alloc
xinyiZzz commented on PR #16989: URL: https://github.com/apache/doris/pull/16989#issuecomment-1438207478 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16989: [fix](memory) fix memory GC details and join probe catch bad_alloc
github-actions[bot] commented on PR #16989: URL: https://github.com/apache/doris/pull/16989#issuecomment-1438208917 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16563: [fix](fold_constant) add defer in fold constant to close
github-actions[bot] commented on PR #16563: URL: https://github.com/apache/doris/pull/16563#issuecomment-1438212099 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16977: [fix](alter inverted index) fix write edit log in replaymodifyTableAddOrDropInvertedIndices function
hello-stephen commented on PR #16977: URL: https://github.com/apache/doris/pull/16977#issuecomment-1438219428 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.12 seconds stream load tsv: 464 seconds loaded 74807831229 Bytes, about 153 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230221101955_clickbench_pr_101022.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yulei-Yang opened a new pull request, #16990: [Improvement](meta) support return total statistics of all databases
Yulei-Yang opened a new pull request, #16990: URL: https://github.com/apache/doris/pull/16990 # Proposed changes currently, show proc jobs commamd can only used on a specific database, if a user want to see overall data of the whole cluster, he has to look into every database and sum them up, it's troublesome. now he can achieve it simply by giving a -1 as dbId. mysql> show proc '/jobs/-1'; +---+-+-+--+---+---+ | JobType | Pending | Running | Finished | Cancelled | Total | +---+-+-+--+---+---+ | load | 0 | 0 | 0| 0 | 0 | | delete| 0 | 0 | 0| 0 | 0 | | rollup| 0 | 0 | 0| 0 | 0 | | schema_change | 0 | 0 | 0| 0 | 0 | | export| 0 | 0 | 0| 0 | 0 | +---+-+-+--+---+---+ Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [x] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16982: [Thirdparty](clucene) update clucene to v2.4.6 to fix bthread/pthread context bug
github-actions[bot] commented on code in PR #16982:
URL: https://github.com/apache/doris/pull/16982#discussion_r1112883828
##
be/src/olap/column_predicate.h:
##
@@ -139,6 +139,13 @@ class ColumnPredicate {
"Not Implemented evaluate with inverted index, please check
the predicate");
}
+//evaluate range predicate on inverted
+virtual Status evaluate(const Schema& schema, InvertedIndexIterator*
iterator,
Review Comment:
warning: class member cannot be redeclared [clang-diagnostic-error]
```cpp
virtual Status evaluate(const Schema& schema, InvertedIndexIterator*
iterator,
^
```
**be/src/olap/column_predicate.h:135:** previous definition is here
```cpp
virtual Status evaluate(const Schema& schema, InvertedIndexIterator*
iterator,
^
```
##
be/src/olap/rowset/segment_v2/inverted_index_reader.h:
##
@@ -61,6 +62,148 @@ enum class InvertedIndexQueryType {
MATCH_PHRASE_QUERY = 7,
};
+//template
+template
+class InvertedIndexQueryRange {
+public:
+//using CppType = typename
PredicatePrimitiveTypeTraits::PredicateFieldType;
+using CppType = typename CppTypeTraits::CppType;
+
+InvertedIndexQueryRange(const TypeInfo* type_info);
+//static Status add_range(InvertedIndexQueryRange* range,
InvertedIndexQueryOp op, CppType value);
+Status add_range(InvertedIndexQueryOp op, CppType value);
+CppType lower_value() { return _low_value; }
+CppType upper_value() { return _high_value; }
+
+private:
+//std::string _low_value;
+//std::string _high_value;
+CppType _low_value;
+CppType _high_value;
+InvertedIndexQueryOp _low_op;
+InvertedIndexQueryOp _high_op;
+const TypeInfo* _type_info {};
+const KeyCoder* _value_key_coder {};
+};
+
+using InvertedIndexQueryRangeType =
+std::variant,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange>;
+/*using InvertedIndexQueryRangeType =
+std::variant,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange,
+ InvertedIndexQueryRange>;*/
+
+template
+static Status add_range(InvertedIndexQueryRange query_range,
std::string& value,
+InvertedIndexQueryOp query_op) {
+using CppType = typename CppTypeTraits::CppType;
+CppType tmp;
+FieldTypeTraits::from_string(&tmp, value, 0, 0);
+query_range.add_range(query_op, tmp);
+}
+
+static Status make_inverted_index_query_range(const TypeInfo* type_info,
FieldType ft, InvertedIndexQueryRangeType& ret) {
+switch (ft) {
+case OLAP_FIELD_TYPE_DATETIME: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_DATE: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_DATETIMEV2: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_DATEV2: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_TINYINT: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_SMALLINT: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_UNSIGNED_INT: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_INT: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_LARGEINT: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_DECIMAL: {
+ret = InvertedIndexQueryRange(type_info);
+break;
+}
+case OLAP_FIELD_TYPE_DECIMAL32: {
+ret = InvertedIn
[GitHub] [doris] zhangstar333 commented on pull request #16929: [vectorized](jdbc) fix jdbc connect sql server error
zhangstar333 commented on PR #16929: URL: https://github.com/apache/doris/pull/16929#issuecomment-1438268036 run build all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
freemandealer commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438272778 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16759: [fix](planner)fix multi partition support datetime column
github-actions[bot] commented on PR #16759: URL: https://github.com/apache/doris/pull/16759#issuecomment-1438273274 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16759: [fix](planner)fix multi partition support datetime column
github-actions[bot] commented on PR #16759: URL: https://github.com/apache/doris/pull/16759#issuecomment-1438273223 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] DongLiang-0 opened a new pull request, #105: fix decimal(65,18) loss of precision
DongLiang-0 opened a new pull request, #105: URL: https://github.com/apache/doris-flink-connector/pull/105 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ZhangGuoqiang666 commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
ZhangGuoqiang666 commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438286788 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 opened a new pull request, #16991: [chore](macOS) Support LLVM Clang 15
adonis0147 opened a new pull request, #16991: URL: https://github.com/apache/doris/pull/16991 # Proposed changes 1. Remove the deprecated classes [std::codecvt_utf8_utf16](https://en.cppreference.com/w/cpp/locale/codecvt_utf8_utf16) and [std::wstring_convert](https://en.cppreference.com/w/cpp/locale/wstring_convert). 2. Use `libiconv` to convert `UTF-8` strings to `UTF-16LE` ones. ## Problem summary If we use LLVM Clang 15 to build the codebase on macOS, the compiler will complain the deprecated declarations errors. See the following. ```shell /Programs/doris/be/src/exec/table_connector.cpp:49:31: error: 'codecvt_utf8_utf16' is deprecated [-Werror,-Wdeprecated-declarations] std::wstring_convert, char16_t> utf8_utf16_cvt; ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/codecvt:588:28: note: 'codecvt_utf8_utf16' has been explicitly marked deprecated here class _LIBCPP_TEMPLATE_VIS _LIBCPP_DEPRECATED_IN_CXX17 codecvt_utf8_utf16 ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/__config:800:41: note: expanded from macro '_LIBCPP_DEPRECATED_IN_CXX17' #define _LIBCPP_DEPRECATED_IN_CXX17 _LIBCPP_DEPRECATED ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/__config:773:49: note: expanded from macro '_LIBCPP_DEPRECATED' # define _LIBCPP_DEPRECATED __attribute__((deprecated)) ^ /Programs/doris/be/src/exec/table_connector.cpp:49:10: error: 'wstring_convert, char16_t>' is deprecated [-Werror,-Wdeprecated-declarations] std::wstring_convert, char16_t> utf8_utf16_cvt; ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/locale:3621:28: note: 'wstring_convert, char16_t>' has been explicitly marked deprecated here class _LIBCPP_TEMPLATE_VIS _LIBCPP_DEPRECATED_IN_CXX17 wstring_convert ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/__config:800:41: note: expanded from macro '_LIBCPP_DEPRECATED_IN_CXX17' #define _LIBCPP_DEPRECATED_IN_CXX17 _LIBCPP_DEPRECATED ^ /opt/homebrew/opt/llvm/bin/../include/c++/v1/__config:773:49: note: expanded from macro '_LIBCPP_DEPRECATED' # define _LIBCPP_DEPRECATED __attribute__((deprecated)) ^ 2 errors generated. ``` std::codecvt_utf8_utf16 and std::wstring_convert are marked deprecated in C++ 17, we should use an alternative to handle it. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #16991: [chore](macOS) Support LLVM Clang 15
adonis0147 commented on PR #16991: URL: https://github.com/apache/doris/pull/16991#issuecomment-1438299594 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16991: [chore](macOS) Support LLVM Clang 15
github-actions[bot] commented on PR #16991: URL: https://github.com/apache/doris/pull/16991#issuecomment-1438303139 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 commented on pull request #16736: [chore](dependency) show progress bar when downloading third-party dependencies
adonis0147 commented on PR #16736: URL: https://github.com/apache/doris/pull/16736#issuecomment-1438304847 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann opened a new pull request, #16992: [feat](Nereids): Add hint `NTH_OPTIMIZED_PLAN` to let the optimzier can select n-th optimized plan
XieJiann opened a new pull request, #16992: URL: https://github.com/apache/doris/pull/16992 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16987: [improvement](memory) Support GC segment cache, when memory insufficient
hello-stephen commented on PR #16987: URL: https://github.com/apache/doris/pull/16987#issuecomment-1438306331 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.98 seconds stream load tsv: 461 seconds loaded 74807831229 Bytes, about 154 MB/s stream load json: 38 seconds loaded 2358488459 Bytes, about 59 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/2023022840_clickbench_pr_101045.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16666: [ehancement](planner) Support filter the output of set operation node
morrySnow merged PR #1: URL: https://github.com/apache/doris/pull/1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [ehancement](planner) Support filter the output of set operation node (#16666)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a95f47ac0a [ehancement](planner) Support filter the output of set
operation node (#1)
a95f47ac0a is described below
commit a95f47ac0af5d4ff6d667754b6e4d64647fa90a2
Author: AKIRA <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Tue Feb 21 19:22:09 2023 +0800
[ehancement](planner) Support filter the output of set operation node
(#1)
---
.../src/main/java/org/apache/doris/analysis/Analyzer.java | 11 +++
.../src/main/java/org/apache/doris/planner/PlanNode.java| 4
.../java/org/apache/doris/planner/SingleNodePlanner.java| 13 ++---
regression-test/data/query_p0/union/test_union.out | 3 +++
regression-test/suites/query_p0/union/test_union.groovy | 2 ++
5 files changed, 30 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
index 9e198aa152..76aa3bb4dd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
@@ -397,6 +397,8 @@ public class Analyzer {
private final Set markTupleIdsNotProcessed =
Sets.newHashSet();
+private final Map> migrateFailedConjuncts =
Maps.newHashMap();
+
public GlobalState(Env env, ConnectContext context) {
this.env = env;
this.context = context;
@@ -1204,6 +1206,15 @@ public class Analyzer {
}
}
+public void registerMigrateFailedConjuncts(InlineViewRef ref, Expr e) {
+Set exprSet =
globalState.migrateFailedConjuncts.computeIfAbsent(ref, (k) -> new HashSet<>());
+exprSet.add(e);
+}
+
+public Set findMigrateFailedConjuncts(InlineViewRef inlineViewRef) {
+return globalState.migrateFailedConjuncts.get(inlineViewRef);
+}
+
/**
* register expr id
* @param expr
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
index 86c0520b27..08f59353af 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
@@ -1134,4 +1134,8 @@ public abstract class PlanNode extends TreeNode
implements PlanStats {
public List getOutputSlotIds() {
return outputSlotIds;
}
+
+public void setVConjunct(Set exprs) {
+vconjunct = convertConjunctsToAndCompoundPredicate(new
ArrayList<>(exprs));
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index 6b7162c54f..44d85396b9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -1814,8 +1814,9 @@ public class SingleNodePlanner {
newConjuncts = cloneExprs(newConjuncts);
}
} else {
-Preconditions.checkArgument(select.getTableRefs().size() == 1);
-viewAnalyzer.registerConjuncts(newConjuncts,
select.getTableRefs().get(0).getId());
+for (Expr e : conjuncts) {
+viewAnalyzer.registerMigrateFailedConjuncts(inlineViewRef,
e);
+}
}
} else {
Preconditions.checkArgument(stmt instanceof SetOperationStmt);
@@ -2150,7 +2151,13 @@ public class SingleNodePlanner {
scanNode = createScanNode(analyzer, tblRef, selectStmt);
}
if (tblRef instanceof InlineViewRef) {
-scanNode = createInlineViewPlan(analyzer, (InlineViewRef) tblRef);
+InlineViewRef inlineViewRef = (InlineViewRef) tblRef;
+scanNode = createInlineViewPlan(analyzer, inlineViewRef);
+Analyzer viewAnalyzer = inlineViewRef.getAnalyzer();
+Set exprs =
viewAnalyzer.findMigrateFailedConjuncts(inlineViewRef);
+if (CollectionUtils.isNotEmpty(exprs)) {
+scanNode.setVConjunct(exprs);
+}
}
if (scanNode == null) {
throw new UserException("unknown TableRef node");
diff --git a/regression-test/data/query_p0/union/test_union.out
b/regression-test/data/query_p0/union/test_union.out
index fe0f9c1810..4d398f2f2a 100644
--- a/regression-test/data/query_p0/union/test_union.out
+++ b/regression-test/data/query_p0/union/test_union.out
@@ -364,3 +364,6 @@ hell0
2016-07-01
2016-07-02
+-- !union36 --
+1 2
+
diff --git a/regression-test/suites/query_p0/union/test_union.groovy
b/regression-test/suites/query_p0/union/test
[GitHub] [doris] sohardforaname opened a new pull request, #16993: [Feature](planner)add dayofweek for FEFunctions to support fold constant.
sohardforaname opened a new pull request, #16993: URL: https://github.com/apache/doris/pull/16993 # Proposed changes Issue Number: close #xxx ## Problem summary add dayofweek for FEFunctions to support fold constant. use Zellar algorithm ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on pull request #16639: [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread
chenlinzhong commented on PR #16639: URL: https://github.com/apache/doris/pull/16639#issuecomment-1438312796 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zclllyybb commented on pull request #16935: [Fix](function)fix datatime-diff function's overflow
zclllyybb commented on PR #16935: URL: https://github.com/apache/doris/pull/16935#issuecomment-1438316639 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16935: [Fix](function)fix datatime-diff function's overflow
github-actions[bot] commented on PR #16935: URL: https://github.com/apache/doris/pull/16935#issuecomment-1438316924 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16731: segcompaction featuring verticalcompaction
github-actions[bot] commented on code in PR #16731:
URL: https://github.com/apache/doris/pull/16731#discussion_r1112928298
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -116,35 +118,46 @@ Status BetaRowsetWriter::add_block(const
vectorized::Block* block) {
return _add_block(block, &_segment_writer);
}
-RowwiseIteratorUPtr BetaRowsetWriter::_get_segcompaction_reader(
-SegCompactionCandidatesSharedPtr segments, std::shared_ptr
schema,
-OlapReaderStatistics* stat, uint64_t* merged_row_stat) {
+std::unique_ptr
BetaRowsetWriter::_get_segcompaction_reader(
Review Comment:
warning: method '_get_segcompaction_reader' can be made const
[readability-make-member-function-const]
be/src/olap/rowset/beta_rowset_writer.cpp:122:
```diff
- std::shared_ptr schema, OlapReaderStatistics* stat,
uint64_t* merged_row_stat, vectorized::RowSourcesBuffer& row_sources_buf, bool
is_key, std::vector& return_columns) {
+ std::shared_ptr schema, OlapReaderStatistics* stat,
uint64_t* merged_row_stat, vectorized::RowSourcesBuffer& row_sources_buf, bool
is_key, std::vector& return_columns) const {
```
be/src/olap/rowset/beta_rowset_writer.h:121:
```diff
-
vectorized::RowSourcesBuffer& row_sources_buf, bool is_key,
std::vector& return_columns);
+
vectorized::RowSourcesBuffer& row_sources_buf, bool is_key,
std::vector& return_columns) const;
```
##
be/src/vec/olap/vertical_block_reader.cpp:
##
@@ -86,9 +86,26 @@ Status VerticalBlockReader::_init_collect_iter(const
ReaderParams& read_params)
std::vector segment_iters;
std::vector iterator_init_flag;
std::vector rowset_ids;
-RETURN_IF_ERROR(
-_get_segment_iterators(read_params, &segment_iters,
&iterator_init_flag, &rowset_ids));
-CHECK(segment_iters.size() == iterator_init_flag.size());
+std::vector* segment_iters_ptr =
read_params.segment_iters_ptr;
+std::vector segment_iters;
Review Comment:
warning: redefinition of 'segment_iters' with a different type:
'vector' vs 'vector'
[clang-diagnostic-error]
```cpp
std::vector segment_iters;
^
```
**be/src/vec/olap/vertical_block_reader.cpp:85:** previous definition is here
```cpp
std::vector segment_iters;
^
```
##
be/test/olap/segcompaction_test.cpp:
##
@@ -200,12 +223,124 @@
std::unique_ptr _data_dir;
};
+TEST_F(SegCompactionTest, SegCompactionThenRead) {
+config::enable_segcompaction = true;
+config::enable_storage_vectorization = true;
+Status s;
+TabletSchemaSPtr tablet_schema = std::make_shared();
+create_tablet_schema(tablet_schema, DUP_KEYS);
+
+RowsetSharedPtr rowset;
+const int num_segments = 15;
+const uint32_t rows_per_segment = 4096;
+config::segcompaction_small_threshold = 6000; // set threshold above
+ // rows_per_segment
+config::segcompaction_threshold_segment_num = 10;
+std::vector segment_num_rows;
+{ // write `num_segments * rows_per_segment` rows to rowset
+RowsetWriterContext writer_context;
+create_rowset_writer_context(10047, tablet_schema, &writer_context);
+
+std::unique_ptr rowset_writer;
+s = RowsetFactory::create_rowset_writer(writer_context, false,
&rowset_writer);
+EXPECT_EQ(Status::OK(), s);
+
+RowCursor input_row;
+input_row.init(tablet_schema);
+
+// for segment "i", row "rid"
+// k1 := rid*10 + i
+// k2 := k1 * 10
+// k3 := rid
+for (int i = 0; i < num_segments; ++i) {
+MemPool mem_pool;
+for (int rid = 0; rid < rows_per_segment; ++rid) {
+uint32_t k1 = rid * 100 + i;
+uint32_t k2 = i;
+uint32_t k3 = rid;
+input_row.set_field_content(0, reinterpret_cast(&k1),
&mem_pool);
+input_row.set_field_content(1, reinterpret_cast(&k2),
&mem_pool);
+input_row.set_field_content(2, reinterpret_cast(&k3),
&mem_pool);
+s = rowset_writer->add_row(input_row);
Review Comment:
warning: no member named 'set_field_content' in 'doris::RowCursor'
[clang-diagnostic-error]
```cpp
input_row.set_field_content(2, reinterpret_cast(&k3),
&mem_pool);
^
```
##
be/test/olap/segcompaction_test.cpp:
##
@@ -184,6 +186,27 @@
rowset_writer_context->tablet_schema = tablet_schema;
rowset_writer_context->version.first = 10;
rowset_writer_context->version.second = 10;
+
+#if 0
+TCreateTabletReq req;
+req.table_id =
+re
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16731: segcompaction featuring verticalcompaction
github-actions[bot] commented on code in PR #16731:
URL: https://github.com/apache/doris/pull/16731#discussion_r1112933467
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -116,35 +118,46 @@ Status BetaRowsetWriter::add_block(const
vectorized::Block* block) {
return _add_block(block, &_segment_writer);
}
-RowwiseIteratorUPtr BetaRowsetWriter::_get_segcompaction_reader(
-SegCompactionCandidatesSharedPtr segments, std::shared_ptr
schema,
-OlapReaderStatistics* stat, uint64_t* merged_row_stat) {
+std::unique_ptr
BetaRowsetWriter::_get_segcompaction_reader(
Review Comment:
warning: method '_get_segcompaction_reader' can be made const
[readability-make-member-function-const]
be/src/olap/rowset/beta_rowset_writer.cpp:124:
```diff
- std::vector& return_columns) {
+ std::vector& return_columns) const {
```
```suggestion
std::vector& return_columns) const;
```
##
be/test/olap/segcompaction_test.cpp:
##
@@ -184,6 +186,27 @@
rowset_writer_context->tablet_schema = tablet_schema;
rowset_writer_context->version.first = 10;
rowset_writer_context->version.second = 10;
+
+#if 0
+TCreateTabletReq req;
+req.table_id =
+req.tablet_id =
+req.tablet_scheme =
+req.partition_id =
+l_engine->create_tablet(req);
+rowset_writer_context->tablet =
l_engine->tablet_manager()->get_tablet(TTabletId tablet_id);
+#endif
+std::shared_ptr data_dir =
std::make_shared(lTestDir);
+TabletMetaSharedPtr tablet_meta = std::make_shared();
+tablet_meta->_tablet_id = 1;
+tablet_meta->_schema = tablet_schema;
Review Comment:
warning: '_schema' is a private member of 'doris::TabletMeta'
[clang-diagnostic-error]
```cpp
tablet_meta->_schema = tablet_schema;
^
```
**be/src/olap/tablet_meta.h:229:** declared private here
```cpp
TabletSchemaSPtr _schema;
^
```
##
be/test/olap/segcompaction_test.cpp:
##
@@ -184,6 +186,27 @@ class SegCompactionTest : public testing::Test {
rowset_writer_context->tablet_schema = tablet_schema;
rowset_writer_context->version.first = 10;
rowset_writer_context->version.second = 10;
+
+#if 0
+TCreateTabletReq req;
+req.table_id =
+req.tablet_id =
+req.tablet_scheme =
+req.partition_id =
+l_engine->create_tablet(req);
+rowset_writer_context->tablet =
l_engine->tablet_manager()->get_tablet(TTabletId tablet_id);
+#endif
+std::shared_ptr data_dir =
std::make_shared(lTestDir);
+TabletMetaSharedPtr tablet_meta = std::make_shared();
+tablet_meta->_tablet_id = 1;
Review Comment:
warning: '_tablet_id' is a private member of 'doris::TabletMeta'
[clang-diagnostic-error]
```cpp
tablet_meta->_tablet_id = 1;
^
```
**be/src/olap/tablet_meta.h:217:** declared private here
```cpp
int64_t _tablet_id = 0;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16971: [fix](demo)fix cdc failed to synchronize datetime type in mysql, and added JsonDebeziumSchemaSerializer
github-actions[bot] commented on PR #16971: URL: https://github.com/apache/doris/pull/16971#issuecomment-1438329832 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16971: [fix](demo)fix cdc failed to synchronize datetime type in mysql, and added JsonDebeziumSchemaSerializer
github-actions[bot] commented on PR #16971: URL: https://github.com/apache/doris/pull/16971#issuecomment-1438329891 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
zhannngchen commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1438331931 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16989: [fix](memory) fix memory GC details and join probe catch bad_alloc
hello-stephen commented on PR #16989: URL: https://github.com/apache/doris/pull/16989#issuecomment-1438332397 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.73 seconds stream load tsv: 469 seconds loaded 74807831229 Bytes, about 152 MB/s stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230221113952_clickbench_pr_101050.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16910: [feature](Nereids): Eliminate duplicate join condition.
github-actions[bot] commented on PR #16910: URL: https://github.com/apache/doris/pull/16910#issuecomment-1438333163 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #16910: [feature](Nereids): Eliminate duplicate join condition.
jackwener merged PR #16910: URL: https://github.com/apache/doris/pull/16910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a95f47ac0a -> 54bf40b6e7)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from a95f47ac0a [ehancement](planner) Support filter the output of set
operation node (#1)
add 54bf40b6e7 [feature](Nereids): Eliminate duplicate join condition.
(#16910)
No new revisions were added by this update.
Summary of changes:
.../jobs/batch/NereidsRewriteJobExecutor.java | 2 ++
.../doris/nereids/jobs/joinorder/JoinOrderJob.java | 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 9 +++
.../rules/exploration/join/JoinReorderHelper.java | 2 +-
...iJoin.java => EliminateDedupJoinCondition.java} | 27 +++
.../apache/doris/nereids/stats/JoinEstimation.java | 4 +--
.../nereids/trees/expressions/Expression.java | 4 ---
.../nereids/trees/plans/logical/LogicalJoin.java | 16 +---
...t.java => EliminateDedupJoinConditionTest.java} | 22
.../logical/PruneOlapScanPartitionTest.java| 9 +++
.../org/apache/doris/nereids/sqltest/JoinTest.java | 19 ++
.../apache/doris/nereids/util/PlanConstructor.java | 30 --
.../main/java/org/apache/doris/udf/UdfUtils.java | 4 +--
13 files changed, 90 insertions(+), 60 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/{EliminateNullAwareLeftAntiJoin.java
=> EliminateDedupJoinCondition.java} (53%)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/{ExtractFilterFromCrossJoinTest.java
=> EliminateDedupJoinConditionTest.java} (69%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
github-actions[bot] commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438335680 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
dataroaring merged PR #16950: URL: https://github.com/apache/doris/pull/16950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](log) clarify error msg for tablet writer write failure (#14078) (#16954) (#16950)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 09d41c3479 [fix](log) clarify error msg for tablet writer write
failure (#14078) (#16954) (#16950)
09d41c3479 is described below
commit 09d41c34791e0caad896772a15970d4e95d3f6d6
Author: zhengyu
AuthorDate: Tue Feb 21 19:42:49 2023 +0800
[fix](log) clarify error msg for tablet writer write failure (#14078)
(#16954) (#16950)
fmt::format dosen't support non-template object as args, even if it
implements
`to_string()` or `operator<<`. so orignal code may cause `false` to be
printed
instead of real cause of the failure. So to_string() need to be manually
invoked.
Signed-off-by: freemandealer
---
be/src/runtime/tablets_channel.cpp | 25 ++---
1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/be/src/runtime/tablets_channel.cpp
b/be/src/runtime/tablets_channel.cpp
index 146137c026..c692ca03d8 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -111,8 +111,11 @@ Status TabletsChannel::close(
if (_partition_ids.count(it.second->partition_id()) > 0) {
auto st = it.second->close();
if (!st.ok()) {
-LOG(WARNING) << "close tablet writer failed, tablet_id="
<< it.first
- << ", transaction_id=" << _txn_id << ", err="
<< st;
+auto err_msg = fmt::format(
+"close tablet writer failed, tablet_id={}, "
+"transaction_id={}, err={}",
+it.first, _txn_id, st.to_string());
+LOG(WARNING) << err_msg;
PTabletError* tablet_error = tablet_errors->Add();
tablet_error->set_tablet_id(it.first);
tablet_error->set_msg(st.to_string());
@@ -247,12 +250,12 @@ Status TabletsChannel::_open_all_writers(const
PTabletWriterOpenRequest& request
DeltaWriter* writer = nullptr;
auto st = DeltaWriter::open(&wrequest, &writer, _load_id);
if (!st.ok()) {
-std::stringstream ss;
-ss << "open delta writer failed, tablet_id=" << tablet.tablet_id()
- << ", txn_id=" << _txn_id << ", partition_id=" <<
tablet.partition_id()
- << ", err=" << st;
-LOG(WARNING) << ss.str();
-return Status::InternalError(ss.str());
+auto err_msg = fmt::format(
+"open delta writer failed, tablet_id={}"
+", txn_id={}, partition_id={}, err={}",
+tablet.tablet_id(), _txn_id, tablet.partition_id(),
st.to_string());
+LOG(WARNING) << err_msg;
+return Status::InternalError(err_msg);
}
{
std::lock_guard l(_tablet_writers_lock);
@@ -338,7 +341,7 @@ Status TabletsChannel::add_batch(const
TabletWriterAddRequest& request,
if (!st.ok()) {
auto err_msg =
fmt::format("tablet writer write failed, tablet_id={},
txn_id={}, err={}",
-tablet_to_rowidxs_it.first, _txn_id, st);
+tablet_to_rowidxs_it.first, _txn_id,
st.to_string());
LOG(WARNING) << err_msg;
PTabletError* error = tablet_errors->Add();
error->set_tablet_id(tablet_to_rowidxs_it.first);
@@ -381,7 +384,7 @@ void TabletsChannel::flush_memtable_async(int64_t
tablet_id) {
auto err_msg = fmt::format(
"tablet writer failed to reduce mem consumption by flushing
memtable, "
"tablet_id={}, txn_id={}, err={}",
-tablet_id, _txn_id, st);
+tablet_id, _txn_id, st.to_string());
LOG(WARNING) << err_msg;
iter->second->cancel_with_status(st);
_broken_tablets.insert(iter->second->tablet_id());
@@ -409,7 +412,7 @@ void TabletsChannel::wait_flush(int64_t tablet_id) {
auto err_msg = fmt::format(
"tablet writer failed to reduce mem consumption by flushing
memtable, "
"tablet_id={}, txn_id={}, err={}",
-tablet_id, _txn_id, st);
+tablet_id, _txn_id, st.to_string());
LOG(WARNING) << err_msg;
iter->second->cancel_with_status(st);
_broken_tablets.insert(iter->second->tablet_id());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on pull request #16950: [fix](log) fix and clarify error msg for tablet writer write failure …
freemandealer commented on PR #16950: URL: https://github.com/apache/doris/pull/16950#issuecomment-1438338157 TODO: merge this to 1.2lts after PR #14744 be merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] DongLiang-0 opened a new pull request, #106: fix thrift compiler
DongLiang-0 opened a new pull request, #106: URL: https://github.com/apache/doris-flink-connector/pull/106 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] XieJiann commented on pull request #16992: [feat](Nereids): Add hint `NTH_OPTIMIZED_PLAN` to let the optimzier can select n-th optimized plan
XieJiann commented on PR #16992: URL: https://github.com/apache/doris/pull/16992#issuecomment-1438342089 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r449982
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java:
##
@@ -120,38 +156,70 @@ public void close() {
}
}
-protected int readAll(ByteBuffer dstBuf) throws IOException {
+// all packet header is not encrypted, packet body is not sure.
+protected int readAll(ByteBuffer dstBuf, boolean isHeader) throws
IOException {
int readLen = 0;
while (dstBuf.remaining() != 0) {
int ret = channel.read(dstBuf);
// return -1 when remote peer close the channel
if (ret == -1) {
+decryptData(dstBuf, isHeader);
return readLen;
}
readLen += ret;
}
+// if use ssl mode, wo need to decrypt received net data(ciphertext)
to app data(plaintext).
+decryptData(dstBuf,isHeader);
return readLen;
}
+protected void decryptData(ByteBuffer dstBuf, boolean isHeader) throws
SSLException {
+// after decrypt we get a mysql packet with mysql header.
+if (isSslMode && !isHeader) {
+dstBuf.flip();
+ByteBuffer appData =
ByteBuffer.allocate(sslEngine.getSession().getApplicationBufferSize());
+// unwrap will remove ssl header.
+sslEngine.unwrap(dstBuf, appData);
+appData.flip();
+dstBuf.clear();
+dstBuf.put(appData);
+dstBuf.flip();
+
+}
+}
+
// read one logical mysql protocol packet
// null for channel is closed.
// NOTE: all of the following code is assumed that the channel is in block
mode.
+// if in handshaking mode we return a packet with header otherwise without
header.
public ByteBuffer fetchOnePacket() throws IOException {
int readLen;
ByteBuffer result = defaultBuffer;
result.clear();
while (true) {
-headerByteBuffer.clear();
-readLen = readAll(headerByteBuffer);
-if (readLen != PACKET_HEADER_LEN) {
-// remote has close this channel
-LOG.debug("Receive packet header failed, remote may close the
channel.");
-return null;
-}
-if (packetId() != sequenceId) {
-LOG.warn("receive packet sequence id[" + packetId() + "] want
to get[" + sequenceId + "]");
-throw new IOException("Bad packet sequence.");
+if(isSslMode || isHandshaking){
Review Comment:
Code style, need a space after `if` or `)`
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlChannel.java:
##
@@ -120,38 +156,70 @@ public void close() {
}
}
-protected int readAll(ByteBuffer dstBuf) throws IOException {
+// all packet header is not encrypted, packet body is not sure.
+protected int readAll(ByteBuffer dstBuf, boolean isHeader) throws
IOException {
int readLen = 0;
while (dstBuf.remaining() != 0) {
int ret = channel.read(dstBuf);
// return -1 when remote peer close the channel
if (ret == -1) {
+decryptData(dstBuf, isHeader);
return readLen;
}
readLen += ret;
}
+// if use ssl mode, wo need to decrypt received net data(ciphertext)
to app data(plaintext).
+decryptData(dstBuf,isHeader);
return readLen;
}
+protected void decryptData(ByteBuffer dstBuf, boolean isHeader) throws
SSLException {
+// after decrypt we get a mysql packet with mysql header.
+if (isSslMode && !isHeader) {
+dstBuf.flip();
+ByteBuffer appData =
ByteBuffer.allocate(sslEngine.getSession().getApplicationBufferSize());
+// unwrap will remove ssl header.
+sslEngine.unwrap(dstBuf, appData);
+appData.flip();
+dstBuf.clear();
+dstBuf.put(appData);
+dstBuf.flip();
+
+}
+}
+
// read one logical mysql protocol packet
// null for channel is closed.
// NOTE: all of the following code is assumed that the channel is in block
mode.
+// if in handshaking mode we return a packet with header otherwise without
header.
public ByteBuffer fetchOnePacket() throws IOException {
int readLen;
ByteBuffer result = defaultBuffer;
result.clear();
while (true) {
-headerByteBuffer.clear();
-readLen = readAll(headerByteBuffer);
-if (readLen != PACKET_HEADER_LEN) {
-// remote has close this channel
-LOG.debug("Receive packet header failed, remote may close the
channel.");
-return null;
-}
-if (packetId() != sequenceId) {
-LOG.warn("receive packet sequence id[" +
[GitHub] [doris] gavinchou commented on a diff in pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on code in PR #16340:
URL: https://github.com/apache/doris/pull/16340#discussion_r1112950961
##
regression-test/suites/mysql_ssl_p0/test_mysql_connection.groovy:
##
@@ -0,0 +1,24 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_mysql_connection") {
+def sout = new StringBuilder(), serr = new StringBuilder()
+'mysql -uroot -P9030 -h127.0.0.1'.execute()
Review Comment:
Extract host and port from config.jdbcUrl
and add a command to exec
```
mysql -u .. -h ... -P... -e "show databases"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] gavinchou commented on pull request #16340: [feature](mysql) Support secure MySQL connection to FE
gavinchou commented on PR #16340: URL: https://github.com/apache/doris/pull/16340#issuecomment-1438347869 run compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Tanya-W opened a new pull request, #16994: [enhancement](inverted index) Support fulltext index evaluate equal query and list query
Tanya-W opened a new pull request, #16994: URL: https://github.com/apache/doris/pull/16994 # Proposed changes Issue Number: close #xxx ## Problem summary fulltext index is the inverted index of the specified tokenizer, before this pr, fulltext index only can evaluate match predicate, this pr to support evaluate equal predicate and list predicate. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16994: [enhancement](inverted index) Support fulltext index evaluate equal query and list query
github-actions[bot] commented on PR #16994: URL: https://github.com/apache/doris/pull/16994#issuecomment-1438353922 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on pull request #16889: [fix](replica) Fix inconsistent replica id between BE and FE in corner case of tablet rebalance
platoneko commented on PR #16889: URL: https://github.com/apache/doris/pull/16889#issuecomment-1438355910 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16616: [Feature](nereids) Support Mark join
github-actions[bot] commented on PR #16616: URL: https://github.com/apache/doris/pull/16616#issuecomment-1438357145 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16994: [enhancement](inverted index) Support fulltext index evaluate equal query and list query
github-actions[bot] commented on PR #16994: URL: https://github.com/apache/doris/pull/16994#issuecomment-1438357300 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #16995: [fix](nereids)dphyper join reorder may lost join condition in some case
starocean999 opened a new pull request, #16995: URL: https://github.com/apache/doris/pull/16995 # Proposed changes Issue Number: close #xxx ## Problem summary when emitCsgCmp, we should check if there is some missed edges should be used as connection edge. If there is missed edge but can't be used as connection edge, the emitCsgCmp should return and seek for another plan. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16948: [Improve](map-type) Add contains_null for map
github-actions[bot] commented on PR #16948: URL: https://github.com/apache/doris/pull/16948#issuecomment-1438371260 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16948: [Improve](map-type) Add contains_null for map
github-actions[bot] commented on PR #16948: URL: https://github.com/apache/doris/pull/16948#issuecomment-1438371350 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] jeffreys-cat opened a new pull request, #199: [PDF] update generate pdf action
jeffreys-cat opened a new pull request, #199: URL: https://github.com/apache/doris-website/pull/199 [[PDF] update generate pdf action](https://github.com/apache/doris-website/commit/685e852f88581eef4998ac7f654254d5bbf73430) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amorynan commented on pull request #16948: [Improve](map-type) Add contains_null for map
amorynan commented on PR #16948: URL: https://github.com/apache/doris/pull/16948#issuecomment-1438375396 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] morningman merged pull request #199: [PDF] update generate pdf action
morningman merged PR #199: URL: https://github.com/apache/doris-website/pull/199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [PDF] update generate pdf action (#199)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0424a591082 [PDF] update generate pdf action (#199)
0424a591082 is described below
commit 0424a591082a776c5f23f9689a8922924eac6eb1
Author: Jeffrey
AuthorDate: Tue Feb 21 20:15:29 2023 +0800
[PDF] update generate pdf action (#199)
Merge PDF upload and website upload actions, two action makes error, delete
some files
---
.github/workflows/cron-deploy-website.yml | 20 ++
.github/workflows/cron-generate-pdf.yml | 54 ---
.github/workflows/generate-pdf.yml | 58 -
.github/workflows/manual-deploy-website.yml | 28 +-
4 files changed, 39 insertions(+), 121 deletions(-)
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index 9f2eef4a2ef..fdcf74300fc 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -15,6 +15,18 @@ jobs:
persist-credentials: false
submodules: recursive
+- name: Add Chinese Fonts
+ run: |
+ sudo apt-get install ttf-mscorefonts-installer
+ sudo apt-get install fontconfig
+ fc-list :lang=zh
+ ls -lh /usr/share/fonts/
+ cp -rf ./static/fonts/* /usr/share/fonts/
+ mkfontscale
+ mkfontdir
+ fc-cache
+ fc-list :lang=zh
+
- name: Use Node.js
uses: actions/setup-node@v1
with:
@@ -47,6 +59,14 @@ jobs:
ls build
export DORIS_COMMIT=`git rev-parse HEAD`
+- name: Generate PDF
+ run: |
+npx vitpress-generate-pdf
--initialDocURLs="https://doris.apache.org/docs/dev/get-starting/";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris
Docs (English).pdf" --tocOnlyH1=true
+npx vitpress-generate-pdf
--initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs
(中文).pdf" --tocOnlyH1=true
+mkdir -p ./build/assets/files/
+cp *.pdf ./build/assets/files/
+ls ./build/assets/files/
+
- name: Upload files to OSS
uses: ./.github/actions/aliyun-oss-website-action
with:
diff --git a/.github/workflows/cron-generate-pdf.yml
b/.github/workflows/cron-generate-pdf.yml
deleted file mode 100644
index 40d48610351..000
--- a/.github/workflows/cron-generate-pdf.yml
+++ /dev/null
@@ -1,54 +0,0 @@
-name: Cron Generate PDF
-
-on:
- schedule:
-- cron: '0 1 * * *'
-env:
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-
-jobs:
- Generate-PDF:
-runs-on: ubuntu-latest
-environment: Production
-steps:
-- name: Checkout
- uses: actions/checkout@master
- with:
- persist-credentials: false
- submodules: recursive
-
-- name: Add Chinese Fonts
- run: |
- sudo apt-get install ttf-mscorefonts-installer
- sudo apt-get install fontconfig
- fc-list :lang=zh
- ls -lh /usr/share/fonts/
- cp -rf ./static/fonts/* /usr/share/fonts/
- mkfontscale
- mkfontdir
- fc-cache
- fc-list :lang=zh
-
-- name: Use Node.js
- uses: actions/setup-node@v1
- with:
-node-version: 16.14.0
-
-- name: Generate PDF
- run: |
-npx vitpress-generate-pdf
--initialDocURLs="https://doris.apache.org/docs/dev/get-starting/";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris
Docs (English).pdf" --tocOnlyH1=true
-npx vitpress-generate-pdf
--initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/get-starting/";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs
(中文).pdf" --tocOnlyH1=true
-ls
-rm -rf ./build/**
-mkdir -p ./build/assets/files/
-cp *.pdf ./build/assets/files/
-ls ./build/assets/files/
-
-- name: Upload files to OSS
- uses: ./.github/actions/aliyun-oss-website-action
- with:
- accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
- accessKey
[GitHub] [doris] yiguolei opened a new pull request, #16996: [enhancement](streamload) make stream load context as shared ptr not
yiguolei opened a new pull request, #16996: URL: https://github.com/apache/doris/pull/16996 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
zhannngchen commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1438387005 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on pull request #16878: [Feature](load) Add submitter and comments to load job
yangzhg commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1438393300 run allbuild -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
