[GitHub] [incubator-doris] BiteTheDDDDt commented on a change in pull request #8408: [Benchmark] Add TPC-H benchmark tools
BiteThet commented on a change in pull request #8408:
URL: https://github.com/apache/incubator-doris/pull/8408#discussion_r822382490
##
File path: tools/tpch-tools/run-tpch-queries.sh
##
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+##
+# This script is used to create TPC-H tables
+##
+
+set -eo pipefail
+
+ROOT=$(dirname "$0")
+ROOT=$(
+cd "$ROOT"
+pwd
+)
+
+CURDIR=${ROOT}
+QUERIES_DIR=$CURDIR/queries
+
+usage() {
+echo "
+This script is used to run TPC-H 22queries,
+will use mysql client to connect Doris server which parameter is specified in
doris-cluster.conf file.
+Usage: $0
+ "
+exit 1
+}
+
+OPTS=$(getopt \
+-n $0 \
+-o '' \
+-- "$@")
+
+eval set -- "$OPTS"
+HELP=0
+
+if [ $# == 0 ]; then
+usage
+fi
+
+while true; do
+case "$1" in
+-h)
+HELP=1
+shift
+;;
+--)
+shift
+break
+;;
+*)
+echo "Internal error"
+exit 1
+;;
+esac
+done
+
+if [[ ${HELP} -eq 1 ]]; then
+usage
+exit
+fi
+
+check_prerequest() {
+local CMD=$1
+local NAME=$2
+if ! $CMD; then
+echo "$NAME is missing. This script depends on mysql to create tables
in Doris."
+exit 1
+fi
+}
+
+check_prerequest "mysql --version" "mysql"
+
+source $CURDIR/doris-cluster.conf
+
+echo "FE_HOST: $FE_HOST"
+echo "FE_QUERY_PORT: $FE_QUERY_PORT"
+echo "USER: $USER"
+echo "PASSWORD: $PASSWORD"
+echo "DB: $DB"
+
+pre_set() {
+echo $@
+mysql -h$FE_HOST -u$USER -P$FE_QUERY_PORT -D$DB -e "$@"
+}
+
+pre_set "set global enable_vectorized_engine=1;"
+pre_set "set global parallel_fragment_exec_instance_num=8;"
+pre_set "set global exec_mem_limit=48G;"
+pre_set "set global batch_size=1024;"
Review comment:
here can set batch_size=4096
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 opened a new pull request #8412: [Enhancement](function)Support function YEARWEEK(DATE,INT,INT) RETURN INT
weizuo93 opened a new pull request #8412:
URL: https://github.com/apache/incubator-doris/pull/8412
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Support function `INT YEARWEEK(DATE date, INT week_start, INT
day_in_first_week)`.
* Returns year and week for the input date.
* The parameter `week_start` means the start day of a week, which value is
between 1 and 7，as is shown in following table：
|week_start|meaning|
|:-:|:-|
|1|Monday is the first day in a week|
|2|Tuesday is the first day in a week|
|3|Wednesday is the first day in a week|
|4|Thursday is the first day in a week|
|5|Friday is the first day in a week|
|6|Saturday is the first day in a week|
|7|Sunday is the first day in a week|
* The parameter `day_in_first_week` means which day in January belongs to
first week，which value is between 1 and 7,
as is shown in following table：
|day_in_first_week|meaning|
|:-:|:-|
|1|The week including January 1st is the first week in the year|
|2|The week including January 2nd is the first week in the year|
|3|The week including January 3rd is the first week in the year|
|4|The week including January 4th is the first week in the year|
|5|The week including January 5th is the first week in the year|
|6|The week including January 6th is the first week in the year|
|7|The week including January 7th is the first week in the year|
For example:
```
mysql> select yearweek('2022-1-1', 1, 1);
+---+
| yearweek('2022-01-01 00:00:00', 1, 1) |
+---+
|202201 |
+---+
```
```
mysql> select yearweek('2022-1-1', 1, 3);
+---+
| yearweek('2022-01-01 00:00:00', 1, 3) |
+---+
|202153 |
+---+
```
```
mysql> select yearweek('2021-12-29', 1, 1);
+---+
| yearweek('2021-12-29 00:00:00', 1, 1) |
+---+
|202201 |
+---+
```
## Checklist(Required)
1. Does it affect the original behavior: No
2. Has unit tests been added: Yes
3. Has document been added or modified: Yes
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8409: [Feature][Vectorized] support digital_masking
github-actions[bot] commented on pull request #8409: URL: https://github.com/apache/incubator-doris/pull/8409#issuecomment-1062664409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8358: [Doc][Vectorized] Add vectorized execution engine docs
github-actions[bot] commented on pull request #8358: URL: https://github.com/apache/incubator-doris/pull/8358#issuecomment-1062678595 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8373: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
HappenLee commented on a change in pull request #8373:
URL: https://github.com/apache/incubator-doris/pull/8373#discussion_r822403837
##
File path: be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
##
@@ -52,73 +51,115 @@ struct AggregateFunctionHLLData {
Int64 get_cardinality() const { return dst_hll.estimate_cardinality(); }
-HyperLogLog get() const {
-return dst_hll;
+HyperLogLog get() const { return dst_hll; }
+
+void result_union_agg_impl(IColumn& to) const {
+if constexpr (is_nullable) {
+auto& null_column = assert_cast(to);
+null_column.get_null_map_data().push_back(0);
+assert_cast(null_column.get_nested_column())
+.get_data()
+.push_back(get_cardinality());
+} else {
+
assert_cast(to).get_data().push_back(get_cardinality());
+}
+}
+
+void result_union_impl(IColumn& to) const {
+if constexpr (is_nullable) {
+auto& null_column = assert_cast(to);
+null_column.get_null_map_data().push_back(0);
+
assert_cast(null_column.get_nested_column()).get_data().emplace_back(get());
+} else {
+auto& column = static_cast(to);
+column.get_data().emplace_back(get());
+}
+}
+
+static const DataTypePtr return_type_union_agg() {
+if constexpr (is_nullable) {
+return make_nullable(std::make_shared());
+} else {
+return std::make_shared();
+}
}
+static const DataTypePtr return_type_union() {
+if constexpr (is_nullable) {
+return make_nullable(std::make_shared());
+} else {
+return std::make_shared();
+}
+}
+
+void add(const IColumn* column, size_t row_num) {
+if (auto* nullable_column = check_and_get_column(*column)) {
Review comment:
if constexpr ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8373: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
HappenLee commented on a change in pull request #8373:
URL: https://github.com/apache/incubator-doris/pull/8373#discussion_r822408744
##
File path: be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
##
@@ -52,73 +51,115 @@ struct AggregateFunctionHLLData {
Int64 get_cardinality() const { return dst_hll.estimate_cardinality(); }
-HyperLogLog get() const {
-return dst_hll;
+HyperLogLog get() const { return dst_hll; }
+
+void result_union_agg_impl(IColumn& to) const {
+if constexpr (is_nullable) {
+auto& null_column = assert_cast(to);
+null_column.get_null_map_data().push_back(0);
+assert_cast(null_column.get_nested_column())
+.get_data()
+.push_back(get_cardinality());
+} else {
+
assert_cast(to).get_data().push_back(get_cardinality());
+}
+}
+
+void result_union_impl(IColumn& to) const {
+if constexpr (is_nullable) {
Review comment:
maybe the result is always not nullable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] dongweizhao opened a new issue #8415: 内存和CPU都不高，查询sql会被阻塞
dongweizhao opened a new issue #8415: URL: https://github.com/apache/incubator-doris/issues/8415 为了便于快速定位和解决大家的问题，简单定了一个模板大家以后遇到问题按照这个模板提供信息，社区的开发者看到问题之后会很快给出解决方案，并在你的帖子下面回复，这样也便于Doris知识体系的形成，对后面新人使用Doris也是一个非常好的沉淀 ## 提问模板 ### 问题描述： ``` 描述你的问题情况，及引发这个异常的操作 ``` ### 问题复现的case： ``` 怎么能复现你的问题，这个越详细越好，问题解决人员可以按照你提供的这个case按照步骤很快定位到问题 最好提供你当时操作的表结构及SQL语句及几条每张表的示例数据 ``` ### Doris版本： ``` 比如：0.15 ``` ### Doris集群基本信息： ``` fe，be节点数据及配置情况 ``` ### 异常信息： ``` 如果是FE，请在fe.log文件中找到对应异常信息，如果是be找到对应be的be.out及be.INFO里的信息，如果是BE挂掉的情况，先执行dmesg -T查看是不是OOM，然后打开linux core dump，在下次出现问题的时候，提供be.out及core.xxx文件以便调试定位问题 ``` ### 解决方案（社区技术人员或者其他用户给出的回复解决方案） ``` 问题定位及解决以后，会在你的帖子下面给出解决方案，同时这样也会沉淀成Doris的知识文档方便后面新用户遇到问题快速找到解决方案 ``` _Originally posted by @hf200012 in https://github.com/apache/incubator-doris/discussions/8411_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] dongweizhao closed issue #8415: 内存和CPU都不高，查询sql会被阻塞
dongweizhao closed issue #8415: URL: https://github.com/apache/incubator-doris/issues/8415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8410: [fix] fix compile error
github-actions[bot] commented on pull request #8410: URL: https://github.com/apache/incubator-doris/pull/8410#issuecomment-1062689793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8373: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
HappenLee commented on a change in pull request #8373:
URL: https://github.com/apache/incubator-doris/pull/8373#discussion_r822418648
##
File path: be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
##
@@ -52,73 +51,115 @@ struct AggregateFunctionHLLData {
Int64 get_cardinality() const { return dst_hll.estimate_cardinality(); }
-HyperLogLog get() const {
-return dst_hll;
+HyperLogLog get() const { return dst_hll; }
+
+void result_union_agg_impl(IColumn& to) const {
+if constexpr (is_nullable) {
+auto& null_column = assert_cast(to);
+null_column.get_null_map_data().push_back(0);
+assert_cast(null_column.get_nested_column())
+.get_data()
+.push_back(get_cardinality());
+} else {
+
assert_cast(to).get_data().push_back(get_cardinality());
+}
+}
+
+void result_union_impl(IColumn& to) const {
+if constexpr (is_nullable) {
+auto& null_column = assert_cast(to);
+null_column.get_null_map_data().push_back(0);
+
assert_cast(null_column.get_nested_column()).get_data().emplace_back(get());
+} else {
+auto& column = static_cast(to);
+column.get_data().emplace_back(get());
+}
+}
+
+static const DataTypePtr return_type_union_agg() {
+if constexpr (is_nullable) {
+return make_nullable(std::make_shared());
+} else {
+return std::make_shared();
+}
}
+static const DataTypePtr return_type_union() {
+if constexpr (is_nullable) {
+return make_nullable(std::make_shared());
+} else {
+return std::make_shared();
+}
+}
+
+void add(const IColumn* column, size_t row_num) {
+if (auto* nullable_column = check_and_get_column(*column)) {
+if (nullable_column->is_null_at(row_num)) {
+return;
+}
+const auto& sources =
+static_cast((nullable_column->get_nested_column()));
+dst_hll.merge(sources.get_element(row_num));
+} else {
+const auto& sources = static_cast(*column);
+dst_hll.merge(sources.get_element(row_num));
+}
+}
+};
+
+template
Review comment:
refactor the struct template
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zbtzbtzbt commented on a change in pull request #8399: [improvement]avoid read bitmap index when no index works
zbtzbtzbt commented on a change in pull request #8399:
URL: https://github.com/apache/incubator-doris/pull/8399#discussion_r822420441
##
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##
@@ -76,6 +85,27 @@ class SegmentIterator::BitmapRangeIterator {
return true;
}
+bool get_next_range_without_bitmap_index(uint32_t max_range_size,
uint32_t* from, uint32_t* to) {
+if (_curr_pos == _max_value) {
+return false;
+}
+if (_curr_pos + max_range_size <= _max_value) {
Review comment:
should we add `_eof` like `get_next_range_with_bitmap_index()`? ( i am
not sure)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HackToday opened a new issue #8417: [Bug] ssb load data failed with Missing required fields in request: need_gen_rollup
HackToday opened a new issue #8417: URL: https://github.com/apache/incubator-doris/issues/8417 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version latest master branch commit id: f4663ad2eb3fc8ce929304ccdea09d87bb86ec8a ### What's Wrong? Following https://doris.apache.org/zh-CN/benchmark/star-schema-benchmark.html#%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87 to import data, with `bash load-dimension-data.sh` run with with following error ``` W0309 17:33:55.149286 765291 stream_load_executor.cpp:80] fragment execute failed, query_id=ed488e005d1adc92-796838cb0f1bd68d, err_msg=NodeChannel[11686-11001], load_id=ed488e005d1adc92-796838cb0f1bd68d, txn_id=1013, node=192.168.42.126:8060, open failed, err: failed to open tablet writer, error=Bad request, error_text=[E1003]Missing required fields in request: need_gen_rollup, hos, id=ed488e005d1adc92-796838cb0f1bd68d, job_id=-1, txn_id=1013, label=74da78b0-5146-4e60-aed7-3f4d0e100f20, elapse(s)=0 W0309 17:33:55.149302 765570 status.h:260] warning: Status msg truncated, OK: cancelled: NodeChannel[11686-11001], load_id=ed488e005d1adc92-796838cb0f1bd68d, txn_id=1013, node=:8060, open failed, err: failed to open tablet writer, error=Bad request, error_text=[E1003]Missing required fields in request: need_gen_rollup, hos precise_code:1 ``` ### What You Expected? bash load-dimension-data.sh would run success ### How to Reproduce? Just follow the official build and run and load data guide as following https://doris.apache.org/zh-CN/installing/compilation.html#%E4%BD%BF%E7%94%A8-docker-%E5%BC%80%E5%8F%91%E9%95%9C%E5%83%8F%E7%BC%96%E8%AF%91-%E6%8E%A8%E8%8D%90 https://doris.apache.org/zh-CN/benchmark/star-schema-benchmark.html#%E6%95%B0%E6%8D%AE%E5%87%86%E5%A4%87 ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morrySnow opened a new issue #8418: [Enhancement] do we need to add case insensitive string fuzzy matching udf: ilike ?
morrySnow opened a new issue #8418: URL: https://github.com/apache/incubator-doris/issues/8418 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description `ilike` is a common udf for case insensitive string fuzzy matching that implemented by other database or query engine[1]. it is vary useful to do string search when we don't care about character case. [1]: [redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_patternmatching_condition_like.html) [databricks](https://docs.databricks.com/sql/language-manual/functions/ilike.html) [postgreSQL](https://www.postgresql.org/docs/7.3/functions-matching.html) [snowflake](https://docs.snowflake.com/en/sql-reference/functions/ilike.html) ### Solution _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HackToday commented on issue #8417: [Bug] ssb load data failed with Missing required fields in request: need_gen_rollup
HackToday commented on issue #8417: URL: https://github.com/apache/incubator-doris/issues/8417#issuecomment-1062731438 @morningman Could someone help check if new code breaks ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #8412: [Enhancement](function)Support function YEARWEEK(DATE,INT,INT) RETURN INT
HappenLee commented on pull request #8412: URL: https://github.com/apache/incubator-doris/pull/8412#issuecomment-1062750496 please support it in vec exec engine too， thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on pull request #8412: [Enhancement](function)Support function YEARWEEK(DATE,INT,INT) RETURN INT
weizuo93 commented on pull request #8412: URL: https://github.com/apache/incubator-doris/pull/8412#issuecomment-1062757372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 removed a comment on pull request #8412: [Enhancement](function)Support function YEARWEEK(DATE,INT,INT) RETURN INT
weizuo93 removed a comment on pull request #8412: URL: https://github.com/apache/incubator-doris/pull/8412#issuecomment-1062757372 > please support it in vec exec engine too， thank you I had planned to implement vectorization for this func in another PR. That's all right, I will add vectorization code in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HackToday commented on issue #8417: [Bug] ssb load data failed with Missing required fields in request: need_gen_rollup
HackToday commented on issue #8417: URL: https://github.com/apache/incubator-doris/issues/8417#issuecomment-1062776518 I used commit `b536c1850fc8e29a11130dd4743796eee261f7dc`, it works, so it should be a new bug between such two commits -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #8410: [fix] fix compile error
yangzhg merged pull request #8410: URL: https://github.com/apache/incubator-doris/pull/8410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (f4663ad -> 2a43313)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from f4663ad [improvement](vectorized) Merge block in scanner to speed up query with conjunct (#8395) add 2a43313 [fix] fix compile error (#8410) No new revisions were added by this update. Summary of changes: be/src/agent/cgroups_mgr.cpp| 38 ++--- be/src/olap/task/engine_batch_load_task.cpp | 3 ++- 2 files changed, 21 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #8410: [fix] fix compile error
yangzhg commented on pull request #8410: URL: https://github.com/apache/incubator-doris/pull/8410#issuecomment-1062799065 merge it to fix master compile error -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r822616794
##
File path: be/src/util/quantile_state.cpp
##
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include
+#include
+#include "util/quantile_state.h"
+#include "util/coding.h"
+#include "common/logging.h"
+
+
+namespace doris{
+
+template
+QuantileState::QuantileState():_type(EMPTY),compression(2048){}
+
+template
+QuantileState::QuantileState(float
compression):_type(EMPTY),compression(compression){}
+
+
+template
+QuantileState::QuantileState(const Slice& slice) {
+if (!deserialize(slice)) {
+_type = EMPTY;
+}
+}
+
+template
+QuantileState::~QuantileState() {
+clear();
+}
+
+template
+size_t QuantileState::get_serialized_size() {
+size_t size = 1 + sizeof(float); // type(QuantileStateType) +
compression(float)
+switch(_type) {
+case EMPTY:
+break;
+case SINGLE:
+size += sizeof(T);
+break;
+case EXPLICIT:
+size += sizeof(uint16_t) + sizeof(T) * _explicit_data.size();
+break;
+case TDIGEST:
+size += tdigest_ptr->serialized_size();
+break;
+}
+return size;
+}
+
+template
+void QuantileState::set_compression(float compression) {
+this->compression = compression;
+}
+
+template
+bool QuantileState::is_valid(const Slice& slice) {
+
+if (slice.size < 1) {
+return false;
+}
+const uint8_t* ptr = (uint8_t*)slice.data;
+const uint8_t* end = (uint8_t*)slice.data + slice.size;
+float compress_value = *reinterpret_cast(ptr);
+if (compress_value < 2048 || compress_value > 1) {
+return false;
+}
+ptr += sizeof(float);
+
+
+auto type = (QuantileStateType)*ptr++;
+switch (type) {
+case EMPTY:
+break;
+case SINGLE:{
+if ((ptr + sizeof(T)) > end) {
+return false;
+}
+ptr += sizeof(T);
+// _single_data = *reinterpret_cast(ptr)++;
+break;
+}
+case EXPLICIT: {
+if ((ptr + sizeof(uint16_t)) > end) {
+return false;
+}
+uint16_t num_explicits = decode_fixed16_le(ptr);
+ptr += sizeof(uint16_t);
+ptr += num_explicits * sizeof(T);
+break;
+}
+case TDIGEST: {
+if ((ptr + sizeof(uint32_t)) > end) {
+return false;
+}
+uint32_t tdigest_serialized_length = decode_fixed32_le(ptr);
+ptr += tdigest_serialized_length;
Review comment:
Here just check whether the total length is correct.
If check the content , we should call `deserialize()` first. It will take
lots of time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] taberylyang commented on pull request #8393: 1.support shared hash table for broadcast hash join,use enable_shared…
taberylyang commented on pull request #8393: URL: https://github.com/apache/incubator-doris/pull/8393#issuecomment-1062901300 > BTW, Please add an issue including the function and structure design ~ [shared hash table.docx](https://github.com/apache/incubator-doris/files/8214875/shared.hash.table.docx) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] cambyzju commented on pull request #8401: [feature][array-type]add array_contains function
cambyzju commented on pull request #8401: URL: https://github.com/apache/incubator-doris/pull/8401#issuecomment-1062952035 > yes, I will add some UT in this pr later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei opened a new pull request #8420: [Fix bugs] brpc will check required field in proto and need_gen_rollup is moved will throw exception
yiguolei opened a new pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420 # Proposed changes Issue Number: close #8417 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7234: [Thread] Add thread local variable ThreadContext
morningman merged pull request #7234: URL: https://github.com/apache/incubator-doris/pull/7234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7196: [Feature] Refactored memory statistics framework MemTracker
morningman closed issue #7196: URL: https://github.com/apache/incubator-doris/issues/7196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8358: [Doc][Vectorized] Add vectorized execution engine docs
morningman merged pull request #8358: URL: https://github.com/apache/incubator-doris/pull/8358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8379: Fix dead link in build docker environment in developer guide
morningman merged pull request #8379: URL: https://github.com/apache/incubator-doris/pull/8379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8378: [Enhancement] [doc] there are some dead link in fe doc pages
morningman closed issue #8378: URL: https://github.com/apache/incubator-doris/issues/8378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8407: [Feature][Vectorized] support digital_masking
morningman closed issue #8407: URL: https://github.com/apache/incubator-doris/issues/8407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8409: [Feature][Vectorized] support digital_masking
morningman merged pull request #8409: URL: https://github.com/apache/incubator-doris/pull/8409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8420: [fix] brpc will check required field in proto and need_gen_rollup is moved will throw exception
github-actions[bot] commented on pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420#issuecomment-1063543261 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8305: [feature-wip][array-type] Support nested array insertion.
morningman commented on a change in pull request #8305:
URL: https://github.com/apache/incubator-doris/pull/8305#discussion_r823247577
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
##
@@ -373,6 +373,9 @@ private void toChildrenThrift(Column column, TColumn
tColumn) {
childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
childrenTColumn.setColumnType(childrenTColumnType);
childrenTColumn.setIsAllowNull(children.isAllowNull());
+if (tColumn.getAggregationType() != null) {
+
childrenTColumn.setAggregationType(tColumn.getAggregationType());
Review comment:
It seems to be a design problem. Please add this comment to the code for
notice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (2a43313 -> 8eec4bf)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 2a43313 [fix] fix compile error (#8410)
add 8eec4bf [feature](thread-local) Add thread local variable
ThreadContext (#7234)
No new revisions were added by this update.
Summary of changes:
be/src/runtime/CMakeLists.txt | 2 +
.../{multi_precision.h => thread_context.cpp} | 15 +--
be/src/runtime/thread_context.h| 143 +
be/src/runtime/threadlocal.cc | 84
be/src/runtime/threadlocal.h | 127 ++
5 files changed, 363 insertions(+), 8 deletions(-)
copy be/src/runtime/{multi_precision.h => thread_context.cpp} (74%)
create mode 100644 be/src/runtime/thread_context.h
create mode 100644 be/src/runtime/threadlocal.cc
create mode 100644 be/src/runtime/threadlocal.h
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (8eec4bf -> 8c84160)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 8eec4bf [feature](thread-local) Add thread local variable ThreadContext (#7234) add 8c84160 [doc](vectorized) Add vectorized execution engine docs (#8358) No new revisions were added by this update. Summary of changes: .../public/images/vectorized-execution-engine1.png | Bin 0 -> 113770 bytes .../public/images/vectorized-execution-engine2.png | Bin 0 -> 179015 bytes docs/.vuepress/sidebar/en.js | 1 + docs/.vuepress/sidebar/zh-CN.js| 1 + .../vectorized-execution-engine.md | 126 + .../vectorized-execution-engine.md | 123 6 files changed, 251 insertions(+) create mode 100644 docs/.vuepress/public/images/vectorized-execution-engine1.png create mode 100644 docs/.vuepress/public/images/vectorized-execution-engine2.png create mode 100644 docs/en/administrator-guide/vectorized-execution-engine.md create mode 100644 docs/zh-CN/administrator-guide/vectorized-execution-engine.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (8c84160 -> 548fde6)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 8c84160 [doc](vectorized) Add vectorized execution engine docs (#8358) add 548fde6 [doc] Fix dead link in build docker environment in developer guide (#8379) No new revisions were added by this update. Summary of changes: docs/en/developer-guide/docker-dev.md| 2 +- docs/zh-CN/developer-guide/docker-dev.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (548fde6 -> d3d8301)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 548fde6 [doc] Fix dead link in build docker environment in developer guide (#8379) add d3d8301 [feature](function) support vectorized digital_masking (#8409) No new revisions were added by this update. Summary of changes: .../org/apache/doris/catalog/AliasFunction.java| 35 ++ .../java/org/apache/doris/catalog/Function.java| 2 +- 2 files changed, 23 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #8417: [Bug] ssb load data failed with Missing required fields in request: need_gen_rollup
yiguolei commented on issue #8417: URL: https://github.com/apache/incubator-doris/issues/8417#issuecomment-1063550307 @HackToday Sorry, it's my fault. I have fixed this bug in 8420 pr. I have tested it on my local machine. Could you please test it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8368: [feature-wip][array-type] Array data can be loaded in stream load.
github-actions[bot] commented on pull request #8368: URL: https://github.com/apache/incubator-doris/pull/8368#issuecomment-1063551174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8368: [feature-wip][array-type] Array data can be loaded in stream load.
morningman merged pull request #8368: URL: https://github.com/apache/incubator-doris/pull/8368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch array-type updated (30e93e0 -> 07d16d4)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch array-type in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 30e93e0 [feature-wip](array-type)support select ARRAY data type on vectorized engine (#8217) add 07d16d4 [feature-wip][array-type] Array data can be loaded in stream load. (#8368) No new revisions were added by this update. Summary of changes: be/src/exprs/cast_functions.cpp| 7 + be/src/exprs/cast_functions.h | 2 + be/src/util/array_parser.hpp | 212 + be/test/util/CMakeLists.txt| 1 + be/test/util/array_parser_test.cpp | 134 + .../java/org/apache/doris/analysis/CastExpr.java | 19 +- .../main/java/org/apache/doris/catalog/Type.java | 4 +- 7 files changed, 371 insertions(+), 8 deletions(-) create mode 100644 be/src/util/array_parser.hpp create mode 100644 be/test/util/array_parser_test.cpp - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei closed pull request #8420: [fix] brpc will check required field in proto and need_gen_rollup is moved will throw exception
yiguolei closed pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on pull request #8420: [fix] brpc will check required field in proto and need_gen_rollup is moved will throw exception
yiguolei commented on pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420#issuecomment-1063561964 > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei removed a comment on pull request #8420: [fix] brpc will check required field in proto and need_gen_rollup is moved will throw exception
yiguolei removed a comment on pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420#issuecomment-1063561964 > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on pull request #8420: [fix] brpc will check required field in proto and need_gen_rollup is moved will throw exception
yiguolei commented on pull request #8420: URL: https://github.com/apache/incubator-doris/pull/8420#issuecomment-1063562505 > It is proto buf. Do you mean I could add default value in pb and did not need to set its valued in my code? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8328: small fixes and code cleanups related to schema change
github-actions[bot] commented on pull request #8328: URL: https://github.com/apache/incubator-doris/pull/8328#issuecomment-1063573723 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8393: 1.support shared hash table for broadcast hash join,use enable_shared…
morningman commented on a change in pull request #8393: URL: https://github.com/apache/incubator-doris/pull/8393#discussion_r823273769 ## File path: be/src/exprs/shared_hash_table.cpp ## @@ -0,0 +1,166 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "runtime/shared_hash_table.h" Review comment: Why the header in under `runtime/` but the .cpp is under `expr/`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 01/05: [feature-wip](array-type) Create table with nested array type. (#8003)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b4f227e329f0491e581bb01b3a71c488382d4375
Author: Adonis Ling
AuthorDate: Wed Feb 16 12:44:30 2022 +0800
[feature-wip](array-type) Create table with nested array type. (#8003)
```
create table array_type_table(k1 INT, k2 Array>) duplicate key
(k1)
distributed by hash(k1) buckets 1 properties('replication_num' = '1');
```
---
be/src/olap/tablet_meta.cpp | 9 -
.../org/apache/doris/analysis/CreateTableStmt.java | 16
.../main/java/org/apache/doris/analysis/TypeDef.java | 10 --
.../java/org/apache/doris/catalog/ArrayType.java | 9 +
.../main/java/org/apache/doris/catalog/Column.java | 10 --
.../src/main/java/org/apache/doris/catalog/Type.java | 20 +++-
.../org/apache/doris/catalog/CreateTableTest.java| 13 +
7 files changed, 41 insertions(+), 46 deletions(-)
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index 6e6195d..73632a1 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -128,11 +128,6 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t
partition_id, int64_t tablet_id
}
}
}
-
-if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
-ColumnPB* children_column = column->add_children_columns();
-_init_column_from_tcolumn(0, tcolumn.children_column[0],
children_column);
-}
}
schema->set_next_column_unique_id(next_unique_id);
@@ -210,6 +205,10 @@ void TabletMeta::_init_column_from_tcolumn(uint32_t
unique_id, const TColumn& tc
if (tcolumn.__isset.is_bloom_filter_column) {
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
}
+if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
+ColumnPB* children_column = column->add_children_columns();
+_init_column_from_tcolumn(0, tcolumn.children_column[0],
children_column);
+}
}
OLAPStatus TabletMeta::create_from_file(const string& file_path) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index ed2b689..5fb937e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -17,8 +17,11 @@
package org.apache.doris.analysis;
+import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.commons.collections.CollectionUtils;
import org.apache.doris.catalog.AggregateType;
-import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Index;
@@ -35,12 +38,6 @@ import org.apache.doris.common.util.PrintableMap;
import org.apache.doris.external.elasticsearch.EsUtil;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
-
-import com.google.common.base.Strings;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
-import org.apache.commons.collections.CollectionUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -372,11 +369,6 @@ public class CreateTableStmt extends DdlStmt {
columnDef.analyze(engineName.equals("olap"));
if (columnDef.getType().isArrayType()) {
-ArrayType tp = (ArrayType) columnDef.getType();
-if (!tp.getItemType().getPrimitiveType().isIntegerType() &&
-!tp.getItemType().getPrimitiveType().isCharFamily()) {
-throw new AnalysisException("Array column just support
INT/VARCHAR sub-type");
-}
if (columnDef.getAggregateType() != null &&
columnDef.getAggregateType() != AggregateType.NONE) {
throw new AnalysisException("Array column can't support
aggregation " + columnDef.getAggregateType());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
index 6bce9b5..d81d633 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
@@ -92,8 +92,10 @@ public class TypeDef implements ParseNode {
throw new AnalysisException("Unsupported data type: " + type.toSql());
}
if (type.isArrayType()) {
-ScalarType itemType = (ScalarType) ((ArrayType) type).getItemType();
-analyzeNestedType(itemType);
+Type itemType = ((ArrayType) type).getItem
[GitHub] [incubator-doris] adonis0147 commented on a change in pull request #8305: [feature-wip][array-type] Support nested array insertion.
adonis0147 commented on a change in pull request #8305:
URL: https://github.com/apache/incubator-doris/pull/8305#discussion_r823287292
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
##
@@ -373,6 +373,9 @@ private void toChildrenThrift(Column column, TColumn
tColumn) {
childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
childrenTColumn.setColumnType(childrenTColumnType);
childrenTColumn.setIsAllowNull(children.isAllowNull());
+if (tColumn.getAggregationType() != null) {
+
childrenTColumn.setAggregationType(tColumn.getAggregationType());
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 03/05: [feature-wip](array-type)support select ARRAY data type on vectorized engine (#8217)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit d041fd0e1a4a5a2c52fcba355c525f6d084a577a
Author: camby <104178...@qq.com>
AuthorDate: Tue Mar 8 20:26:51 2022 +0800
[feature-wip](array-type)support select ARRAY data type on vectorized
engine (#8217)
Usage Example:
1. create table for test;
`CREATE TABLE `array_test` (
`k1` tinyint(4) NOT NULL COMMENT "",
`k2` smallint(6) NULL COMMENT "",
`k3` ARRAY NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`k1`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`k1`) BUCKETS 5
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);`
2. insert some data
`insert into array_test values(1, 2, [1, 2]);`
`insert into array_test values(2, 3, null);`
`insert into array_test values(3, null, null);`
`insert into array_test values(4, null, []);`
3. open vectorized
`set enable_vectorized_engine=true;`
4. query array data
`select * from array_test;`
+--+--++
| k1 | k2 | k3 |
+--+--++
|4 | NULL | [] |
|2 |3 | NULL |
|1 |2 | [1, 2] |
|3 | NULL | NULL |
+--+--++
4 rows in set (0.061 sec)
Code Changes include：
1. add column_array, data_type_array codes;
2. codes about data_type creation by Field, TabletColumn, TypeDescriptor,
PColumnMeta move to DataTypeFactory;
3. support create data_type for ARRAY date type;
4. RowBlockV2::convert_to_vec_block support ARRAY date type;
5. VMysqlResultWriter::append_block support ARRAY date type;
6. vectorized::Block serialize and deserialize support ARRAY date type;
---
be/src/olap/column_vector.h| 2 +-
be/src/olap/field.h| 3 +-
be/src/olap/row_block2.cpp | 256
be/src/olap/row_block2.h | 2 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 21 +-
be/src/olap/schema.cpp | 51 +-
be/src/olap/schema.h | 2 +-
be/src/olap/tablet_schema.cpp | 4 +-
be/src/runtime/descriptors.cpp | 15 +-
be/src/runtime/descriptors.h | 3 +-
be/src/runtime/types.h | 54 +-
be/src/vec/CMakeLists.txt | 3 +
be/src/vec/columns/column_array.cpp| 704 +
be/src/vec/columns/column_array.h | 185 ++
be/src/vec/core/block.cpp | 84 +--
be/src/vec/core/types.h| 1 +
be/src/vec/data_types/data_type.cpp| 135 +---
be/src/vec/data_types/data_type.h | 2 -
be/src/vec/data_types/data_type_array.cpp | 97 +++
be/src/vec/data_types/data_type_array.h| 77 +++
be/src/vec/data_types/data_type_factory.cpp| 254
be/src/vec/data_types/data_type_factory.hpp| 16 +
be/src/vec/data_types/data_type_number_base.h | 1 +
be/src/vec/exprs/vectorized_agg_fn.cpp | 7 +-
be/src/vec/exprs/vexpr.cpp | 11 +-
be/src/vec/olap/vgeneric_iterators.cpp | 2 +-
be/src/vec/sink/mysql_result_writer.cpp| 106 +++-
be/src/vec/sink/mysql_result_writer.h | 3 +-
be/test/vec/core/CMakeLists.txt| 1 +
be/test/vec/core/column_array_test.cpp | 85 +++
be/test/vec/exec/vgeneric_iterators_test.cpp | 2 +-
31 files changed, 1830 insertions(+), 359 deletions(-)
diff --git a/be/src/olap/column_vector.h b/be/src/olap/column_vector.h
index ceb..302773e 100644
--- a/be/src/olap/column_vector.h
+++ b/be/src/olap/column_vector.h
@@ -69,7 +69,7 @@ public:
bool is_nullable() const { return _nullable; }
-bool is_null_at(size_t row_idx) { return _nullable &&
_null_signs[row_idx]; }
+bool is_null_at(size_t row_idx) const { return _nullable &&
_null_signs[row_idx]; }
void set_is_null(size_t idx, bool is_null) {
if (_nullable) {
diff --git a/be/src/olap/field.h b/be/src/olap/field.h
index 48fc9b5..d4fedf2 100644
--- a/be/src/olap/field.h
+++ b/be/src/olap/field.h
@@ -298,7 +298,8 @@ public:
void add_sub_field(std::unique_ptr sub_field) {
_sub_fields.emplace_back(std::move(sub_field));
}
-Field* get_sub_field(int i) { return _sub_fields[i].get(); }
+Field* get_sub_field(int i) const { return _sub_fields[i].get(); }
+size_t get_sub_field_count() const { return _sub_fields.size(); }
protected:
std::shared_ptr _type_info;
diff --git a/be/src/olap/row_block2.cpp b/be/src/
[incubator-doris] 04/05: [feature-wip][array-type] Array data can be loaded in stream load. (#8368)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8dba572bfa817f24cb6f6a22f3d108432afec27e
Author: Adonis Ling
AuthorDate: Thu Mar 10 09:15:48 2022 +0800
[feature-wip][array-type] Array data can be loaded in stream load. (#8368)
---
be/src/exprs/cast_functions.cpp| 7 +
be/src/exprs/cast_functions.h | 2 +
be/src/util/array_parser.hpp | 212 +
be/test/util/CMakeLists.txt| 1 +
be/test/util/array_parser_test.cpp | 134 +
.../java/org/apache/doris/analysis/CastExpr.java | 19 +-
.../main/java/org/apache/doris/catalog/Type.java | 4 +-
7 files changed, 371 insertions(+), 8 deletions(-)
diff --git a/be/src/exprs/cast_functions.cpp b/be/src/exprs/cast_functions.cpp
index baadd09..8903d5b 100644
--- a/be/src/exprs/cast_functions.cpp
+++ b/be/src/exprs/cast_functions.cpp
@@ -26,6 +26,7 @@
#include "runtime/datetime_value.h"
#include "runtime/string_value.h"
#include "string_functions.h"
+#include "util/array_parser.hpp"
#include "util/mysql_global.h"
#include "util/string_parser.hpp"
@@ -357,4 +358,10 @@ DateTimeVal
CastFunctions::cast_to_date_val(FunctionContext* ctx, const StringVa
return result;
}
+CollectionVal CastFunctions::cast_to_array_val(FunctionContext* context, const
StringVal& val) {
+CollectionVal array_val;
+Status status = ArrayParser::parse(array_val, context, val);
+return status.ok() ? array_val : CollectionVal::null();
+}
+
} // namespace doris
diff --git a/be/src/exprs/cast_functions.h b/be/src/exprs/cast_functions.h
index 0160275..fa75d7c 100644
--- a/be/src/exprs/cast_functions.h
+++ b/be/src/exprs/cast_functions.h
@@ -136,6 +136,8 @@ public:
static DateTimeVal cast_to_date_val(FunctionContext* context, const
DoubleVal& val);
static DateTimeVal cast_to_date_val(FunctionContext* context, const
DateTimeVal& val);
static DateTimeVal cast_to_date_val(FunctionContext* context, const
StringVal& val);
+
+static CollectionVal cast_to_array_val(FunctionContext* context, const
StringVal& val);
};
} // namespace doris
diff --git a/be/src/util/array_parser.hpp b/be/src/util/array_parser.hpp
new file mode 100644
index 000..746695a
--- /dev/null
+++ b/be/src/util/array_parser.hpp
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+
+#include
+
+#include "common/status.h"
+#include "exprs/anyval_util.h"
+#include "runtime/collection_value.h"
+#include "runtime/primitive_type.h"
+#include "runtime/types.h"
+#include "util/mem_util.hpp"
+
+namespace doris {
+
+template
+using ConstArray = typename rapidjson::GenericValue::ConstArray;
+
+template
+using ConstArrayIterator = typename ConstArray::ValueIterator;
+
+class ArrayParser {
+public:
+static Status parse(CollectionVal& array_val, FunctionContext* context,
+const StringVal& str_val) {
+rapidjson::Document document;
+if (document.Parse(reinterpret_cast(str_val.ptr),
str_val.len).HasParseError() ||
+!document.IsArray()) {
+return Status::RuntimeError("Failed to parse the json to array.");
+}
+if (document.IsNull()) {
+array_val = CollectionVal::null();
+return Status::OK();
+}
+auto type_desc =
_convert_to_type_descriptor(context->get_return_type());
+return _parse>(
+array_val, context,
+reinterpret_cast(&document)->GetArray(), type_desc);
+}
+
+private:
+static TypeDescriptor _convert_to_type_descriptor(
+FunctionContext::TypeDesc function_type_desc) {
+auto iterator = _types_mapping.find(function_type_desc.type);
+if (iterator == _types_mapping.end()) {
+return TypeDescriptor();
+}
+auto type_desc = TypeDescriptor(iterator->second);
+type_desc.len = function_type_desc.len;
+type_desc.precision = function_type_desc.precision;
+type_desc.scale = fun
[incubator-doris] branch array-type updated (b67314d -> 3f74fb6)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch array-type in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. discard b67314d [feature-wip][array-type] Support nested array insertion. (#8305) omit 07d16d4 [feature-wip][array-type] Array data can be loaded in stream load. (#8368) omit 30e93e0 [feature-wip](array-type)support select ARRAY data type on vectorized engine (#8217) omit 28ea295 [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089) omit 3e2d90d [feature-wip](array-type) Create table with nested array type. (#8003) add d9c2c2c Revert "[refactor] remove unused new_in_predicate code (#8263)" (#8372) add c647def [community] Modify doris connector release doc (#8275) add ca3be24 [typo]fix some typo in fe_config (#8325) add 50a59f3 [license] Organize third-party dependent licenses for bianry releases (#8350) add 089ef53 [chore] fix build with parallel parameter only (#8352) add d711d64 [fix](vectorization)Some small fix for SegmentIter Vectorization (#8267) add 454b45b [feature](vectorize)(function) support regexp&&sm4&&aes functions (#8307) add 3eedd15 [optimize] optimze tablet read, avoid to create too much scanner for small tablet (#8096) add b9010e3 [doc] Translate Chinese comment to English (#8340) add 1e70f99 [improvement][fix](insert)(replay) support SHOW LAST INSERT stmt and fix json replay bug (#8355) add 22bafef [fix](broker-load) fix bug that a cancelled job's state is LOADING (#8363) add 9c7d519 [improvement](regression-test) add aggregation tests from trino to doris (#8375) add 8214f32 [fix] fix core dump on minmax_filter with decimal type (#8381) add cd8694e [feature][vectorized] support replace() (#8384) add 40a5fac [fix](vectorized) Fix the datetime type read error and is_same set error in reader (#8386) add 58e8537 [fix](vectorized) fix float to string inaccurate (#8392) add b536c18 [fix] fix wrong examples in week.md (#8397) add 826467e [fix](replica) handle replica version missing info to avoid -214 error (#8209) add 10c3712 [fix](vectorized) fix arithmetic calculate get wrong result(#8226) add 0ff7de4 [refactor] remove agent status (#8273) add 3d30209 [docs] add document conditional-functions (#8339) add d880559 [refactor] remove old schema change code on BE (#8342) add 25532c6 [doc] Update BROKER LOAD.md (#8361) add 97020bc [doc] update substring.md (#8398) add 51103dc [typo] translate the comments of delete_handler.cpp (#8402) add f4663ad [improvement](vectorized) Merge block in scanner to speed up query with conjunct (#8395) add 2a43313 [fix] fix compile error (#8410) add 8eec4bf [feature](thread-local) Add thread local variable ThreadContext (#7234) add 8c84160 [doc](vectorized) Add vectorized execution engine docs (#8358) add 548fde6 [doc] Fix dead link in build docker environment in developer guide (#8379) add d3d8301 [feature](function) support vectorized digital_masking (#8409) new b4f227e [feature-wip](array-type) Create table with nested array type. (#8003) new 5ad6472 [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089) new d041fd0 [feature-wip](array-type)support select ARRAY data type on vectorized engine (#8217) new 8dba572 [feature-wip][array-type] Array data can be loaded in stream load. (#8368) new 3f74fb6 [feature-wip][array-type] Support nested array insertion. (#8305) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b67314d) \ N -- N -- N refs/heads/array-type (3f74fb6) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: NOTICE.txt |2 +- be/src/agent/cgroups_mgr.cpp | 99 +- be/src/agent/cgroups_mgr.h | 20 +- be/src/agent/heartbeat_server.cpp |6 +- be/src/agent/heartbeat_server.h|4 +- be/src/agent/status.h | 48 - be/src/agent/task_worker_pool.cpp
[incubator-doris] branch array-type updated: [feature-wip][array-type] Support nested array insertion. (#8305)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/array-type by this push:
new b67314d [feature-wip][array-type] Support nested array insertion.
(#8305)
b67314d is described below
commit b67314d345fec126bdce3d152b8436a0c5cace6b
Author: Adonis Ling
AuthorDate: Thu Mar 10 10:51:26 2022 +0800
[feature-wip][array-type] Support nested array insertion. (#8305)
---
be/src/exec/olap_scanner.cpp | 43 +-
be/src/olap/aggregate_func.cpp | 2 +
be/src/olap/field.h| 4 +-
be/src/olap/rowset/segment_v2/column_reader.cpp| 9 +-
be/src/olap/types.cpp | 13 +
be/src/olap/types.h| 49 +-
be/src/runtime/collection_value.cpp| 241 +++--
be/src/runtime/collection_value.h | 38 +-
be/src/runtime/mysql_result_writer.cpp | 17 +-
be/src/runtime/raw_value.cpp | 10 +-
be/src/runtime/row_batch.cpp | 50 +-
be/src/runtime/tuple.cpp | 129 ++---
be/test/runtime/CMakeLists.txt | 1 +
be/test/runtime/array_test.cpp | 556 +
.../main/java/org/apache/doris/catalog/Column.java | 6 +
15 files changed, 927 insertions(+), 241 deletions(-)
diff --git a/be/src/exec/olap_scanner.cpp b/be/src/exec/olap_scanner.cpp
index d7dc839..f480135 100644
--- a/be/src/exec/olap_scanner.cpp
+++ b/be/src/exec/olap_scanner.cpp
@@ -353,39 +353,16 @@ Status OlapScanner::get_batch(RuntimeState* state,
RowBatch* batch, bool* eof) {
// Copy collection slot
for (auto desc : _parent->_collection_slots) {
CollectionValue* slot =
tuple->get_collection_slot(desc->tuple_offset());
-
-TypeDescriptor item_type = desc->type().children.at(0);
-size_t item_size = item_type.get_slot_size() *
slot->length();
-
-size_t nulls_size = slot->length();
-uint8_t* data =
batch->tuple_data_pool()->allocate(item_size + nulls_size);
-
-// copy null_signs
-memory_copy(data, slot->null_signs(), nulls_size);
-memory_copy(data + nulls_size, slot->data(), item_size);
-
-slot->set_null_signs(reinterpret_cast(data));
-slot->set_data(reinterpret_cast(data + nulls_size));
-
-if (!item_type.is_string_type()) {
-continue;
-}
-
-// when string type, copy every item
-for (int i = 0; i < slot->length(); ++i) {
-int item_offset = nulls_size + i *
item_type.get_slot_size();
-if (slot->is_null_at(i)) {
-continue;
-}
-StringValue* dst_item_v =
-reinterpret_cast(data +
item_offset);
-if (dst_item_v->len != 0) {
-char* string_copy = reinterpret_cast(
-
batch->tuple_data_pool()->allocate(dst_item_v->len));
-memory_copy(string_copy, dst_item_v->ptr,
dst_item_v->len);
-dst_item_v->ptr = string_copy;
-}
-}
+const TypeDescriptor& item_type =
desc->type().children.at(0);
+auto pool = batch->tuple_data_pool();
+CollectionValue::deep_copy_collection(
+slot, item_type, [pool](int size) -> MemFootprint {
+int64_t offset = pool->total_allocated_bytes();
+uint8_t* data = pool->allocate(size);
+return { offset, data };
+},
+false
+);
}
// the memory allocate by mem pool has been copied,
// so we should release these memory immediately
diff --git a/be/src/olap/aggregate_func.cpp b/be/src/olap/aggregate_func.cpp
index f3c6d8b..6e844e7 100644
--- a/be/src/olap/aggregate_func.cpp
+++ b/be/src/olap/aggregate_func.cpp
@@ -119,6 +119,8 @@ AggregateFuncResolver::AggregateFuncResolver() {
OLAP_FIELD_TYPE_VARCHAR>();
add_aggregate_mapping();
+add_aggregate_mapping();
// Min Aggregate Function
add_aggregate_mapping();
diff --git a/be/src/olap/field.h b/be/src/olap/field.h
index d4fedf2..e3b35da 100644
--- a/be/src/olap/field.h
+++ b/be/src/olap/field.h
@@ -454,9 +454,7 @@ public:
char* all
[GitHub] [incubator-doris] morningman merged pull request #8305: [feature-wip][array-type] Support nested array insertion.
morningman merged pull request #8305: URL: https://github.com/apache/incubator-doris/pull/8305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 02/05: [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5ad647222afc8a31ae1ae03a5ccf7e794bfda196
Author: Adonis Ling
AuthorDate: Fri Feb 18 10:59:57 2022 +0800
[feature-wip][array-type] Support ArrayLiteral in SQL. (#8089)
Please refer to #8074
---
be/src/olap/column_vector.cpp | 21 ++--
be/src/olap/rowset/segment_v2/column_reader.cpp| 8 +-
be/src/olap/rowset/segment_v2/column_reader.h | 13 +++
be/src/olap/rowset/segment_v2/column_writer.cpp| 10 +-
be/src/olap/rowset/segment_v2/column_writer.h | 5 +-
be/src/runtime/collection_value.cpp| 16 ++-
be/src/runtime/mysql_result_writer.cpp | 6 +-
be/src/runtime/raw_value.cpp | 1 +
be/src/runtime/row_batch.cpp | 10 +-
be/src/runtime/tuple.cpp | 29 +++---
.../segment_v2/column_reader_writer_test.cpp | 24 +
fe/fe-core/src/main/cup/sql_parser.cup | 14 +++
.../org/apache/doris/analysis/ArrayLiteral.java| 28 +++--
.../org/apache/doris/analysis/CreateTableStmt.java | 10 +-
.../main/java/org/apache/doris/analysis/Expr.java | 3 +-
.../java/org/apache/doris/catalog/ArrayType.java | 4 +
.../main/java/org/apache/doris/catalog/Column.java | 13 ++-
.../main/java/org/apache/doris/catalog/Type.java | 4 +-
.../org/apache/doris/analysis/ColumnDefTest.java | 18 +++-
.../apache/doris/analysis/InsertArrayStmtTest.java | 114 +
.../org/apache/doris/utframe/UtFrameUtils.java | 9 ++
21 files changed, 306 insertions(+), 54 deletions(-)
diff --git a/be/src/olap/column_vector.cpp b/be/src/olap/column_vector.cpp
index 3340628..0237b62 100644
--- a/be/src/olap/column_vector.cpp
+++ b/be/src/olap/column_vector.cpp
@@ -210,14 +210,19 @@ void ArrayColumnVectorBatch::prepare_for_read(size_t
start_idx, size_t size, boo
DCHECK(start_idx + size <= capacity());
for (size_t i = 0; i < size; ++i) {
if (!is_null_at(start_idx + i)) {
-_data[start_idx + i] = CollectionValue(
-
_elements->mutable_cell_ptr(*(_offsets->scalar_cell_ptr(start_idx + i))),
-*(_offsets->scalar_cell_ptr(start_idx + i + 1)) -
-*(_offsets->scalar_cell_ptr(start_idx + i)),
-item_has_null,
-_elements->is_nullable() ?
const_cast(&_elements->null_signs()[*(
-
_offsets->scalar_cell_ptr(start_idx + i))])
- : nullptr);
+auto next_offset = *(_offsets->scalar_cell_ptr(start_idx + i + 1));
+auto offset = *(_offsets->scalar_cell_ptr(start_idx + i));
+uint32_t length = next_offset - offset;
+if (length == 0) {
+_data[start_idx + i] = CollectionValue(length);
+} else {
+_data[start_idx + i] = CollectionValue(
+_elements->mutable_cell_ptr(offset),
+length,
+item_has_null,
+_elements->is_nullable() ?
const_cast(&_elements->null_signs()[offset])
+ : nullptr);
+}
}
}
}
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 20d2918..9530d80 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -127,7 +127,7 @@ Status ColumnReader::init() {
"Bad file $0: invalid column index type $1",
_path_desc.filepath, index_meta.type()));
}
}
-if (_ordinal_index_meta == nullptr) {
+if (!is_empty() && _ordinal_index_meta == nullptr) {
return Status::Corruption(strings::Substitute(
"Bad file $0: missing ordinal index for column $1",
_path_desc.filepath, _meta.column_id()));
}
@@ -339,6 +339,10 @@ Status ColumnReader::seek_at_or_before(ordinal_t ordinal,
OrdinalPageIndexIterat
}
Status ColumnReader::new_iterator(ColumnIterator** iterator) {
+if (is_empty()) {
+*iterator = new EmptyFileColumnIterator();
+return Status::OK();
+}
if (is_scalar_type((FieldType)_meta.type())) {
*iterator = new FileColumnIterator(this);
return Status::OK();
@@ -427,7 +431,7 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
// read item
size_t item_size = array_batch->get_item_size(dst->current_offset(), *n);
-if (item_size > 0) {
+if (item_size >= 0) {
bool item_has_null = false;
ColumnVectorBatch* item_vector_batch = array_batch->elements();
diff --git a/be/src/olap/rowset/segment
[incubator-doris] 05/05: [feature-wip][array-type] Support nested array insertion. (#8305)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 3f74fb619c8f7e514919e712be17a17070ad97ff
Author: Adonis Ling
AuthorDate: Thu Mar 10 10:51:26 2022 +0800
[feature-wip][array-type] Support nested array insertion. (#8305)
---
be/src/exec/olap_scanner.cpp | 43 +-
be/src/olap/aggregate_func.cpp | 2 +
be/src/olap/field.h| 4 +-
be/src/olap/rowset/segment_v2/column_reader.cpp| 9 +-
be/src/olap/types.cpp | 13 +
be/src/olap/types.h| 49 +-
be/src/runtime/collection_value.cpp| 241 +++--
be/src/runtime/collection_value.h | 38 +-
be/src/runtime/mysql_result_writer.cpp | 17 +-
be/src/runtime/raw_value.cpp | 10 +-
be/src/runtime/row_batch.cpp | 50 +-
be/src/runtime/tuple.cpp | 129 ++---
be/test/runtime/CMakeLists.txt | 1 +
be/test/runtime/array_test.cpp | 556 +
.../main/java/org/apache/doris/catalog/Column.java | 6 +
15 files changed, 927 insertions(+), 241 deletions(-)
diff --git a/be/src/exec/olap_scanner.cpp b/be/src/exec/olap_scanner.cpp
index d7dc839..f480135 100644
--- a/be/src/exec/olap_scanner.cpp
+++ b/be/src/exec/olap_scanner.cpp
@@ -353,39 +353,16 @@ Status OlapScanner::get_batch(RuntimeState* state,
RowBatch* batch, bool* eof) {
// Copy collection slot
for (auto desc : _parent->_collection_slots) {
CollectionValue* slot =
tuple->get_collection_slot(desc->tuple_offset());
-
-TypeDescriptor item_type = desc->type().children.at(0);
-size_t item_size = item_type.get_slot_size() *
slot->length();
-
-size_t nulls_size = slot->length();
-uint8_t* data =
batch->tuple_data_pool()->allocate(item_size + nulls_size);
-
-// copy null_signs
-memory_copy(data, slot->null_signs(), nulls_size);
-memory_copy(data + nulls_size, slot->data(), item_size);
-
-slot->set_null_signs(reinterpret_cast(data));
-slot->set_data(reinterpret_cast(data + nulls_size));
-
-if (!item_type.is_string_type()) {
-continue;
-}
-
-// when string type, copy every item
-for (int i = 0; i < slot->length(); ++i) {
-int item_offset = nulls_size + i *
item_type.get_slot_size();
-if (slot->is_null_at(i)) {
-continue;
-}
-StringValue* dst_item_v =
-reinterpret_cast(data +
item_offset);
-if (dst_item_v->len != 0) {
-char* string_copy = reinterpret_cast(
-
batch->tuple_data_pool()->allocate(dst_item_v->len));
-memory_copy(string_copy, dst_item_v->ptr,
dst_item_v->len);
-dst_item_v->ptr = string_copy;
-}
-}
+const TypeDescriptor& item_type =
desc->type().children.at(0);
+auto pool = batch->tuple_data_pool();
+CollectionValue::deep_copy_collection(
+slot, item_type, [pool](int size) -> MemFootprint {
+int64_t offset = pool->total_allocated_bytes();
+uint8_t* data = pool->allocate(size);
+return { offset, data };
+},
+false
+);
}
// the memory allocate by mem pool has been copied,
// so we should release these memory immediately
diff --git a/be/src/olap/aggregate_func.cpp b/be/src/olap/aggregate_func.cpp
index f3c6d8b..6e844e7 100644
--- a/be/src/olap/aggregate_func.cpp
+++ b/be/src/olap/aggregate_func.cpp
@@ -119,6 +119,8 @@ AggregateFuncResolver::AggregateFuncResolver() {
OLAP_FIELD_TYPE_VARCHAR>();
add_aggregate_mapping();
+add_aggregate_mapping();
// Min Aggregate Function
add_aggregate_mapping();
diff --git a/be/src/olap/field.h b/be/src/olap/field.h
index d4fedf2..e3b35da 100644
--- a/be/src/olap/field.h
+++ b/be/src/olap/field.h
@@ -454,9 +454,7 @@ public:
char* allocate_memory(char* cell_ptr, char* variable_ptr) const override {
auto array_v = (CollectionValue*)cell_ptr;
-array_v->set_null_signs(reinterpret_cast(variable_ptr +
size
[GitHub] [incubator-doris] adonis0147 closed issue #8367: [feature-wip][array-type] Array data can be loaded in stream load.
adonis0147 closed issue #8367: URL: https://github.com/apache/incubator-doris/issues/8367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 closed issue #8304: [feature-wip][array-type] Support nested array insertion.
adonis0147 closed issue #8304: URL: https://github.com/apache/incubator-doris/issues/8304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on a change in pull request #8387: [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex
weizuo93 commented on a change in pull request #8387:
URL: https://github.com/apache/incubator-doris/pull/8387#discussion_r823300070
##
File path: be/src/olap/txn_manager.cpp
##
@@ -357,7 +357,7 @@ OLAPStatus TxnManager::delete_txn(OlapMeta* meta,
TPartitionId partition_id,
SchemaHash schema_hash, TabletUid
tablet_uid) {
pair key(partition_id, transaction_id);
TabletInfo tablet_info(tablet_id, schema_hash, tablet_uid);
-WriteLock txn_wrlock(&_get_txn_map_lock(transaction_id));
+WriteLock txn_wlock(_get_txn_map_lock(transaction_id));
Review comment:
Is `txn_wrlock`better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on a change in pull request #8387: [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex
weizuo93 commented on a change in pull request #8387:
URL: https://github.com/apache/incubator-doris/pull/8387#discussion_r823315857
##
File path: be/src/util/thread_group.h
##
@@ -61,7 +61,7 @@ class ThreadGroup {
template
std::thread* create_thread(F threadfunc) {
-std::lock_guard guard(_mutex);
+WriteLock wrlock(_mutex);
Review comment:
Why do you change `std::lock_guard` to
`std::unique_lock` ? I think unique_lock may be slower than
lock_guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] taberylyang commented on a change in pull request #8393: 1.support shared hash table for broadcast hash join,use enable_shared…
taberylyang commented on a change in pull request #8393: URL: https://github.com/apache/incubator-doris/pull/8393#discussion_r823322481 ## File path: be/src/exprs/shared_hash_table.cpp ## @@ -0,0 +1,166 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "runtime/shared_hash_table.h" Review comment: you are right,i will put them in runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] huangyuansheng opened a new issue #8421: [Enhancement] Recover DDL
huangyuansheng opened a new issue #8421: URL: https://github.com/apache/incubator-doris/issues/8421 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description It's serious that overwrite the previous dropped object if drop same object again. caution: when create same object after dropped, if wanna recover it, you should rename first 对于多次删除相同对象会导致垃圾桶中的对象被覆盖(只能恢复最近一次删除)，个人感觉还是蛮危险的事情。 注意：对于删除后又新建同名对象，此时想要恢复之前的对象，请先rename！ ### Solution * dropped objects with timestamp in trash, it can be query using like 'show dropped' * fields: id, type(db or table or partition), owner, name, drop_time * recover ddl: recover [ using id ] [ to newName ] * can recover to a new name * 删除对象可以带时间戳保留在垃圾箱中，并可以通过类似show dropped命令查看 * 大致字段：id, type(db or table or partition), owner, name, drop_time * 恢复语句可考虑增强：recover [ using id ] [ to newName ] * 支持恢复到新的名称 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 opened a new pull request #8422: [feature-wip][array-type] Fix compilation error.
adonis0147 opened a new pull request #8422: URL: https://github.com/apache/incubator-doris/pull/8422 ## Problem Summary: Fix compilation error. ## Checklist(Required) 1. Does it affect the original behavior: No 2. Has unit tests been added: No Need 3. Has document been added or modified: No Need 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8233: [Vectorized][HashJoin] add probe timer
HappenLee commented on a change in pull request #8233:
URL: https://github.com/apache/incubator-doris/pull/8233#discussion_r823328678
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -166,8 +166,58 @@ struct ProcessHashTableProbe {
_items_counts(join_node->_items_counts),
_build_block_offsets(join_node->_build_block_offsets),
_build_block_rows(join_node->_build_block_rows),
- _rows_returned_counter(join_node->_rows_returned_counter) {}
+ _rows_returned_counter(join_node->_rows_returned_counter),
+ _search_hashtable_timer(join_node->_search_hashtable_timer),
+ _build_side_output_timer(join_node->_build_side_output_timer),
+ _probe_side_output_timer(join_node->_probe_side_output_timer) {}
+
+// output build side result column
+Status build_side_output_column(MutableColumns& mcol, int column_offset,
int column_length, int size) {
+constexpr auto is_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
Review comment:
The Function only return OK ？ seems not need to return Status.
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -166,8 +166,58 @@ struct ProcessHashTableProbe {
_items_counts(join_node->_items_counts),
_build_block_offsets(join_node->_build_block_offsets),
_build_block_rows(join_node->_build_block_rows),
- _rows_returned_counter(join_node->_rows_returned_counter) {}
+ _rows_returned_counter(join_node->_rows_returned_counter),
+ _search_hashtable_timer(join_node->_search_hashtable_timer),
+ _build_side_output_timer(join_node->_build_side_output_timer),
+ _probe_side_output_timer(join_node->_probe_side_output_timer) {}
+
+// output build side result column
+Status build_side_output_column(MutableColumns& mcol, int column_offset,
int column_length, int size) {
+constexpr auto is_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
+ JoinOpType::value ==
TJoinOp::RIGHT_SEMI_JOIN ||
+ JoinOpType::value ==
TJoinOp::LEFT_ANTI_JOIN ||
+ JoinOpType::value ==
TJoinOp::LEFT_SEMI_JOIN;
+constexpr auto probe_all = JoinOpType::value ==
TJoinOp::LEFT_OUTER_JOIN ||
+ JoinOpType::value ==
TJoinOp::FULL_OUTER_JOIN;
+
+if constexpr (!is_semi_anti_join) {
+if (_build_blocks.size() == 1) {
+for (int i = 0; i < column_length; i++) {
+auto& column = *_build_blocks[0].get_by_position(i).column;
+mcol[i + column_offset]->insert_indices_from(column,
+_build_block_rows.data(), _build_block_rows.data()
+ size);
+}
+} else {
+for (int i = 0; i < column_length; i++) {
+for (int j = 0; j < size; j++) {
+if constexpr (probe_all) {
+if (_build_block_offsets[j] == -1) {
+DCHECK(mcol[i + column_offset]->is_nullable());
+assert_cast(mcol[i +
column_offset].get())->insert_data(nullptr, 0);
+} else {
+auto& column =
*_build_blocks[_build_block_offsets[j]].get_by_position(i).column;
+mcol[i + column_offset]->insert_from(column,
_build_block_rows[j]);
+}
+} else {
+auto& column =
*_build_blocks[_build_block_offsets[j]].get_by_position(i).column;
+mcol[i + column_offset]->insert_from(column,
_build_block_rows[j]);
+}
+}
+}
+}
+}
+return Status::OK();
+}
+
+// output probe side result column
+Status probe_side_output_column(MutableColumns& mcol, int column_length,
int size) {
Review comment:
same to the up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #8412: [Enhancement](function)Support function YEARWEEK(DATE,INT,INT) RETURN INT
HappenLee commented on pull request #8412: URL: https://github.com/apache/incubator-doris/pull/8412#issuecomment-1063646337 > > please support it in vec exec engine too， thank you > > I had planned to implement vectorization for this func in another PR. That's all right, I will add vectorization code in this PR. another PR is OK, thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #8423: [Fix](transaction) Fix committed transaction couldn't be finished when table is dropped
caiconghui opened a new pull request #8423: URL: https://github.com/apache/incubator-doris/pull/8423 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-spark-connector] hf200012 merged pull request #13: [chore] fix name bug in build.sh
hf200012 merged pull request #13: URL: https://github.com/apache/incubator-doris-spark-connector/pull/13 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-spark-connector] branch master updated: [chore] fix name bug in build.sh (#13)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/incubator-doris-spark-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 268567f [chore] fix name bug in build.sh (#13)
268567f is described below
commit 268567f1c04be0b5cd553ad9c3e13940defe4d8c
Author: Mingyu Chen
AuthorDate: Thu Mar 10 13:04:48 2022 +0800
[chore] fix name bug in build.sh (#13)
fix name bug in build.sh
---
spark-doris-connector/build.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/spark-doris-connector/build.sh b/spark-doris-connector/build.sh
index 30d20a0..b301289 100755
--- a/spark-doris-connector/build.sh
+++ b/spark-doris-connector/build.sh
@@ -95,7 +95,7 @@ else
fi
mkdir ${ROOT}/output/
-cp ${ROOT}/target/doris-spark-*.jar ${ROOT}/output/
+cp ${ROOT}/target/spark-doris-*.jar ${ROOT}/output/
echo "*"
echo "Successfully build Spark-Doris-Connector"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch array-type updated: [feature-wip][array-type] Fix compilation error. (#8422)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/array-type by this push:
new 8278759 [feature-wip][array-type] Fix compilation error. (#8422)
8278759 is described below
commit 82787594d94d5a2645e6ba36874300c2eb13b4db
Author: Adonis Ling
AuthorDate: Thu Mar 10 12:51:27 2022 +0800
[feature-wip][array-type] Fix compilation error. (#8422)
---
be/src/util/array_parser.hpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/util/array_parser.hpp b/be/src/util/array_parser.hpp
index 746695a..83ef7da 100644
--- a/be/src/util/array_parser.hpp
+++ b/be/src/util/array_parser.hpp
@@ -92,7 +92,7 @@ private:
} else if (!_is_type_valid(it, item_type)) {
return Status::RuntimeError("Failed to parse the json to
array.");
}
-AnyVal* val;
+AnyVal* val = nullptr;
Status status = _parse(&val, context, it,
child_type_desc);
if (!status.ok()) {
return status;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8422: [feature-wip][array-type] Fix compilation error.
morningman merged pull request #8422: URL: https://github.com/apache/incubator-doris/pull/8422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on a change in pull request #8408: [Benchmark] Add TPC-H benchmark tools
hf200012 commented on a change in pull request #8408: URL: https://github.com/apache/incubator-doris/pull/8408#discussion_r822529348 ## File path: tools/tpch-tools/README.md ## @@ -0,0 +1,15 @@ +## Usage Review comment: add License head -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8403: [doc] fix metadata operation doc links to external docs
github-actions[bot] commented on pull request #8403: URL: https://github.com/apache/incubator-doris/pull/8403#issuecomment-1063669295 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8408: [Benchmark] Add TPC-H benchmark tools
github-actions[bot] commented on pull request #8408: URL: https://github.com/apache/incubator-doris/pull/8408#issuecomment-1063669556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangjun616 opened a new pull request #8424: Update load-manual.md
yangjun616 opened a new pull request #8424: URL: https://github.com/apache/incubator-doris/pull/8424 typo # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] bridgeDream opened a new pull request #18: [improvement] Support set max bytes in each batch to avoid congestion
bridgeDream opened a new pull request #18: URL: https://github.com/apache/incubator-doris-flink-connector/pull/18 ## Problem Summary: 目前，dorisSink 刷入 batch 的时机依赖 "batchSize" 和 "batchIntervalMs"，一般情形下是能够符合预期的；但是当数据源流量或者单条消息数据量极大时，可能会造成单个 batch 的数据很大，进而会导致数据刷入 doris 时长时间阻塞影响实时性能，甚至请求 BE 超时无法正常写入的问题。 出于上述原因，提供依据 "maxBatchBytes" 刷 batch 到 doris 中，修改点： * 类 `DorisExecutionOptions`: 增加一个可配置选项 `maxBatchBytes` * 类 `DorisDynamicOutputFormat`: 估计当前 batch 的数据量 `batchBytes` 达到阈值 `maxBatchBytes` 时，刷入 doris。 ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8423: [Fix](transaction) Fix committed transaction couldn't be finished when table is dropped
wangbo commented on a change in pull request #8423:
URL: https://github.com/apache/incubator-doris/pull/8423#discussion_r823358665
##
File path:
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
##
@@ -793,19 +793,7 @@ public void finishTransaction(long transactionId,
Set errorReplicaIds) thr
errorReplicaIds.addAll(originalErrorReplicas);
}
-Database db = catalog.getDbNullable(transactionState.getDbId());
-if (db == null) {
-writeLock();
-try {
-
transactionState.setTransactionStatus(TransactionStatus.ABORTED);
-transactionState.setReason("db is dropped");
-LOG.warn("db is dropped during transaction, abort transaction
{}", transactionState);
-unprotectUpsertTransactionState(transactionState, false);
-return;
-} finally {
-writeUnlock();
-}
-}
+Database db = catalog.getDbOrMetaException(transactionState.getDbId());
Review comment:
Why not abort transaction when db is droped? Because it maybe recovered?
##
File path:
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
##
@@ -793,19 +793,7 @@ public void finishTransaction(long transactionId,
Set errorReplicaIds) thr
errorReplicaIds.addAll(originalErrorReplicas);
}
-Database db = catalog.getDbNullable(transactionState.getDbId());
-if (db == null) {
-writeLock();
-try {
-
transactionState.setTransactionStatus(TransactionStatus.ABORTED);
-transactionState.setReason("db is dropped");
-LOG.warn("db is dropped during transaction, abort transaction
{}", transactionState);
-unprotectUpsertTransactionState(transactionState, false);
-return;
-} finally {
-writeUnlock();
-}
-}
+Database db = catalog.getDbOrMetaException(transactionState.getDbId());
Review comment:
Why not abort transaction when db is dropped? Because it maybe recovered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #8423: [Fix](transaction) Fix committed transaction couldn't be finished when table is dropped
caiconghui commented on a change in pull request #8423:
URL: https://github.com/apache/incubator-doris/pull/8423#discussion_r823366808
##
File path:
fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
##
@@ -793,19 +793,7 @@ public void finishTransaction(long transactionId,
Set errorReplicaIds) thr
errorReplicaIds.addAll(originalErrorReplicas);
}
-Database db = catalog.getDbNullable(transactionState.getDbId());
-if (db == null) {
-writeLock();
-try {
-
transactionState.setTransactionStatus(TransactionStatus.ABORTED);
-transactionState.setReason("db is dropped");
-LOG.warn("db is dropped during transaction, abort transaction
{}", transactionState);
-unprotectUpsertTransactionState(transactionState, false);
-return;
-} finally {
-writeUnlock();
-}
-}
+Database db = catalog.getDbOrMetaException(transactionState.getDbId());
Review comment:
committed txn will be checked when db to be dropped. it will be aborted
outside
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #8425: [chore] disable librdkafka assert and update some thirdparty
yangzhg opened a new pull request #8425: URL: https://github.com/apache/incubator-doris/pull/8425 # Proposed changes 1. comment librdkafka `rd_assert(thrd_is_current(rkb->rkb_thread));` to avoid core dump 2. upgrade arrow to 7.0.0 3. upgrade aws sdk to 1.9 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 4. Has unit tests been added: (No Need) 5. Has document been added or modified: (No Need) 6. Does it need to update dependencies: (Yes) 7. Are there any changes that cannot be rolled back: (Yes) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #8426: [Bug] Transaction may not be finished when table does not exist
caiconghui opened a new issue #8426: URL: https://github.com/apache/incubator-doris/issues/8426 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Transaction may not be finished when table does not exist ### What You Expected? Transaction can be finished when table does not exist ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on pull request #8371: Update hive support version，hive：3.1.2
EmmyMiao87 commented on pull request #8371: URL: https://github.com/apache/incubator-doris/pull/8371#issuecomment-1063719413 Please change the comment title ~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8371: [Doc}Update hive support version，hive：3.1.2
github-actions[bot] commented on pull request #8371: URL: https://github.com/apache/incubator-doris/pull/8371#issuecomment-1063720119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] emmataobao commented on issue #8331: [Bug] doris on es 中 连接es ip问题
emmataobao commented on issue #8331: URL: https://github.com/apache/incubator-doris/issues/8331#issuecomment-1063724436 > priority_networks 谢谢，我已经知道我部署的问题在哪里，es的问题我也知道了。我配置参数没有问题，主要部署的测试es是单节点版本所以network.host指向的是docker内部ip，doris的运行逻辑是他会先去拿metadata，他通过我配置的ip端口拿metadata，但是后续的请求会使用metadata的ip和端口号，导致我的配置只能获取metadata，所有的请求没有使用。造成了配置和请求的ip地址不一致问题。谢谢  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] emmataobao closed issue #8331: [Bug] doris on es 中 连接es ip问题
emmataobao closed issue #8331: URL: https://github.com/apache/incubator-doris/issues/8331 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] awakeljw commented on a change in pull request #8233: [Vectorized][HashJoin] add probe timer
awakeljw commented on a change in pull request #8233:
URL: https://github.com/apache/incubator-doris/pull/8233#discussion_r823402079
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -166,8 +166,58 @@ struct ProcessHashTableProbe {
_items_counts(join_node->_items_counts),
_build_block_offsets(join_node->_build_block_offsets),
_build_block_rows(join_node->_build_block_rows),
- _rows_returned_counter(join_node->_rows_returned_counter) {}
+ _rows_returned_counter(join_node->_rows_returned_counter),
+ _search_hashtable_timer(join_node->_search_hashtable_timer),
+ _build_side_output_timer(join_node->_build_side_output_timer),
+ _probe_side_output_timer(join_node->_probe_side_output_timer) {}
+
+// output build side result column
+Status build_side_output_column(MutableColumns& mcol, int column_offset,
int column_length, int size) {
+constexpr auto is_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
Review comment:
i agree.
##
File path: be/src/vec/exec/join/vhash_join_node.cpp
##
@@ -166,8 +166,58 @@ struct ProcessHashTableProbe {
_items_counts(join_node->_items_counts),
_build_block_offsets(join_node->_build_block_offsets),
_build_block_rows(join_node->_build_block_rows),
- _rows_returned_counter(join_node->_rows_returned_counter) {}
+ _rows_returned_counter(join_node->_rows_returned_counter),
+ _search_hashtable_timer(join_node->_search_hashtable_timer),
+ _build_side_output_timer(join_node->_build_side_output_timer),
+ _probe_side_output_timer(join_node->_probe_side_output_timer) {}
+
+// output build side result column
+Status build_side_output_column(MutableColumns& mcol, int column_offset,
int column_length, int size) {
+constexpr auto is_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
+ JoinOpType::value ==
TJoinOp::RIGHT_SEMI_JOIN ||
+ JoinOpType::value ==
TJoinOp::LEFT_ANTI_JOIN ||
+ JoinOpType::value ==
TJoinOp::LEFT_SEMI_JOIN;
+constexpr auto probe_all = JoinOpType::value ==
TJoinOp::LEFT_OUTER_JOIN ||
+ JoinOpType::value ==
TJoinOp::FULL_OUTER_JOIN;
+
+if constexpr (!is_semi_anti_join) {
+if (_build_blocks.size() == 1) {
+for (int i = 0; i < column_length; i++) {
+auto& column = *_build_blocks[0].get_by_position(i).column;
+mcol[i + column_offset]->insert_indices_from(column,
+_build_block_rows.data(), _build_block_rows.data()
+ size);
+}
+} else {
+for (int i = 0; i < column_length; i++) {
+for (int j = 0; j < size; j++) {
+if constexpr (probe_all) {
+if (_build_block_offsets[j] == -1) {
+DCHECK(mcol[i + column_offset]->is_nullable());
+assert_cast(mcol[i +
column_offset].get())->insert_data(nullptr, 0);
+} else {
+auto& column =
*_build_blocks[_build_block_offsets[j]].get_by_position(i).column;
+mcol[i + column_offset]->insert_from(column,
_build_block_rows[j]);
+}
+} else {
+auto& column =
*_build_blocks[_build_block_offsets[j]].get_by_position(i).column;
+mcol[i + column_offset]->insert_from(column,
_build_block_rows[j]);
+}
+}
+}
+}
+}
+return Status::OK();
+}
+
+// output probe side result column
+Status probe_side_output_column(MutableColumns& mcol, int column_length,
int size) {
Review comment:
i agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiluoya opened a new issue #8427: 压缩算法加强[Enhancement]
yiluoya opened a new issue #8427: URL: https://github.com/apache/incubator-doris/issues/8427 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description 目前对于数据的压缩算法是系统统一指定的， 对于部分业务场景，比如日志数据，希望极高的压缩率，对于业务数据，可能需要极高的查询性能，现在的设计无法满足自定义压缩算法， 希望 ： 压缩算法在建表的时候指定，像hbase 一样。 ### Solution 压缩算法在建表的时候指定，像hbase 一样。 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8233: [Vectorized][HashJoin] add probe timer
github-actions[bot] commented on pull request #8233: URL: https://github.com/apache/incubator-doris/pull/8233#issuecomment-1063734886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8428: [Bug][Vectorized] core dump on runtime filter insert because of block address change
BiteThet opened a new issue #8428: URL: https://github.com/apache/incubator-doris/issues/8428 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? core dump on runtime filter insert because of block address change ### What You Expected? fix it ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8429: [Bug][Vectorized] core dump on runtime filter insert because of block address change
BiteThet opened a new pull request #8429: URL: https://github.com/apache/incubator-doris/pull/8429 # Proposed changes Issue Number: close #8428 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #8387: [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex
caiconghui commented on a change in pull request #8387:
URL: https://github.com/apache/incubator-doris/pull/8387#discussion_r823433069
##
File path: be/src/util/thread_group.h
##
@@ -61,7 +61,7 @@ class ThreadGroup {
template
std::thread* create_thread(F threadfunc) {
-std::lock_guard guard(_mutex);
+WriteLock wrlock(_mutex);
Review comment:
already fix it
##
File path: be/src/olap/txn_manager.cpp
##
@@ -357,7 +357,7 @@ OLAPStatus TxnManager::delete_txn(OlapMeta* meta,
TPartitionId partition_id,
SchemaHash schema_hash, TabletUid
tablet_uid) {
pair key(partition_id, transaction_id);
TabletInfo tablet_info(tablet_id, schema_hash, tablet_uid);
-WriteLock txn_wrlock(&_get_txn_map_lock(transaction_id));
+WriteLock txn_wlock(_get_txn_map_lock(transaction_id));
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
