[GitHub] [doris] zhangstar333 commented on a diff in pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
zhangstar333 commented on code in PR #15339:
URL: https://github.com/apache/doris/pull/15339#discussion_r1109413017
##
be/src/vec/aggregate_functions/aggregate_function_collect.h:
##
@@ -49,18 +49,16 @@ struct AggregateFunctionCollectSetData {
data_set.insert(assert_cast(column).get_data()[row_num]);
}
-void merge(const SelfType& rhs) { data_set.merge(rhs.data_set); }
-
-void merge(const SelfType& rhs, bool has_limit) {
-if (!has_limit) {
-merge(rhs);
-return;
-}
-for (auto& rhs_elem : rhs.data_set) {
-if (size() >= max_size) {
-return;
+void merge(const SelfType& rhs) {
+if constexpr (HasLimit::value) {
+data_set.merge(rhs.data_set);
+} else {
Review Comment:
the template condition seems to mistake in writing, have limit but merge all
data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
github-actions[bot] commented on PR #16850: URL: https://github.com/apache/doris/pull/16850#issuecomment-1434271403 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dlimeng opened a new issue, #16873: [Bug] Doris1.2.2 jdbc Catalog connects to sqlserve unable to query
dlimeng opened a new issue, #16873: URL: https://github.com/apache/doris/issues/16873 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris : 1.2.2 sqlserver 2005 ### What's Wrong? This jdbc Catalog connects to sqlserve SQLServerException: The holdability value 1,000 is not valid. https://user-images.githubusercontent.com/16789827/219587423-39ba0abd-572c-4dec-8cab-2f5544af9022.png";> ### What You Expected? Fix the problem ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 commented on a diff in pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
zhangstar333 commented on code in PR #15339: URL: https://github.com/apache/doris/pull/15339#discussion_r1109418237 ## docs/en/docs/sql-manual/sql-functions/aggregate-functions/collect_list.md: ## @@ -30,9 +30,8 @@ under the License. `ARRAY collect_list(expr)` -Returns an array consisting of all values in expr within the group. -The order of elements in the array is non-deterministic. NULL values are excluded. - +Returns an array consisting of all values in expr within the group, and ,with the optional `max_size` parameter limits the size of the resulting array to `max_size` elements.The order of elements in the array is non-deterministic. NULL values are excluded. +It has an alias `group_array`. Review Comment: e... here I'm not sure is that, if have 100 rows in column, but when set max_size = 1, whether result will be same always -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16626: [fix](struct-type) forbid struct-type to be distributed key/aggregation key and add more tests
github-actions[bot] commented on PR #16626: URL: https://github.com/apache/doris/pull/16626#issuecomment-1434277696 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16626: [fix](struct-type) forbid struct-type to be distributed key/aggregation key and add more tests
github-actions[bot] commented on PR #16626: URL: https://github.com/apache/doris/pull/16626#issuecomment-1434277749 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
github-actions[bot] commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1434278746 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16784: [enhancement](profile) add profile for json& csv file converting to doris column
github-actions[bot] commented on PR #16784: URL: https://github.com/apache/doris/pull/16784#issuecomment-1434279768 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16564: [feature](merge-on-write) add DCHECK in compaction to detect data inconsistency
github-actions[bot] commented on PR #16564: URL: https://github.com/apache/doris/pull/16564#issuecomment-1434281724 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16790: [regression](struct-type) add more regression tests for struct and map type
github-actions[bot] commented on PR #16790: URL: https://github.com/apache/doris/pull/16790#issuecomment-1434284269 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16790: [regression](struct-type) add more regression tests for struct and map type
github-actions[bot] commented on PR #16790: URL: https://github.com/apache/doris/pull/16790#issuecomment-1434284327 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16870: [fix][nereids] Fix signatures of some window functions
github-actions[bot] commented on code in PR #16870:
URL: https://github.com/apache/doris/pull/16870#discussion_r1109476755
##
be/src/exec/base_scanner.cpp:
##
@@ -250,6 +265,28 @@
std::move(column_ptr), slot_desc->get_data_type_ptr(),
slot_desc->col_name()));
}
+// handle dynamic generated columns
+if (!_full_base_schema_view->empty()) {
+assert(_is_dynamic_schema);
+for (size_t x = dest_block->columns(); x < _src_block.columns(); ++x) {
+auto& column_type_name = _src_block.get_by_position(x);
+const TColumn& tcolumn =
+
_full_base_schema_view->column_name_to_column[column_type_name.name];
+auto original_type =
vectorized::DataTypeFactory::instance().create_data_type(tcolumn);
+// type conflict free path, always cast to original type
+if (!column_type_name.type->equals(*original_type)) {
+vectorized::ColumnPtr column_ptr;
+
RETURN_IF_ERROR(vectorized::schema_util::cast_column(column_type_name,
+
original_type, &column_ptr));
+column_type_name.column = column_ptr;
+column_type_name.type = original_type;
+}
+
dest_block->insert(vectorized::ColumnWithTypeAndName(std::move(column_type_name.column),
+
std::move(column_type_name.type),
Review Comment:
warning: passing result of std::move() as a const reference argument; no
move will actually happen [performance-move-const-arg]
```suggestion
column_type_name.type,
```
##
be/src/olap/rowset/segment_v2/inverted_index_cache.h:
##
@@ -163,5 +164,102 @@ class InvertedIndexCacheHandle {
DISALLOW_COPY_AND_ASSIGN(InvertedIndexCacheHandle);
};
+enum class InvertedIndexQueryType;
+
+class InvertedIndexQueryCacheHandle;
+
+class InvertedIndexQueryCache {
+public:
+// cache key
+struct CacheKey {
+io::Path index_path; // index file path
+std::string column_name; // column name
+InvertedIndexQueryType query_type; // query type
+std::wstring value;// query value
+
+// Encode to a flat binary which can be used as LRUCache's key
+std::string encode() const {
+std::string key_buf(index_path.string());
+key_buf.append("/");
+key_buf.append(column_name);
+key_buf.append("/");
+key_buf.append(1, static_cast(query_type));
+key_buf.append("/");
+key_buf.append(lucene::util::Misc::toString(value.c_str()));
+return key_buf;
+}
+};
+
+using CacheValue = roaring::Roaring;
+
+// Create global instance of this class
+static void create_global_cache(size_t capacity, int32_t
index_cache_percentage,
+uint32_t num_shards = 16) {
+DCHECK(_s_instance == nullptr);
+static InvertedIndexQueryCache instance(capacity,
index_cache_percentage, num_shards);
+_s_instance = &instance;
+}
+
+// Return global instance.
+// Client should call create_global_cache before.
+static InvertedIndexQueryCache* instance() { return _s_instance; }
+
+InvertedIndexQueryCache() = delete;
+
+InvertedIndexQueryCache(size_t capacity, int32_t index_cache_percentage,
uint32_t num_shards) {
+_cache = std::unique_ptr(
+new_lru_cache("InvertedIndexQueryCache", capacity,
LRUCacheType::SIZE, num_shards));
+}
+
+bool lookup(const CacheKey& key, InvertedIndexQueryCacheHandle* handle);
+
+void insert(const CacheKey& key, roaring::Roaring* bitmap,
+InvertedIndexQueryCacheHandle* handle);
+
+private:
+static InvertedIndexQueryCache* _s_instance;
+std::unique_ptr _cache {nullptr};
+};
+
+class InvertedIndexQueryCacheHandle {
+public:
+InvertedIndexQueryCacheHandle() {}
Review Comment:
warning: use '= default' to define a trivial default constructor
[modernize-use-equals-default]
```suggestion
InvertedIndexQueryCacheHandle() = default;
```
##
be/src/olap/task/engine_alter_tablet_task.h:
##
@@ -39,4 +39,18 @@
std::shared_ptr _mem_tracker;
}; // EngineTask
+class EngineAlterInvertedIndexTask : public EngineTask {
+public:
+virtual Status execute();
+
+public:
Review Comment:
warning: redundant access specifier has the same accessibility as the
previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
**be/src/olap/task/engine_alter_tablet_task.h:42:** previously declared here
```cpp
public:
^
```
##
be/src/exec/base_scanner.cpp:
##
@@ -250,6 +265,28 @@ Status
Ba
[GitHub] [doris] yixiutt opened a new pull request, #16874: [improvement](MOW) move update_delete_bitmap out of txn lock
yixiutt opened a new pull request, #16874: URL: https://github.com/apache/doris/pull/16874 # Proposed changes update_delete_bitmap will cost a lot of time and lock here is unnecessary, txn_lock is used to lock load info ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16874: [improvement](MOW) move update_delete_bitmap out of txn lock
github-actions[bot] commented on PR #16874: URL: https://github.com/apache/doris/pull/16874#issuecomment-1434335345 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16874: [improvement](MOW) move update_delete_bitmap out of txn lock
github-actions[bot] commented on PR #16874: URL: https://github.com/apache/doris/pull/16874#issuecomment-1434336576 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16861: [Fix](inverted index) fix compound directory unlock problem
github-actions[bot] commented on PR #16861: URL: https://github.com/apache/doris/pull/16861#issuecomment-1434351812 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
github-actions[bot] commented on PR #16803: URL: https://github.com/apache/doris/pull/16803#issuecomment-1434355126 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16803: [fix](cooldown) Use `pending_remote_rowsets` to avoid deleting rowset files being uploaded
hello-stephen commented on PR #16803: URL: https://github.com/apache/doris/pull/16803#issuecomment-1434360158 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.63 seconds stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230217091809_clickbench_pr_98833.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #16602: [feature-wip](iceberg) add dlf and glue catalog impl for iceberg catalog
wsjz commented on code in PR #16602:
URL: https://github.com/apache/doris/pull/16602#discussion_r1109504544
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/client/DLFCachedClientPool.java:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg.dlf.client;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.ClientPool;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.thrift.TException;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class DLFCachedClientPool implements ClientPool {
+
+private static Cache clientPoolCache;
+private final Configuration conf;
+private final String endpoint;
+private final int clientPoolSize;
+private final long evictionInterval;
+
+public DLFCachedClientPool(Configuration conf, Map
properties) {
+this.conf = conf;
+this.endpoint = conf.get("", "");
+this.clientPoolSize =
+PropertyUtil.propertyAsInt(
+properties,
+CatalogProperties.CLIENT_POOL_SIZE,
Review Comment:
Use default 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jacktengg opened a new pull request, #16875: [improvement](memory) fix possible double free in vcollect iterator
jacktengg opened a new pull request, #16875:
URL: https://github.com/apache/doris/pull/16875
# Proposed changes
Issue Number: close #xxx
## Problem summary
```
std::list cumu_children;
int i = 0;
for (const auto& child : _children) {
if (i != base_reader_idx) {
cumu_children.push_back(child);
}
++i;
}
bool is_merge = cumu_children.size() > 1;
auto cumu_iter =
std::make_unique(std::move(cumu_children), _reader,
is_merge,
_is_reverse, _skip_same);
RETURN_IF_NOT_EOF_AND_OK(cumu_iter->init());
```
This code in `VCollectIterator::build_heap` is possible to cause double free
if cumu_iter->init() fails and returns early, becuase some `LevelIterator*`
exists both in `VCollectIterator::_children` and `cumu_iter::_children`.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16123: [feature-wip](BE http)Support BE http service with brpc
github-actions[bot] commented on PR #16123: URL: https://github.com/apache/doris/pull/16123#issuecomment-1434365937 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16875: [improvement](memory) fix possible double free in vcollect iterator
github-actions[bot] commented on PR #16875: URL: https://github.com/apache/doris/pull/16875#issuecomment-1434366196 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #16602: [feature-wip](iceberg) add dlf and glue catalog impl for iceberg catalog
wsjz commented on code in PR #16602:
URL: https://github.com/apache/doris/pull/16602#discussion_r1109508493
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/dlf/client/DLFCachedClientPool.java:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.iceberg.dlf.client;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.ClientPool;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.thrift.TException;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class DLFCachedClientPool implements ClientPool {
Review Comment:
It just like CachedClientPool used for the pool of HiveCatalog or
JdbcCatalog.
But the CachedClientPool is package-private class, so create a new
ClientPool here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg opened a new issue, #16876: [Feature] Add submitter and comments to load job
yangzhg opened a new issue, #16876:
URL: https://github.com/apache/doris/issues/16876
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Description
Submitter information and comment information are two very important
functions in the data import task, and their importance to users is as follows:
1. Submitter information: Submitter information can help users understand
the source and responsible person of the data. In the data import task, the
cooperation between multiple users or teams may be involved, so knowing the
source of the data can help users to better follow up and communicate issues.
Additionally, if a problem occurs with the data, users can quickly contact the
submitter for a solution.
2. Note information: Note information can help users better understand the
meaning and purpose of the import task.
Submitter information and comment information are very important for users
of data import tasks, which can help users better understand the source,
meaning, and use of data, so as to better complete tasks.
### Use case
```
MySQL [demo]> show routine load \G
*** 1. row ***
Id: 15005
Name: label_13_04_19_35_471968_2016386449
User: root
CreateTime: 2023-02-17 17:11:03
PauseTime: NULL
EndTime: NULL
DbName: default_cluster:demo
TableName:
tc1b6_ne_load_job_control_test_resume_all_routine_job_tb
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {}
CustomProperties:
{"group.id":"label_13_04_19_35_471968_2016386449_ff40264a-0fbb-4eaf-8c45-919cde738531"}
Statistic:
{"receivedBytes":0,"runningTxns":[4006],"errorRows":0,"committedTaskNum":1,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":0,"errorRowsAfterResumed":0,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":10006}
Progress: {"0":"55334"}
Lag: {"0":585}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg:
Comment: 1234567
1 row in set (0.002 sec)
```
```
MySQL [demo]> show load \G
*** 1. row ***
JobId: 11001
Label: label_06_04_07_06_651161_706414572
User: root
State: FINISHED
Progress: ETL:100%; LOAD:100%
Type: BROKER
EtlInfo: unselected.rows=0; dpp.abnorm.ALL=0; dpp.norm.ALL=1009
TaskInfo: cluster:N/A; timeout(s):14400; max_filter_ratio:0.01
ErrorMsg: NULL
CreateTime: 2023-02-17 13:48:19
EtlStartTime: 2023-02-17 13:48:21
EtlFinishTime: 2023-02-17 13:48:21
LoadStartTime: 2023-02-17 13:48:21
LoadFinishTime: 2023-02-17 13:48:22
URL: NULL
JobDetails: {"Unfinished
backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All
backends":{},"FileNumber":0,"FileSize":0}
TransactionId: 2
ErrorTablets: {}
Comment: 1234567
```
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] FreeOnePlus opened a new pull request, #16877: [docs](docs)Fix FE config max_running_txn_num_per_db default value
FreeOnePlus opened a new pull request, #16877: URL: https://github.com/apache/doris/pull/16877 # Proposed changes max_running_txn_num_per_db old default value:100 fix change:1000 ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16875: [improvement](memory) fix possible double free in vcollect iterator
github-actions[bot] commented on PR #16875: URL: https://github.com/apache/doris/pull/16875#issuecomment-1434369655 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg opened a new pull request, #16878: [Feature](load) Add submitter and comments to load job
yangzhg opened a new pull request, #16878: URL: https://github.com/apache/doris/pull/16878 # Proposed changes Issue Number: close #16876 ## Problem summary Add submitter and comments to load job ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [x] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-thirdparty] morningman opened a new pull request, #38: [fix] skip unsupported auth method
morningman opened a new pull request, #38: URL: https://github.com/apache/doris-thirdparty/pull/38 sometimes the auths may contain unsupported auth method, such as OAUTHBEARER. should skip it and try next method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1434378185 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] deaftstill opened a new issue, #16879: [Bug] 在同时执行几个insert语句时，be节点挂掉
deaftstill opened a new issue, #16879: URL: https://github.com/apache/doris/issues/16879 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris-1.2.0-rc04 ### What's Wrong? 1.执行的5条insert语句数据量都在 2-3万条 2.执行了两次这几条insert语句，第一次不报错，隔了大概三分钟执行第二次开始报错 3.fe.out输出信息如下： `*** Query id: 2b587ec65ee34787-b09fbb9415942417 *** *** Aborted at 1676621475 (unix time) try "date -d @1676621475" if you are using GNU date *** *** Current BE git commitID: Unknown *** *** SIGSEGV address not mapped to object (@0x1e) received by PID 22744 (TID 0x7f5f55bf4700) from PID 1966080; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# os::Linux::chained_handler(int, siginfo*, void*) in /opt/service/jdk1.8.0_341/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /opt/service/jdk1.8.0_341/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo*, void*) in /opt/service/jdk1.8.0_341/jre/lib/amd64/server/libjvm.so 4# 0x7F6424177400 in /lib64/libc.so.6 5# jemalloc_usable_size at ../src/jemalloc.c:3740 6# doris_free at /root/doris/be/src/runtime/memory/jemalloc_hook.cpp:40 7# __GI__dl_deallocate_tls in /lib64/ld-linux-x86-64.so.2 8# __free_stacks in /lib64/libpthread.so.0 9# __deallocate_stack in /lib64/libpthread.so.0 10# start_thread in /lib64/libpthread.so.0 11# clone in /lib64/libc.so.6 ` 4.执行dmesg -T 没有发现被系统kill掉 ### What You Expected? 期望是能找出解决方法，以及产生原因 ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 commented on a diff in pull request #16832: [fix](merge-on-write) fix that be may coredump when sequence column is null
liaoxin01 commented on code in PR #16832:
URL: https://github.com/apache/doris/pull/16832#discussion_r1109524064
##
be/src/olap/rowset/segment_v2/segment_writer.cpp:
##
@@ -389,6 +388,22 @@ std::string SegmentWriter::_full_encode_keys(
return encoded_keys;
}
+void SegmentWriter::_encode_seq_column(const
vectorized::IOlapColumnDataAccessor* seq_column,
+ size_t pos, string* encoded_keys) {
+auto field = seq_column->get_data_at(pos);
+// To facilitate the use of the primary key index, encode the seq column
+// to the minimum value of the corresponding length when the seq column
+// is null
+if (UNLIKELY(!field)) {
+encoded_keys->push_back(KEY_NULL_FIRST_MARKER);
+size_t seq_col_length =
_tablet_schema->column(_tablet_schema->sequence_col_idx()).length();
+encoded_keys->append(seq_col_length, KEY_MINIMAL_MARKER);
Review Comment:
There is no problem. Because the signed number will reverse the sign bit
once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16832: [fix](merge-on-write) fix that be may coredump when sequence column is null
github-actions[bot] commented on PR #16832: URL: https://github.com/apache/doris/pull/16832#issuecomment-1434383760 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16869: [Improvement](jsonb) add suport for JSONB type for arrow
hello-stephen commented on PR #16869: URL: https://github.com/apache/doris/pull/16869#issuecomment-1434384614 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.4 seconds stream load tsv: 473 seconds loaded 74807831229 Bytes, about 150 MB/s stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230217094034_clickbench_pr_98852.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16818: [Improvement](parquet-reader) Optimize and refactor parquet reader to improve performance.
github-actions[bot] commented on code in PR #16818:
URL: https://github.com/apache/doris/pull/16818#discussion_r1109529424
##
be/src/vec/columns/column_string.h:
##
@@ -280,6 +280,98 @@ class ColumnString final : public COWHelper {
}
}
+#define MAX_STRINGS_OVERFLOW_SIZE 128
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t num)
+__attribute__((noinline));
+
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t
num) {
+size_t new_size = 0;
+for (size_t i = 0; i < num; i++) {
+new_size += strings[i].size;
+}
+
+const size_t old_size = chars.size();
+check_chars_length(old_size + new_size, offsets.size() + num);
+chars.resize(old_size + new_size + copy_length);
+
+Char* data = chars.data();
+size_t offset = old_size;
+for (size_t i = 0; i < num; i++) {
+uint32_t len = strings[i].size;
+if (len) {
+memcpy(data + offset, strings[i].data, copy_length);
+offset += len;
+}
+offsets.push_back(offset);
+}
+chars.resize(old_size + new_size);
+}
+
+void insert_many_strings_overflow(const StringRef* strings, size_t num,
+ size_t max_length) override {
+if (max_length <= 8) {
+insert_many_strings_fixed_length<8>(strings, num);
+} else if (max_length <= 16) {
+insert_many_strings_fixed_length<16>(strings, num);
+} else if (max_length <= 32) {
+insert_many_strings_fixed_length<32>(strings, num);
+} else if (max_length <= 64) {
+insert_many_strings_fixed_length<64>(strings, num);
+} else if (max_length <= 128) {
+insert_many_strings_fixed_length<128>(strings, num);
+} else {
+insert_many_strings(strings, num);
+}
+}
+
+#define MAX_STRINGS_OVERFLOW_SIZE 128
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t num)
Review Comment:
warning: class member cannot be redeclared [clang-diagnostic-error]
```cpp
void insert_many_strings_fixed_length(const StringRef* strings, size_t
num)
^
```
**be/src/vec/columns/column_string.h:284:** previous declaration is here
```cpp
void insert_many_strings_fixed_length(const StringRef* strings, size_t
num)
^
```
##
be/src/vec/columns/column_string.h:
##
@@ -280,6 +280,98 @@
}
}
+#define MAX_STRINGS_OVERFLOW_SIZE 128
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t num)
+__attribute__((noinline));
+
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t
num) {
+size_t new_size = 0;
+for (size_t i = 0; i < num; i++) {
+new_size += strings[i].size;
+}
+
+const size_t old_size = chars.size();
+check_chars_length(old_size + new_size, offsets.size() + num);
+chars.resize(old_size + new_size + copy_length);
+
+Char* data = chars.data();
+size_t offset = old_size;
+for (size_t i = 0; i < num; i++) {
+uint32_t len = strings[i].size;
+if (len) {
+memcpy(data + offset, strings[i].data, copy_length);
+offset += len;
+}
+offsets.push_back(offset);
+}
+chars.resize(old_size + new_size);
+}
+
+void insert_many_strings_overflow(const StringRef* strings, size_t num,
+ size_t max_length) override {
+if (max_length <= 8) {
+insert_many_strings_fixed_length<8>(strings, num);
+} else if (max_length <= 16) {
+insert_many_strings_fixed_length<16>(strings, num);
+} else if (max_length <= 32) {
+insert_many_strings_fixed_length<32>(strings, num);
+} else if (max_length <= 64) {
+insert_many_strings_fixed_length<64>(strings, num);
+} else if (max_length <= 128) {
+insert_many_strings_fixed_length<128>(strings, num);
+} else {
+insert_many_strings(strings, num);
+}
+}
+
+#define MAX_STRINGS_OVERFLOW_SIZE 128
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t num)
+__attribute__((noinline));
+
+template
+void insert_many_strings_fixed_length(const StringRef* strings, size_t
num) {
Review Comment:
warning: class member cannot be redeclared [clang-diagnostic-error]
```cpp
void insert_many_strings_fixed_length(const StringRef* strings, size_t
num) {
^
```
**be/src/vec/columns/column_string.h:288:** previous definition is here
```cpp
void insert_many_strings_fixed_length(con
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1434397999 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #16880: [RegressionTest](Pipeline) Fix pipeline failed in regression test
HappenLee opened a new pull request, #16880: URL: https://github.com/apache/doris/pull/16880 # Proposed changes regression-test/suites/inverted_index_p0/test_add_drop_index_with_data.groovy  ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16880: [RegressionTest](Pipeline) Fix pipeline failed in regression test
github-actions[bot] commented on PR #16880: URL: https://github.com/apache/doris/pull/16880#issuecomment-1434403701 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16818: [Improvement](parquet-reader) Optimize and refactor parquet reader to improve performance.
github-actions[bot] commented on code in PR #16818:
URL: https://github.com/apache/doris/pull/16818#discussion_r1109553766
##
be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp:
##
@@ -0,0 +1,531 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/columns/column_dictionary.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/data_types/data_type_nullable.h"
+
+namespace doris::vectorized {
+
+template
+class FixLengthDictDecoder final : public BaseDictDecoder {
+public:
+FixLengthDictDecoder(tparquet::Type::type physical_type)
+: BaseDictDecoder(), _physical_type(physical_type) {};
+~FixLengthDictDecoder() override = default;
+
+Status decode_values(MutableColumnPtr& doris_column, DataTypePtr&
data_type,
+ ColumnSelectVector& select_vector) override {
+size_t non_null_size = select_vector.num_values() -
select_vector.num_nulls();
+if (doris_column->is_column_dictionary() &&
+assert_cast(*doris_column).dict_size() == 0) {
+std::vector dict_items;
+dict_items.reserve(_dict_items.size());
+for (int i = 0; i < _dict_items.size(); ++i) {
+dict_items.emplace_back((char*)(&_dict_items[i]),
_type_length);
+}
+assert_cast(*doris_column)
+.insert_many_dict_data(&dict_items[0], dict_items.size());
+}
+_indexes.resize(non_null_size);
+_index_batch_decoder->GetBatch(&_indexes[0], non_null_size);
+
+if (doris_column->is_column_dictionary()) {
+return _decode_dict_values(doris_column, select_vector);
+}
+
+TypeIndex logical_type = remove_nullable(data_type)->get_type_id();
+switch (logical_type) {
+#define DISPATCH(NUMERIC_TYPE, CPP_NUMERIC_TYPE, PHYSICAL_TYPE)
\
+case NUMERIC_TYPE:
\
+if constexpr (std::is_same_v) {
\
+return _decode_numeric(doris_column,
select_vector); \
+}
+FOR_LOGICAL_NUMERIC_TYPES(DISPATCH)
Review Comment:
warning: use of undeclared identifier 'DISPATCH' [clang-diagnostic-error]
```cpp
FOR_LOGICAL_NUMERIC_TYPES(DISPATCH)
^
```
##
be/src/vec/exec/format/parquet/fix_length_dict_decoder.hpp:
##
@@ -0,0 +1,531 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/columns/column_dictionary.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/data_types/data_type_nullable.h"
+
+namespace doris::vectorized {
+
+template
+class FixLengthDictDecoder final : public BaseDictDecoder {
+public:
+FixLengthDictDecoder(tparquet::Type::type physical_type)
+: BaseDictDecoder(), _physical_type(physical_type) {};
+~FixLengthDictDecoder() override = default;
+
+Status decode_values(MutableColumnPtr& doris_column, DataTypePtr&
data_type,
+ ColumnSelectVector& select_vector) override {
+size_t non_null_size = select_vector.num_values() -
select_vector.num_nulls();
+if (doris_column->is_column_dictionary() &&
+assert_cast(*doris_column).dict_size() == 0) {
+std::vector dict_items;
+dict_items.reserve(_dict_items.size());
+f
[GitHub] [doris] SaintBacchus opened a new issue, #16881: [Feature] Decouple the schema order and the order key to make table created easier
SaintBacchus opened a new issue, #16881:
URL: https://github.com/apache/doris/issues/16881
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no
similar issues.
### Description
As an example of doris table
```sql
CREATE TABLE IF NOT EXISTS example_db.example_tbl
(
`timestamp` DATETIME NOT NULL COMMENT "日志时间",
`type` INT NOT NULL COMMENT "日志类型",
`error_code` INT COMMENT "错误码",
`error_msg` VARCHAR(1024) COMMENT "错误详细信息",
`op_id` BIGINT COMMENT "负责人id",
`op_time` DATETIME COMMENT "处理时间"
)
DUPLICATE KEY(`timestamp`, `type`, `error_code`)
DISTRIBUTED BY HASH(`type`) BUCKETS 1
PROPERTIES ("replication_allocation" = "tag.location.default: 1");
```
The order of table schema must be same with `DUPLICATE KEY(timestamp, type,
error_code)`
```
`timestamp` DATETIME NOT NULL COMMENT "日志时间",
`type` INT NOT NULL COMMENT "日志类型",
`error_code` INT COMMENT "错误码",
```
But the `DUPLICATE KEY` is also a prefix index for the table. So if we want
to change the first order of it, you must also change the order the table
create statement.
For example, if you want change DUPLICATE key into `DUPLICATE KEY(op_id)`,
you must place `op_id` field to be the first column of this table.
It's not convenience for user.
-
建表时字段的顺序需要跟排序键的对齐, 但调优的时候我们会调整排序键, 因此我们同时也要调整建表的字段顺序.
这就很不方便了, 尤其在对接其他SQL系统的时候, 例如hive等, hive没有字段顺序, 和Doris不对齐时, 在建表和导入过程中,
都要非常小心的去处理顺序. 尤其是超过200列的大宽表.
### Use case
- 对接其他系统的时候
- 迁移其他系统数据的时候
### Related issues
NO
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BePPPower opened a new pull request, #16882: [Enhencement](csv reader) Optimize csv reader
BePPPower opened a new pull request, #16882: URL: https://github.com/apache/doris/pull/16882 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] gnehil opened a new pull request, #69: [feature] support datev2/datatimev2/decimalv3/jsonb type
gnehil opened a new pull request, #69: URL: https://github.com/apache/doris-spark-connector/pull/69 # Proposed changes support datev2/datatimev2/decimalv3/jsonb type Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] JNSimba closed pull request #69: [feature] support datev2/datatimev2/decimalv3/jsonb type
JNSimba closed pull request #69: [feature] support datev2/datatimev2/decimalv3/jsonb type URL: https://github.com/apache/doris-spark-connector/pull/69 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing commented on a diff in pull request #16866: [typo](docs)Modify some document label errors
luzhijing commented on code in PR #16866: URL: https://github.com/apache/doris/pull/16866#discussion_r1109591140 ## docs/en/docs/advanced/best-practice/compaction.md: ## @@ -34,6 +34,9 @@ Appropriately adjusting the compaction strategy can greatly improve load and que ## Vertical compaction + Review Comment: since v1.2.2 ## docs/zh-CN/docs/advanced/best-practice/compaction.md: ## @@ -31,6 +31,10 @@ Doris 提供如下2种compaction方式进行调优： ## Vertical compaction + + Review Comment: since v1.2.2 ## docs/en/docs/advanced/best-practice/compaction.md: ## @@ -34,6 +34,9 @@ Appropriately adjusting the compaction strategy can greatly improve load and que ## Vertical compaction + + + Vertical compaction is a new compaction algorithm implemented in Doris 2.0, which is used to optimize compaction execution efficiency and resource overhead in large-scale and wide table scenarios. It can effectively reduce the memory overhead of compaction and improve the execution speed of compaction. The test results show that the memory consumption by vertical compaction is only 1/10 of the original compaction algorithm, and the compaction rate is increased by 15%. Review Comment: Doris 1.2.2 ## docs/zh-CN/docs/advanced/best-practice/compaction.md: ## @@ -31,6 +31,10 @@ Doris 提供如下2种compaction方式进行调优： ## Vertical compaction + + + + Vertical compaction 是 Doris 2.0 版本中实现的新的 Compaction 算法，用于解决大宽表场景下的 Compaction 执行效率和资源开销问题。可以有效降低Compaction的内存开销，并提升 Compaction 的执行速度。 Review Comment: 1.2.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #16883: [fix](Nereids): fix SemiJoinLogicalJoinTransposeProject.
jackwener opened a new pull request, #16883: URL: https://github.com/apache/doris/pull/16883 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [x] Does it affect the original behavior * [x] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16877: [docs](docs)Fix FE config max_running_txn_num_per_db default value
github-actions[bot] commented on PR #16877: URL: https://github.com/apache/doris/pull/16877#issuecomment-1434455131 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yagagagaga opened a new pull request, #16884: [typo](docs)supplement the document content
yagagagaga opened a new pull request, #16884: URL: https://github.com/apache/doris/pull/16884 # Proposed changes Issue Number: close #xxx ## Problem summary The original document is too simple, so I made some additions. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [x] Has document been added or modified * [ ] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16877: [docs](docs)Fix FE config max_running_txn_num_per_db default value
github-actions[bot] commented on PR #16877: URL: https://github.com/apache/doris/pull/16877#issuecomment-1434455178 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16868: [typo](doc)collect_set function documentation added 1.2 label
github-actions[bot] commented on PR #16868: URL: https://github.com/apache/doris/pull/16868#issuecomment-1434457771 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16868: [typo](doc)collect_set function documentation added 1.2 label
github-actions[bot] commented on PR #16868: URL: https://github.com/apache/doris/pull/16868#issuecomment-1434457847 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16819: [refactor](Nereids): remove local sort
github-actions[bot] commented on PR #16819: URL: https://github.com/apache/doris/pull/16819#issuecomment-1434459788 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16819: [refactor](Nereids): remove local sort
github-actions[bot] commented on PR #16819: URL: https://github.com/apache/doris/pull/16819#issuecomment-1434459858 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing commented on a diff in pull request #16746: [docs](doc) Add autobucket doc
luzhijing commented on code in PR #16746:
URL: https://github.com/apache/doris/pull/16746#discussion_r1109615042
##
docs/zh-CN/docs/advanced/autobucket.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "自动分桶",
+"language": "zh-CN"
+}
+---
+
+
+
+# 背景
+
+用户经常设置不合适的bucket，导致各种问题，这里提供一种方式，来自动设置分桶数。暂时而言只对olap表生效
Review Comment:
- 暂时而言只对olap表生效
internal table?
##
docs/zh-CN/docs/advanced/autobucket.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "自动分桶",
+"language": "zh-CN"
+}
+---
+
+
+
Review Comment:
add the version label
##
docs/en/docs/advanced/autobucket.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "AutoBucket",
+"language": "en"
+}
+---
+
+
+
Review Comment:
add the version label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16792: Revert "[test](pipeline) Run nereids cases in p1/p2 (#16130)"
dataroaring merged PR #16792: URL: https://github.com/apache/doris/pull/16792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16819: [refactor](Nereids): remove local sort
morrySnow merged PR #16819: URL: https://github.com/apache/doris/pull/16819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](Nereids) remove local sort (#16819)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fd5d7d6097 [refactor](Nereids) remove local sort (#16819)
fd5d7d6097 is described below
commit fd5d7d6097372cee23cba67ef460966410819489
Author: 谢健
AuthorDate: Fri Feb 17 18:52:41 2023 +0800
[refactor](Nereids) remove local sort (#16819)
After adding phase in sort, the locatSort is no longer needed
change the order of sortPhase in constructor
---
.../apache/doris/nereids/cost/CostCalculator.java | 14 ---
.../glue/translator/PhysicalPlanTranslator.java| 11 ---
.../processor/post/RuntimeFilterPruner.java| 10 --
.../properties/ChildOutputPropertyDeriver.java | 10 --
.../apache/doris/nereids/properties/OrderSpec.java | 8 +-
.../nereids/properties/RequestPropertyDeriver.java | 8 --
.../LogicalSortToPhysicalQuickSort.java| 10 +-
.../implementation/LogicalTopNToPhysicalTopN.java | 8 +-
.../doris/nereids/stats/StatsCalculator.java | 6 --
.../trees/plans/physical/AbstractPhysicalSort.java | 9 +-
.../plans/physical/PhysicalLocalQuickSort.java | 104 -
.../trees/plans/physical/PhysicalQuickSort.java| 29 +++---
.../nereids/trees/plans/physical/PhysicalTopN.java | 29 +++---
.../nereids/trees/plans/visitor/PlanVisitor.java | 5 -
.../properties/ChildOutputPropertyDeriverTest.java | 7 +-
.../doris/nereids/trees/plans/PlanEqualsTest.java | 12 +--
16 files changed, 54 insertions(+), 226 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java
index cabdce770b..b0ac15b93f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java
@@ -31,7 +31,6 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalFileScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalGenerate;
import org.apache.doris.nereids.trees.plans.physical.PhysicalHashAggregate;
import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
-import org.apache.doris.nereids.trees.plans.physical.PhysicalLocalQuickSort;
import org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
@@ -166,19 +165,6 @@ public class CostCalculator {
childStatistics.getRowCount());
}
-@Override
-public CostEstimate visitPhysicalLocalQuickSort(
-PhysicalLocalQuickSort sort, PlanContext
context) {
-// TODO: consider two-phase sort and enforcer.
-StatsDeriveResult statistics = context.getStatisticsWithCheck();
-StatsDeriveResult childStatistics = context.getChildStatistics(0);
-
-return CostEstimate.of(
-childStatistics.getRowCount(),
-statistics.getRowCount(),
-0);
-}
-
@Override
public CostEstimate visitPhysicalDistribute(
PhysicalDistribute distribute, PlanContext
context) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 0033da09d8..a7af0a72a6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -809,17 +809,6 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor sort,
-PlanTranslatorContext context) {
-PlanFragment childFragment = sort.child(0).accept(this, context);
-PlanNode childNode = childFragment.getPlanRoot();
-SortNode sortNode = translateSortNode(sort, childNode, context);
-childFragment.addPlanRoot(sortNode);
-return childFragment;
-}
-
/**
* the contract of hash join node with BE
* 1. hash join contains 3 types of predicates:
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java
index afca448aff..48abc974a1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterPruner.java
@@ -30,7 +30,6 @@ import
org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
import org.apache.doris.nereids.trees.p
[GitHub] [doris] luzhijing commented on a diff in pull request #16884: [typo](docs)supplement the document content
luzhijing commented on code in PR #16884: URL: https://github.com/apache/doris/pull/16884#discussion_r1109620995 ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 Review Comment: add space after and before English letters in CN docs ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 Review Comment: add space before and after English letters in CN docs ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 -GROUPING用在含有CUBE 或 ROLLUP 语句的SQL语句中，当结果集中的数据行是由CUBE 或 ROLLUP 运算产生的则该函数返回1，否则返回0。 +### 语法 -### example +```sql +GROUPING( ) ``` -MySQL > SELECT COL1,GROUPING(COL2) AS 'Grouping' FROM tbl GROUP BY ROLLUP (COL1, COL2); -+--+--+ -| COL1 | Grouping | -+--+--+ -| NULL |1 | -| 2.20 |1 | -| 2.20 |0 | -| 1.10 |1 | -| 1.10 |0 | -+--+--+ + +### 参数 + +`` +是在 `GROUP BY` 子句中包含的列或表达式。 + +### 返回类型 + +BIGINT + +### 备注 + +`ROLLUP`、`CUBE` 或 `GROUPING SETS` 操作返回的汇总结果，会用 NULL 充当被分组的字段的值。因此，`GROUPING` 通常用于区分 `ROLLUP`、`CUBE` 或 `GROUPING SETS` 返回的空值与表中的空值。 + +### 示例 + +下面的例子使用 `camp` 列进行分组操作，并对 `occupation` 的数量进行汇总，`GROUPING` 函数作用于 `camp` 列。 + +```sql +CREATE TABLE `roles` ( + role_id INT, + occupationVARCHAR(32), + camp VARCHAR(32), + register_time DATE +) +UNIQUE KEY(role_id) +DISTRIBUTED BY HASH(role_id) BUCKETS 1 +PROPERTIES ( + "replication_allocation" = "tag.location.default: 1" +); + +INSERT INTO `roles` VALUES +(0, 'who am I', NULL, NULL), +(1, 'mage', 'alliance', '2018-12-03 16:11:28'), +(2, 'paladin', 'alliance', '2018-11-30 16:11:28'), +(3, 'rogue', 'horde', '2018-12-01 16:11:28'), +(4, 'priest', 'alliance', '2018-12-02 16:11:28'), +(5, 'shaman', 'horde', NULL), +(6, 'warrior', 'alliance', NULL), +(7, 'warlock', 'horde', '2018-12-04 16:11:28'), +(8, 'hunter', 'horde', NULL); + +SELECT + camp, + COUNT(occupation) AS 'occ_cnt', + GROUPING(camp)AS 'grouping' +FROM + `roles` +GROUP BY + ROLLUP(camp); -- CUBE(camp) 和 GROUPING SETS((camp)) 同样也有效; +``` + +结果集在 `camp` 列下有两个 NULL 值，第一个 NULL 值表示 `ROLLUP` 操作的列的汇总结果，这一行的 `occ_cnt` 列表示所有 `camp` 的 `occupation` 的计数结果，在 `grouping` 函数中返回 1。第二个 NULL 表示 `camp` 列中本来就存在的 null 值。 Review Comment: null -> NULL, keep the English case consistent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on a diff in pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
TangSiyang2001 commented on code in PR #15339:
URL: https://github.com/apache/doris/pull/15339#discussion_r1109628442
##
be/src/vec/aggregate_functions/aggregate_function_collect.h:
##
@@ -49,18 +49,16 @@ struct AggregateFunctionCollectSetData {
data_set.insert(assert_cast(column).get_data()[row_num]);
}
-void merge(const SelfType& rhs) { data_set.merge(rhs.data_set); }
-
-void merge(const SelfType& rhs, bool has_limit) {
-if (!has_limit) {
-merge(rhs);
-return;
-}
-for (auto& rhs_elem : rhs.data_set) {
-if (size() >= max_size) {
-return;
+void merge(const SelfType& rhs) {
+if constexpr (HasLimit::value) {
+data_set.merge(rhs.data_set);
+} else {
Review Comment:
OK, I'll check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #16783: [enhancement](nereids)add a session variable to control join reorder algorithm
morrySnow commented on code in PR #16783:
URL: https://github.com/apache/doris/pull/16783#discussion_r1109627278
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java:
##
@@ -103,9 +103,13 @@ public NereidsRewriteJobExecutor(CascadesContext
cascadesContext) {
.add(topDownBatch(RuleSet.PUSH_DOWN_FILTERS, false))
.add(visitorJob(RuleType.INFER_PREDICATES, new
InferPredicates()))
.add(topDownBatch(ImmutableList.of(new
ExtractFilterFromCrossJoin(
-.add(topDownBatch(ImmutableList.of(new MergeFilters(
-.add(topDownBatch(ImmutableList.of(new ReorderJoin(
-.add(topDownBatch(ImmutableList.of(new ColumnPruning(
+.add(topDownBatch(ImmutableList.of(new MergeFilters(;
+
+if
(!cascadesContext.getConnectContext().getSessionVariable().isDisableJoinReorder())
{
+jobBuilder.add(topDownBatch(ImmutableList.of(new ReorderJoin(;
Review Comment:
better way to do that is move the if statement into ReorderJoin rule
##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -170,8 +170,10 @@ public Plan plan(LogicalPlan plan, PhysicalProperties
requireProperties, Explain
}
deriveStats();
-if
(statementContext.getConnectContext().getSessionVariable().isEnableDPHypOptimizer())
{
-// TODO: use DPHyp according the number of join table
+if
(!statementContext.getConnectContext().getSessionVariable().isDisableJoinReorder()
+&&
statementContext.getConnectContext().getSessionVariable().isEnableDPHypOptimizer()
+&& statementContext.getMaxNAryInnerJoin() >
statementContext.getConnectContext()
+
.getSessionVariable().getMaxTableCountUseCascadesJoinReorder()) {
dpHypOptimize();
} else {
optimize();
Review Comment:
i think we need always run optimize, and use different rule set according to
join size.
merge dpHypOptimize and optimize into one function, and do if in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] airborne12 opened a new pull request, #16885: [Fix](inverted index) fix inverted index bkd reader memory leak problem
airborne12 opened a new pull request, #16885: URL: https://github.com/apache/doris/pull/16885 # Proposed changes Issue Number: close #xxx ## Problem summary Original implementation of get_bkd_reader's raw pointer usage may cause memory leak problem, use shared_ptr to avoid that. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #16886: [Enchancement](function) remove unused params on aggregate function
BiteThet opened a new pull request, #16886: URL: https://github.com/apache/doris/pull/16886 # Proposed changes remove all unused params on aggregate function ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] luzhijing merged pull request #16868: [typo](doc)collect_set function documentation added 1.2 label
luzhijing merged PR #16868: URL: https://github.com/apache/doris/pull/16868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (fd5d7d6097 -> ea0e090a77)
This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from fd5d7d6097 [refactor](Nereids) remove local sort (#16819) add ea0e090a77 collect_set function documentation added 1.2 label (#16868) No new revisions were added by this update. Summary of changes: .../sql-manual/sql-functions/aggregate-functions/collect_set.md| 7 +++ .../sql-manual/sql-functions/aggregate-functions/collect_set.md| 7 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16885: [Fix](inverted index) fix inverted index bkd reader memory leak problem
github-actions[bot] commented on PR #16885: URL: https://github.com/apache/doris/pull/16885#issuecomment-1434485803 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yagagagaga commented on a diff in pull request #16884: [typo](docs)supplement the document content
yagagagaga commented on code in PR #16884: URL: https://github.com/apache/doris/pull/16884#discussion_r1109642877 ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 -GROUPING用在含有CUBE 或 ROLLUP 语句的SQL语句中，当结果集中的数据行是由CUBE 或 ROLLUP 运算产生的则该函数返回1，否则返回0。 +### 语法 -### example +```sql +GROUPING( ) ``` -MySQL > SELECT COL1,GROUPING(COL2) AS 'Grouping' FROM tbl GROUP BY ROLLUP (COL1, COL2); -+--+--+ -| COL1 | Grouping | -+--+--+ -| NULL |1 | -| 2.20 |1 | -| 2.20 |0 | -| 1.10 |1 | -| 1.10 |0 | -+--+--+ + +### 参数 + +`` +是在 `GROUP BY` 子句中包含的列或表达式。 + +### 返回类型 + +BIGINT + +### 备注 + +`ROLLUP`、`CUBE` 或 `GROUPING SETS` 操作返回的汇总结果，会用 NULL 充当被分组的字段的值。因此，`GROUPING` 通常用于区分 `ROLLUP`、`CUBE` 或 `GROUPING SETS` 返回的空值与表中的空值。 + +### 示例 + +下面的例子使用 `camp` 列进行分组操作，并对 `occupation` 的数量进行汇总，`GROUPING` 函数作用于 `camp` 列。 + +```sql +CREATE TABLE `roles` ( + role_id INT, + occupationVARCHAR(32), + camp VARCHAR(32), + register_time DATE +) +UNIQUE KEY(role_id) +DISTRIBUTED BY HASH(role_id) BUCKETS 1 +PROPERTIES ( + "replication_allocation" = "tag.location.default: 1" +); + +INSERT INTO `roles` VALUES +(0, 'who am I', NULL, NULL), +(1, 'mage', 'alliance', '2018-12-03 16:11:28'), +(2, 'paladin', 'alliance', '2018-11-30 16:11:28'), +(3, 'rogue', 'horde', '2018-12-01 16:11:28'), +(4, 'priest', 'alliance', '2018-12-02 16:11:28'), +(5, 'shaman', 'horde', NULL), +(6, 'warrior', 'alliance', NULL), +(7, 'warlock', 'horde', '2018-12-04 16:11:28'), +(8, 'hunter', 'horde', NULL); + +SELECT + camp, + COUNT(occupation) AS 'occ_cnt', + GROUPING(camp)AS 'grouping' +FROM + `roles` +GROUP BY + ROLLUP(camp); -- CUBE(camp) 和 GROUPING SETS((camp)) 同样也有效; +``` + +结果集在 `camp` 列下有两个 NULL 值，第一个 NULL 值表示 `ROLLUP` 操作的列的汇总结果，这一行的 `occ_cnt` 列表示所有 `camp` 的 `occupation` 的计数结果，在 `grouping` 函数中返回 1。第二个 NULL 表示 `camp` 列中本来就存在的 null 值。 Review Comment: ok -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yagagagaga commented on a diff in pull request #16884: [typo](docs)supplement the document content
yagagagaga commented on code in PR #16884: URL: https://github.com/apache/doris/pull/16884#discussion_r1109644590 ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 Review Comment: I have made corrections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yagagagaga commented on a diff in pull request #16884: [typo](docs)supplement the document content
yagagagaga commented on code in PR #16884: URL: https://github.com/apache/doris/pull/16884#discussion_r1109644910 ## docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md: ## @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +用在含有 CUBE、ROLLUP 或 GROUPING SETS 的SQL语句中，用于表示进行 CUBE、ROLLUP 或 GROUPING SETS 操作的列是否汇总。当结果集中的数据行是 CUBE、ROLLUP 或 GROUPING SETS 操作产生的汇总结果时，该函数返回 1，否则返回 0。GROUPING 函数可以在 `SELECT`、`HAVING` 和 `ORDER BY` 子句当中使用。 Review Comment: I have made corrections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16885: [Fix](inverted index) fix inverted index bkd reader memory leak problem
github-actions[bot] commented on PR #16885: URL: https://github.com/apache/doris/pull/16885#issuecomment-1434489527 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16861: [Fix](inverted index) fix compound directory unlock problem
hello-stephen commented on PR #16861: URL: https://github.com/apache/doris/pull/16861#issuecomment-1434497825 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.24 seconds stream load tsv: 472 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 35 seconds loaded 2358488459 Bytes, about 64 MB/s stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230217111748_clickbench_pr_98931.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16777: [feature] Make VUnionNode support conjecture logic
github-actions[bot] commented on PR #16777: URL: https://github.com/apache/doris/pull/16777#issuecomment-1434498207 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16875: [improvement](memory) fix possible double free in vcollect iterator
hello-stephen commented on PR #16875: URL: https://github.com/apache/doris/pull/16875#issuecomment-1434509900 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 34.3 seconds stream load tsv: 475 seconds loaded 74807831229 Bytes, about 150 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230217112919_clickbench_pr_98949.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #16842: [chore](build) add a build type named DEBUG_O3
dataroaring commented on PR #16842: URL: https://github.com/apache/doris/pull/16842#issuecomment-1434518982 > ```shell > ```shell > BUILD_TYPE=DEBUG EXTRA_CXX_FLAGS=-O3 ./run- > ``` > > > > > > > > > > > > ``` it works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring closed pull request #16842: [chore](build) add a build type named DEBUG_O3
dataroaring closed pull request #16842: [chore](build) add a build type named DEBUG_O3 URL: https://github.com/apache/doris/pull/16842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16866: [typo](docs)Modify some document label errors
github-actions[bot] commented on PR #16866: URL: https://github.com/apache/doris/pull/16866#issuecomment-1434526288 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16866: [typo](docs)Modify some document label errors
github-actions[bot] commented on PR #16866: URL: https://github.com/apache/doris/pull/16866#issuecomment-1434526343 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1434527874 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16884: [typo](docs)supplement the document content
github-actions[bot] commented on PR #16884: URL: https://github.com/apache/doris/pull/16884#issuecomment-1434527999 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16884: [typo](docs)supplement the document content
github-actions[bot] commented on PR #16884: URL: https://github.com/apache/doris/pull/16884#issuecomment-1434528048 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1434531040 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhilaohu32 commented on issue #3906: 官方有微信群或者QQ群吗，建一个，方便大家交流
zhilaohu32 commented on issue #3906: URL: https://github.com/apache/doris/issues/3906#issuecomment-1434536044 > 有微信群，可以加我微信 morningman-cmy 我希望加入这个群，可以是我申请添加您的微信没有通过 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16878: [Feature](load) Add submitter and comments to load job
github-actions[bot] commented on PR #16878: URL: https://github.com/apache/doris/pull/16878#issuecomment-1434547010 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16689: [Enchancement](merge-on-write) check the correctness of rowid conversion after compaction
github-actions[bot] commented on PR #16689: URL: https://github.com/apache/doris/pull/16689#issuecomment-143480 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16689: [Enchancement](merge-on-write) check the correctness of rowid conversion after compaction
github-actions[bot] commented on PR #16689: URL: https://github.com/apache/doris/pull/16689#issuecomment-1434557941 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16886: [Enchancement](function) remove unused params on aggregate function
github-actions[bot] commented on PR #16886: URL: https://github.com/apache/doris/pull/16886#issuecomment-1434562839 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16859: [fix](merge-on-write) The delete bitmap of the currently imported rowset is not persistent
github-actions[bot] commented on PR #16859: URL: https://github.com/apache/doris/pull/16859#issuecomment-1434563787 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16810: [enhance](cooldown)skip follow cooldown once failed to do follow cooldown
github-actions[bot] commented on PR #16810: URL: https://github.com/apache/doris/pull/16810#issuecomment-1434567999 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong closed pull request #16639: [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread
chenlinzhong closed pull request #16639: [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread URL: https://github.com/apache/doris/pull/16639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong closed issue #16634: [Enhancement] replace brpc bthread with pthead
chenlinzhong closed issue #16634: [Enhancement] replace brpc bthread with pthead URL: https://github.com/apache/doris/issues/16634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16880: [RegressionTest](Pipeline) Fix pipeline failed in regression test
yiguolei merged PR #16880: URL: https://github.com/apache/doris/pull/16880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [RegressionTest](Pipeline) Fix pipeline failed in regression test (#16880)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fda4afecf5 [RegressionTest](Pipeline) Fix pipeline failed in
regression test (#16880)
fda4afecf5 is described below
commit fda4afecf503a674b154d34829473c225e055ef2
Author: HappenLee
AuthorDate: Fri Feb 17 20:49:17 2023 +0800
[RegressionTest](Pipeline) Fix pipeline failed in regression test (#16880)
regression-test/suites/inverted_index_p0/test_add_drop_index_with_data.groovy
---
be/src/pipeline/pipeline_fragment_context.cpp | 1 -
1 file changed, 1 deletion(-)
diff --git a/be/src/pipeline/pipeline_fragment_context.cpp
b/be/src/pipeline/pipeline_fragment_context.cpp
index 524e1be67e..6b41647d58 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -120,7 +120,6 @@ void PipelineFragmentContext::cancel(const
PPlanFragmentCancelReason& reason,
}
if (reason != PPlanFragmentCancelReason::LIMIT_REACH) {
_exec_status = Status::Cancelled(msg);
-_set_is_report_on_cancel(false);
}
_runtime_state->set_is_cancelled(true);
if (_pipe != nullptr) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16776: [Feature](map-type) Support stream load and fix some bugs for map type
github-actions[bot] commented on code in PR #16776:
URL: https://github.com/apache/doris/pull/16776#discussion_r1109765169
##
be/test/olap/rowset/rowset_tree_test.cpp:
##
@@ -255,11 +255,13 @@
int r = strcmp(s1.c_str(), s2.c_str());
switch (op) {
case BOUND_LESS_THAN:
-if (r == 0) continue; // pass through.
+if (r == 0) continue;
+[[fallthrough]];
case BOUND_LESS_EQUAL:
return std::pair(std::min(s1, s2),
std::max(s1, s2));
case BOUND_GREATER_THAN:
-if (r == 0) continue; // pass through.
+if (r == 0) continue;
Review Comment:
warning: statement should be inside braces
[readability-braces-around-statements]
```suggestion
if (r == 0) { continue;
}
```
##
be/test/olap/rowset/rowset_tree_test.cpp:
##
@@ -255,11 +255,13 @@ TEST_F(TestRowsetTree, TestTreeRandomized) {
int r = strcmp(s1.c_str(), s2.c_str());
switch (op) {
case BOUND_LESS_THAN:
-if (r == 0) continue; // pass through.
+if (r == 0) continue;
Review Comment:
warning: statement should be inside braces
[readability-braces-around-statements]
```suggestion
if (r == 0) { continue;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16639: [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread
github-actions[bot] commented on PR #16639: URL: https://github.com/apache/doris/pull/16639#issuecomment-1434604807 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16861: [Fix](inverted index) fix compound directory unlock problem
github-actions[bot] commented on PR #16861: URL: https://github.com/apache/doris/pull/16861#issuecomment-1434606513 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
github-actions[bot] commented on PR #15339: URL: https://github.com/apache/doris/pull/15339#issuecomment-1434607988 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15339: [enhancement](aggregate-function) enhance aggregate funtion collect and add group_array aliases
github-actions[bot] commented on PR #15339: URL: https://github.com/apache/doris/pull/15339#issuecomment-1434608312 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs)Modify some document label errors (#16866)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new adc42600b4 [typo](docs)Modify some document label errors (#16866)
adc42600b4 is described below
commit adc42600b4c7051494469ea5ce30eff2664764d1
Author: yongkang.zhong
AuthorDate: Fri Feb 17 20:55:17 2023 +0800
[typo](docs)Modify some document label errors (#16866)
* [typo](docs)Modify some document label errors
* fix
---
docs/en/docs/advanced/best-practice/compaction.md | 5 -
docs/en/docs/advanced/hight-concurrent-point-query.md | 6 +++---
docs/en/docs/data-table/dynamic-schema-table.md | 6 --
docs/en/docs/data-table/index/inverted-index.md | 2 +-
docs/en/docs/data-table/index/ngram-bloomfilter-index.md| 3 +++
docs/sidebars.json | 1 +
docs/zh-CN/docs/advanced/best-practice/compaction.md| 6 +-
docs/zh-CN/docs/advanced/hight-concurrent-point-query.md| 8
docs/zh-CN/docs/data-table/dynamic-schema-table.md | 3 +++
docs/zh-CN/docs/data-table/index/inverted-index.md | 2 +-
docs/zh-CN/docs/data-table/index/ngram-bloomfilter-index.md | 3 +++
11 files changed, 32 insertions(+), 13 deletions(-)
diff --git a/docs/en/docs/advanced/best-practice/compaction.md
b/docs/en/docs/advanced/best-practice/compaction.md
index 95f4bf26cc..5963a1bdfb 100644
--- a/docs/en/docs/advanced/best-practice/compaction.md
+++ b/docs/en/docs/advanced/best-practice/compaction.md
@@ -34,7 +34,10 @@ Appropriately adjusting the compaction strategy can greatly
improve load and que
## Vertical compaction
-Vertical compaction is a new compaction algorithm implemented in Doris 2.0,
which is used to optimize compaction execution efficiency and resource overhead
in large-scale and wide table scenarios. It can effectively reduce the memory
overhead of compaction and improve the execution speed of compaction. The test
results show that the memory consumption by vertical compaction is only 1/10 of
the original compaction algorithm, and the compaction rate is increased by 15%.
+
+
+
+Vertical compaction is a new compaction algorithm implemented in Doris 1.2.2,
which is used to optimize compaction execution efficiency and resource overhead
in large-scale and wide table scenarios. It can effectively reduce the memory
overhead of compaction and improve the execution speed of compaction. The test
results show that the memory consumption by vertical compaction is only 1/10 of
the original compaction algorithm, and the compaction rate is increased by 15%.
In vertical compaction, merging by row is changed to merging by column group.
The granularity of each merge is changed to column group, which reduces the
amount of data involved in single compaction and reduces the memory usage
during compaction.
diff --git a/docs/en/docs/advanced/hight-concurrent-point-query.md
b/docs/en/docs/advanced/hight-concurrent-point-query.md
index 62c35cf820..91e5d9b53e 100644
--- a/docs/en/docs/advanced/hight-concurrent-point-query.md
+++ b/docs/en/docs/advanced/hight-concurrent-point-query.md
@@ -51,9 +51,9 @@ CREATE TABLE `tbl_point_query` (
`v6` float NULL,
`v7` datev2 NULL
) ENGINE=OLAP
-UNIQUE KEY(key)
+UNIQUE KEY(`key`)
COMMENT 'OLAP'
-DISTRIBUTED BY HASH(key) BUCKETS 1
+DISTRIBUTED BY HASH(`key)` BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"enable_unique_key_merge_on_write" = "true",
@@ -71,7 +71,7 @@ In order to reduce CPU cost for parsing query SQL and SQL
expressions, we provid
1. Setup JDBC url and enable server side prepared statement
```
-url = jdbc:mysql://127.0.0.1:9137/ycsb?useServerPrepStmts=true
+url = jdbc:mysql://127.0.0.1:9030/ycsb?useServerPrepStmts=true
``
2. Using `PreparedStatement`
diff --git a/docs/en/docs/data-table/dynamic-schema-table.md
b/docs/en/docs/data-table/dynamic-schema-table.md
index 0b21f6f572..374176dcbd 100644
--- a/docs/en/docs/data-table/dynamic-schema-table.md
+++ b/docs/en/docs/data-table/dynamic-schema-table.md
@@ -1,7 +1,7 @@
---
{
"title": "dynamie schema table",
-"language": "zh-CN"
+"language": "en"
}
---
@@ -25,9 +25,11 @@ under the License.
-->
# Dynamic Table
-A dynamic schema table is a special kind of table which schema expands
automatically with the import procedure. Currently, this feature is mainly used
for importing semi-structured data such as JSON. Because JSON is
self-describing, we can extract the schema information from the original
document and infer the final type information. This special table can reduce
manual schema change operations and easily import semi-structured data and
automatically expand its schema.
+
+
+A dynamic schema table is a special kind of table which schema expands
au
[GitHub] [doris] yiguolei merged pull request #16866: [typo](docs)Modify some document label errors
yiguolei merged PR #16866: URL: https://github.com/apache/doris/pull/16866 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16877: [docs](docs)Fix FE config max_running_txn_num_per_db default value
yiguolei merged PR #16877: URL: https://github.com/apache/doris/pull/16877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16884: [typo](docs)supplement the document content
yiguolei merged PR #16884: URL: https://github.com/apache/doris/pull/16884 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (90ae8dcf01 -> d5c393f413)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 90ae8dcf01 [typo](docs)supplement the document content (#16884) add d5c393f413 [docs](docs)Fix FE config max_running_txn_num_per_db default value (#16877) No new revisions were added by this update. Summary of changes: docs/en/docs/admin-manual/config/fe-config.md| 2 +- docs/zh-CN/docs/admin-manual/config/fe-config.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs)supplement the document content (#16884)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 90ae8dcf01 [typo](docs)supplement the document content (#16884) 90ae8dcf01 is described below commit 90ae8dcf01f31e5f5ab0a41aa5098aa5235cf065 Author: yagagagaga AuthorDate: Fri Feb 17 20:55:34 2023 +0800 [typo](docs)supplement the document content (#16884) * [typo](docs)supplement the document content * Update grouping.md Add space before and after English letters in CN docs and keep the English case consistent. * Update grouping.md Change the Chinese title to English --- .../sql-functions/aggregate-functions/grouping.md | 90 + .../sql-functions/aggregate-functions/grouping.md | 91 ++ 2 files changed, 147 insertions(+), 34 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-functions/aggregate-functions/grouping.md b/docs/en/docs/sql-manual/sql-functions/aggregate-functions/grouping.md index e8e5d53659..e642ebd14e 100755 --- a/docs/en/docs/sql-manual/sql-functions/aggregate-functions/grouping.md +++ b/docs/en/docs/sql-manual/sql-functions/aggregate-functions/grouping.md @@ -25,26 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +Indicates whether a specified column expression in a `GROUP BY` list is aggregated or not. `GROUPING` returns 1 for aggregated or 0 for not aggregated in the result set. `GROUPING` can be used only in the `SELECT list`, `HAVING`, and `ORDER BY` clauses when `GROUP BY` is specified. -GROUPING is used in SQL statements containing CUBE or ROLLUP statements. When the data rows in the result set are generated by CUBE or ROLLUP operations, this function returns 1. Otherwise, it returns 0. +### Syntax -### example +```sql +GROUPING( ) ``` -MySQL > SELECT COL1,GROUPING(COL2) AS 'Grouping' FROM tbl GROUP BY ROLLUP (COL1, COL2); -+--+--+ -| COL1 | Grouping | -+--+--+ -| NULL |1 | -| 2.20 |1 | -| 2.20 |0 | -| 1.10 |1 | -| 1.10 |0 | -+--+--+ + +### Arguments + +`` +Is a column or an expression that contains a column in a `GROUP BY` clause. + +### Return Types + +BIGINT + +### Remarks + +`GROUPING` is used to distinguish the null values that are returned by `ROLLUP`, `CUBE` or `GROUPING SETS` from standard null values. The `NULL` returned as the result of a `ROLLUP`, `CUBE` or `GROUPING SETS` operation is a special use of `NULL`. This acts as a column placeholder in the result set and means all. + +### Example + +The following example groups `camp` and aggregates `occupation` amounts in the database. The `GROUPING` function is applied to the `camp` column. + +```sql +CREATE TABLE `roles` ( + role_id INT, + occupationVARCHAR(32), + camp VARCHAR(32), + register_time DATE +) +UNIQUE KEY(role_id) +DISTRIBUTED BY HASH(role_id) BUCKETS 1 +PROPERTIES ( + "replication_allocation" = "tag.location.default: 1" +); + +INSERT INTO `roles` VALUES +(0, 'who am I', NULL, NULL), +(1, 'mage', 'alliance', '2018-12-03 16:11:28'), +(2, 'paladin', 'alliance', '2018-11-30 16:11:28'), +(3, 'rogue', 'horde', '2018-12-01 16:11:28'), +(4, 'priest', 'alliance', '2018-12-02 16:11:28'), +(5, 'shaman', 'horde', NULL), +(6, 'warrior', 'alliance', NULL), +(7, 'warlock', 'horde', '2018-12-04 16:11:28'), +(8, 'hunter', 'horde', NULL); + +SELECT + camp, + COUNT(occupation) AS 'occ_cnt', + GROUPING(camp)AS 'grouping' +FROM + `roles` +GROUP BY + ROLLUP(camp); -- CUBE(camp) and GROUPING SETS((camp)) also can work; +``` + +The result set shows two null value under `camp`. The first NULL is in the summary row added by the `ROLLUP` operation. The summary row shows the occupation counts for all `camp` groups and is indicated by 1 in the Grouping column. The second NULL represents the group of null values from this column in the table. + +Here is the result set. + +```log ++--+-+--+ +| camp | occ_cnt | grouping | ++--+-+--+ +| NULL | 9 |1 | +| NULL | 1 |0 | +| alliance | 4 |0 | +| horde| 4 |0 | ++--+-+--+ +4 rows in set (0.01 sec) ``` -### keywords -GROUPING +### See Also + +[GROUPING_ID](./grouping_id.md) diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md b/docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md index e990ab2c30..5943205c14 100755 --- a/docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md +++ b/docs/zh-CN/docs/sql-manual/sql-functions/aggregate-functions/grouping.md @@ -25,25 +25,82 @@ under the License. --> ## GROUPING -### description - Syntax -`GROUPING(expr)` +
