[GitHub] [incubator-doris] morningman opened a new pull request #7173: [doc] add FAQ document
morningman opened a new pull request #7173: URL: https://github.com/apache/incubator-doris/pull/7173 ## Proposed changes From Apache Doris wechat count, authorized. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7025: Add tablets number limit to affend wrong usage
github-actions[bot] commented on pull request #7025: URL: https://github.com/apache/incubator-doris/pull/7025#issuecomment-975297628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 commented on a change in pull request #7131: Refactor olap scannode
Gabriel39 commented on a change in pull request #7131:
URL: https://github.com/apache/incubator-doris/pull/7131#discussion_r754122278
##
File path: be/src/exec/olap_scan_node.h
##
@@ -81,8 +80,8 @@ class OlapScanNode : public ScanNode {
};
typedef std::priority_queue,
MergeComparison> Heap;
-
-void display_heap(Heap& heap) {
+
Review comment:
my IDE has the same behavior as you for a blank line. I wonder should we
keep non-space and non-tab style before a blank line? @morningman
##
File path: be/src/exec/olap_scan_node.cpp
##
@@ -58,8 +58,7 @@ OlapScanNode::OlapScanNode(ObjectPool* pool, const TPlanNode&
tnode, const Descr
_buffered_bytes(0),
_eval_conjuncts_fn(nullptr),
_runtime_filter_descs(tnode.runtime_filters) {}
-
-OlapScanNode::~OlapScanNode() {}
+
Review comment:
keep single blank line
##
File path: be/src/runtime/fragment_mgr.h
##
@@ -58,7 +58,7 @@ class FragmentMgr : public RestMonitorIface {
using FinishCallback = std::function;
FragmentMgr(ExecEnv* exec_env);
-virtual ~FragmentMgr();
+virtual ~FragmentMgr();//why virtual dtor?
Review comment:
what does this comment mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #7174: [doc] Add build-dev image 1.4.2 to compilation document
morningman opened a new pull request #7174: URL: https://github.com/apache/incubator-doris/pull/7174 ## Proposed changes Add build-dev image 1.4.2 to compilation document ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7098: Support remote storage, step1: use a struct instead of string for parameter path, add basic remote method
morningman commented on a change in pull request #7098:
URL: https://github.com/apache/incubator-doris/pull/7098#discussion_r754038577
##
File path: be/src/env/env_remote.cpp
##
@@ -0,0 +1,333 @@
+// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
Review comment:
header is incorrect
##
File path: be/src/env/env_remote.cpp
##
@@ -0,0 +1,333 @@
+// Copyright (c) 2011-present, Facebook, Inc. All rights reserved.
+// This source code is licensed under both the GPLv2 (found in the
+// COPYING file in the root directory) and Apache 2.0 License
+// (found in the LICENSE.Apache file in the root directory).
+//
+// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style license that can be
+// found in the LICENSE file. See the AUTHORS file for names of contributors
+
+#include "common/logging.h"
+#include "common/status.h"
+#include "env/env.h"
+#include "env/env_remote.h"
+#include "util/s3_storage_backend.h"
+#include "util/s3_util.h"
+
+#include "gutil/strings/substitute.h"
+
+namespace doris {
+
+using std::string;
+using strings::Substitute;
+
+std::map RemoteEnv::_storage_prop;
+
+class RemoteRandomAccessFile : public RandomAccessFile {
+public:
+RemoteRandomAccessFile(std::string filename, std::map storage_prop)
+: _filename(std::move(filename)), _storage_backend(new
S3StorageBackend(storage_prop)) {}
+~RemoteRandomAccessFile() {
+}
+
+Status read_at(uint64_t offset, const Slice* result) const override {
+return readv_at(offset, result, 1);
+}
+
+Status readv_at(uint64_t offset, const Slice* result, size_t res_cnt)
const override {
+return Status::IOError("No support", 1, "");
Review comment:
```suggestion
return Status::NotSupport("No support", 1, "");
```
##
File path: be/src/olap/data_dir.cpp
##
@@ -77,9 +74,13 @@ DataDir::DataDir(const std::string& path, int64_t
capacity_bytes,
_tablet_manager(tablet_manager),
_txn_manager(txn_manager),
_cluster_id(-1),
+ _cluster_id_incomplete(false),
_to_be_deleted(false),
_current_shard(0),
_meta(nullptr) {
+_env = Env::get_env(storage_medium);
+_path_desc.storage_medium = storage_medium;
Review comment:
put these to initialize list
##
File path: be/src/olap/fs/block_manager.h
##
@@ -229,7 +236,7 @@ class BlockManager {
// may fail.
//
// Does not modify 'block' on error.
-virtual Status open_block(const std::string& path,
std::unique_ptr* block) = 0;
+virtual Status open_block(FilePathDesc path_desc,
std::unique_ptr* block) = 0;
Review comment:
```suggestion
virtual Status open_block(const FilePathDesc& path_desc,
std::unique_ptr* block) = 0;
```
##
File path: be/src/olap/fs/file_block_manager.cpp
##
@@ -63,7 +63,7 @@ namespace internal {
// FileWritableBlock instances is expected to be low.
class FileWritableBlock : public WritableBlock {
public:
-FileWritableBlock(FileBlockManager* block_manager, string path,
+FileWritableBlock(FileBlockManager* block_manager, FilePathDesc path_desc,
Review comment:
```suggestion
FileWritableBlock(FileBlockManager* block_manager, const FilePathDesc&
path_desc,
```
##
File path: be/src/util/file_utils.cpp
##
@@ -41,58 +41,20 @@ namespace doris {
using strings::Substitute;
Status FileUtils::create_dir(const std::string& path, Env* env) {
-if (path.empty()) {
-return Status::InvalidArgument(strings::Substitute("Unknown primitive
type($0)", path));
-}
-
-std::filesystem::path p(path);
-
-std::string partial_path;
-for (std::filesystem::path::iterator it = p.begin(); it != p.end(); ++it) {
-partial_path = partial_path + it->string() + "/";
-bool is_dir = false;
-
-Status s = env->is_directory(partial_path, &is_dir);
-
-if (s.ok()) {
-if (is_dir) {
-// It's a normal directory.
-continue;
-}
-
-// Maybe a file or a symlink. Let's try to follow the symlink.
-std::string real_partial_path;
-RETURN_IF_ERROR(env->canonicalize(partial_path,
&real_partial_path));
-
-RETURN_IF_ERROR(env->is_directory(real_partial_path, &is_dir));
-if (is_dir) {
-// It's a symlink to a directory.
-continue;
-} else {
-return Status::IOError(partial_path + " exists but is not a
directory");
-}
-}
-
-RETURN_IF_ERROR(env->create_dir_if_missing(partial_path));
-}
-
-return Status::OK();
+return env->create_dirs(path);
}
Status FileUtils::create_dir(const std::string& dir_path) {
return create_dir(dir_path, Env::Default());
}
Status FileUtils::re
[GitHub] [incubator-doris] luzhijing closed pull request #7156: [docs]Created commit-format-specification.md
luzhijing closed pull request #7156: URL: https://github.com/apache/incubator-doris/pull/7156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7167: [optimize]Add Doris restart script, optimize doris start and stop commands
morningman commented on a change in pull request #7167:
URL: https://github.com/apache/incubator-doris/pull/7167#discussion_r754179070
##
File path: bin/doris_be.sh
##
@@ -0,0 +1,61 @@
+#!/bin/bash
Review comment:
License header.
##
File path: bin/doris_be.sh
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+export DORIS_HOME=`cd "$curdir"; pwd`
+export LOG_DIR=${DORIS_HOME}/log
Review comment:
the `LOG_DIR` should read from config file.
##
File path: bin/doris_be.sh
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+export DORIS_HOME=`cd "$curdir"; pwd`
+export LOG_DIR=${DORIS_HOME}/log
+export PID_DIR=`cd "$curdir"; pwd`
+function start() {
+echo "Starting Apache Doris Frontend"
+sh $DORIS_HOME/bin/start_be.sh --daemon
+if [ -f "$DORIS_HOME/bin/be.pid" ]; then
+ echo "Apache Doris Backend start fail!"
+else
+ sleep 1s
+ pid=`cat $DORIS_HOME/bin/be.pid`
+ echo "Doris Be pid : ${pid}"
+ echo `sed -n '$p' $DORIS_HOME/log/be.out`
+ echo "Apache Doris Backend start success!"
+fi
+}
+
+function stop() {
+echo "stop Apache Doris Frontend"
+sh $DORIS_HOME/bin/stop_be.sh
+if [ -f "$DORIS_HOME/bin/be.pid" ]; then
+ echo "Apache Doris Backend stop fail!"
+else
+ echo "Apache Doris Backend stop success!"
+fi
+}
+
+function restart() {
+echo "Restart Apache Doris Backend"
+echo ""
+echo "Ready to stop Doris Backend"
+stop
Review comment:
The process may be in defunct status for a long time, so start right
after stop may failed to start BE process.
##
File path: bin/doris_be.sh
##
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+export DORIS_HOME=`cd "$curdir"; pwd`
+export LOG_DIR=${DORIS_HOME}/log
+export PID_DIR=`cd "$curdir"; pwd`
+function start() {
+echo "Starting Apache Doris Frontend"
+sh $DORIS_HOME/bin/start_be.sh --daemon
+if [ -f "$DORIS_HOME/bin/be.pid" ]; then
+ echo "Apache Doris Backend start fail!"
+else
+ sleep 1s
+ pid=`cat $DORIS_HOME/bin/be.pid`
+ echo "Doris Be pid : ${pid}"
+ echo `sed -n '$p' $DORIS_HOME/log/be.out`
Review comment:
the log dir may be elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7054: [Feature] Refactor predicate function creator
github-actions[bot] commented on pull request #7054: URL: https://github.com/apache/incubator-doris/pull/7054#issuecomment-975433026 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7109: [Doc]Modify the wrong comment of the ScanTime
github-actions[bot] commented on pull request #7109: URL: https://github.com/apache/incubator-doris/pull/7109#issuecomment-975433439 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X opened a new issue #7179: [Bug] insert load should have the same behavior with stream load
spaces-X opened a new issue #7179:
URL: https://github.com/apache/incubator-doris/issues/7179
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
version 0.14 or older
### What's Wrong?
The config `default_max_filter_ratio` does not work in the INSERT statement.
If not set `ENABLE_INSERT_STRICT = true`, most of the data may be filtered
without notice, which does not match the expectation of
`default_max_filter_ratio`
### What You Expected?
Insert load should have the same behavior with stream load.
If `default_max_filter_ratio` is set, the number of filtered rows should not
be more than `default_max_filter_ratio` * `total_rows` in the insert load.
### How to Reproduce?
DDL:
1.
```
CREATE TABLE `unique_null_bitmap` (
`dt` int(11) NULL COMMENT "日期分区",
`city_id_bitmap` bitmap BITMAP_UNION COMMENT "city_id_bitmap"
) ENGINE=OLAP
AGGREGATE KEY(`dt`)
COMMENT "交易轻度聚合表"
PARTITION BY RANGE(`dt`)
(PARTITION p20211012 VALUES [("20211011"), ("20211012")),
PARTITION p20211109 VALUES [("20211108"), ("20211109")))
DISTRIBUTED BY HASH(`dt`) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
```
2.
```
CREATE TABLE `agg_bug_test` (
`dt` bigint(20) NULL COMMENT "",
`id1` bigint(20) NULL COMMENT "",
`id2` bigint(20) NULL COMMENT "",
`id3` bigint(20) NULL COMMENT "",
`first_time` varchar(20) MIN NULL COMMENT "",
`last_time` varchar(20) MAX NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(`dt`, `id1`, `id2`, `id3`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id1`, `id2`, `id3`) BUCKETS 10
PROPERTIES (
"replication_num" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
```
```
insert into agg_bug_test values (20211108,10,10,10,null,null);
insert into agg_bug_test values (20210111,10,10,10,null,null);
```
Insert SQL: 1 row inserted and 8 rows filtered without notice.
Insert transaction should be aborted when `default_max_filter_ratio` is set.
```
insert into unique_null_bitmap (dt,city_id_bitmap) (
(select dt, bitmap_empty() as city_id_bitmap from agg_bug_test where dt
= 20211108 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1) union all
(select dt, null as city_id_bitmap from agg_bug_test where dt =
20211011 limit 1)
);
```
Query: only returned one row with dt = 20211108
```
select * from unique_null_bitmap
```
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X opened a new pull request #7180: [Bug] fix insert load should have the same behavior as stream load
spaces-X opened a new pull request #7180: URL: https://github.com/apache/incubator-doris/pull/7180 ## Proposed changes see #7179 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new pull request #7181: [Bug] Fix mem limit -1 in partition aggregate node
HappenLee opened a new pull request #7181: URL: https://github.com/apache/incubator-doris/pull/7181 ## Proposed changes Fix mem limit -1 in partition aggregate node ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [x] I have added tests that prove my fix is effective or that my feature works - [x] If these changes need document changes, I have updated the document - [x] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7181: [Bug] Fix mem limit -1 in partition aggregate node
morningman commented on a change in pull request #7181:
URL: https://github.com/apache/incubator-doris/pull/7181#discussion_r754758749
##
File path: be/src/exec/partitioned_aggregation_node.cc
##
@@ -849,7 +849,7 @@ Status PartitionedAggregationNode::Partition::Spill(bool
more_aggregate_rows) {
// TODO(ml): enable spill
std::stringstream msg;
msg << "New partitioned Aggregation in spill";
-LIMIT_EXCEEDED(parent->mem_tracker(), parent->state_, msg.str());
+LIMIT_EXCEEDED(parent->state_->query_mem_tracker(), parent->state_,
msg.str());
Review comment:
I think you can just use `instance_mem_tracker()`
cc @xinyiZzz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7132: [Clang-Format] Add Clang Format/Syswalking eyes github action
github-actions[bot] commented on pull request #7132: URL: https://github.com/apache/incubator-doris/pull/7132#issuecomment-976101008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7173: [doc] add FAQ document
github-actions[bot] commented on pull request #7173: URL: https://github.com/apache/incubator-doris/pull/7173#issuecomment-976101161 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7174: [doc] Add build-dev image 1.4.2 to compilation document
github-actions[bot] commented on pull request #7174: URL: https://github.com/apache/incubator-doris/pull/7174#issuecomment-976101286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] luzhijing opened a new pull request #7182: [docs]Created commit-format-specification.md
luzhijing opened a new pull request #7182: URL: https://github.com/apache/incubator-doris/pull/7182 ## Proposed changes We found that many commit messages submitted at present have ambiguous information. Clear commit messages can help developers submit pull requests more readable, committers merge easilerly and Release Manager easy to release. Therefore, we have sorted out a version of the commit format specification. We hope that subsequent contributors can sort out the commit messages according to the specification when submitting Pull Request. 我们发现当前提交的很多Commit Message存在信息模糊的情况，清晰的Commit Message可以帮助开发者提交的Pull Request更加易读，便于Committer合入，也便于Release Manager进行发版。因此我们整理了一版Commit格式规范，希望后续Contributor在提交PR时可以按照规范整理Commit Message。 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [x] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #7181: [Bug] Fix mem limit -1 in partition aggregate node
HappenLee commented on a change in pull request #7181:
URL: https://github.com/apache/incubator-doris/pull/7181#discussion_r754773344
##
File path: be/src/exec/partitioned_aggregation_node.cc
##
@@ -849,7 +849,7 @@ Status PartitionedAggregationNode::Partition::Spill(bool
more_aggregate_rows) {
// TODO(ml): enable spill
std::stringstream msg;
msg << "New partitioned Aggregation in spill";
-LIMIT_EXCEEDED(parent->mem_tracker(), parent->state_, msg.str());
+LIMIT_EXCEEDED(parent->state_->query_mem_tracker(), parent->state_,
msg.str());
Review comment:
`instance_mem_tracker()` do not have mem limit so the limit of
`instance_mem_tracker` is -1. `instanc_mem_tracker` is child of
`query_mem_tracker()` which have real limit number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] littleeleventhwolf opened a new pull request #7183: [BUG] Fix `curdate()+0` behave like mysql
littleeleventhwolf opened a new pull request #7183: URL: https://github.com/apache/incubator-doris/pull/7183 ## Proposed changes I proposed this point in [PR-6005](https://github.com/apache/incubator-doris/pull/6005). Now I try to make `curdate()+0` behave like mysql. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] LangYuanzh opened a new issue #7185: 【routine load】kafka 消息有堆积的情况下为什么仍然存在 abortedTask
LangYuanzh opened a new issue #7185: URL: https://github.com/apache/incubator-doris/issues/7185 ### Discussed in https://github.com/apache/incubator-doris/discussions/7184 Originally posted by **LangYuanzh** November 23, 2021 环境信息 ： 1. kafka topic 3个patition 2. doris 1fe 3be 3. routine load：PROPERTIES ( "desired_concurrent_number" = "3" ) 问题描述： 通过 kafka 导入 100w条数据，kafka消息一直有堆积的情况下，仍然存在abortedTask 任务初始状态： "committedTaskNum":16,"loadedRows":160759,"loadRowsRate":1000,"abortedTaskNum":8,"totalRows":160759, 任务中间状态： "committedTaskNum":48,"loadedRows":578200,"loadRowsRate":2000,"abortedTaskNum":14,"totalRows":578200, 期间kafka消息一直存在堆积，但是abortedTaskNum仍然增加到了14。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] luzhijing closed pull request #7182: [docs]Created commit-format-specification.md
luzhijing closed pull request #7182: URL: https://github.com/apache/incubator-doris/pull/7182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] luzhijing opened a new pull request #7190: [doc]Created commit-format-specification.md
luzhijing opened a new pull request #7190: URL: https://github.com/apache/incubator-doris/pull/7190 ## Proposed changes We found that many commit messages submitted at present have ambiguous information. Clear commit messages can help developers submit pull requests more readable, committers merge easilerly and Release Manager easy to release. Therefore, we have sorted out a version of the commit format specification. We hope that subsequent contributors can sort out the commit messages according to the specification when submitting Pull Request. 我们发现当前提交的很多Commit Message存在信息模糊的情况，清晰的Commit Message可以帮助开发者提交的Pull Request更加易读，便于Committer合入，也便于Release Manager进行发版。因此我们整理了一版Commit格式规范，希望后续Contributor在提交PR时可以按照规范整理Commit Message。 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [x] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] luzhijing closed pull request #7189: [docs]Created commit-format-specification.md
luzhijing closed pull request #7189: URL: https://github.com/apache/incubator-doris/pull/7189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] minghong opened a new issue #7191: [Bug] compile error
minghong opened a new issue #7191: URL: https://github.com/apache/incubator-doris/issues/7191 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version commit ce7fa5d6d9454d3b89e75859690fb216ca68167b (HEAD -> master, origin/master, origin/HEAD) Author: renzhimin7 <1240388...@qq.com> Date: Mon Nov 22 14:47:00 2021 +0800 [typo] Update multi-tenant.md (#7162) A double quote is missing ### What's Wrong? compilation using docker image failed. docker image: apache/incubator-doris:build-env-1.4.1 ``` sh build.sh --clean --be --fe --ui ``` we got error: ``` ninja: error: '/var/local/thirdparty/installed/lib64/libbreakpad_client.a', needed by 'src/service/palo_be', missing and no known rule to make it``` ### What You Expected? compilation success ### How to Reproduce? docker pull apache/incubator-doris:build-env-1.4.1 git clone g...@github.com:minghong/talent-plan.git docker run -it -v ~/.m2:/root/.m2 -v ~/work/incubator-doris/:/root/incubator-doris-DORIS-x.x.x-release/ apache/incubator-doris:build-env-1.4.1 inside docker container: [root@7eec959b89ab incubator-doris-DORIS-x.x.x-release]# sh build.sh --clean --be --fe --ui We got following error message -- The ASM compiler identification is GNU -- Found assembler: /opt/rh/devtoolset-10/root/usr/bin/gcc -- Looking for sys/random.h -- Looking for sys/random.h - not found -- Some symbols from glibc will be replaced for compatibility -- Configuring done -- Generating done -- Build files have been written to: /root/incubator-doris-DORIS-x.x.x-release/be/build_Release [0/2] Re-checking globbed directories... ninja: error: '/var/local/thirdparty/installed/lib64/libbreakpad_client.a', needed by 'src/service/palo_be', missing and no known rule to make it ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] e0c9 commented on pull request #7157: [enhancement] Optimize get_json_xx function
e0c9 commented on pull request #7157: URL: https://github.com/apache/incubator-doris/pull/7157#issuecomment-976194131 > The BE unit test failed. And if you change the signature of functions, it may cause incompatible issue. Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7157: [enhancement] Optimize get_json_xx function
morningman commented on pull request #7157: URL: https://github.com/apache/incubator-doris/pull/7157#issuecomment-976195531 > And if you change the signature of functions, it may cause incompatible issue. hi @e0c9 What about this problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new issue #7192: [Bug] preceding filter may cause BE crash
morningman opened a new issue #7192:
URL: https://github.com/apache/incubator-doris/issues/7192
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
0.15
### What's Wrong?
When loading data with preceding filter like:
```
LOAD LABEL p20210107 (
DATA INFILE("s3://cmy-repo/1.csv")
INTO TABLE testTable
COLUMNS TERMINATED BY ","
PRECEDING FILTER col1 = "abc"
)
WITH S3 (
xxx
)
properties(
xxx
);
```
BE may crash with stack:
```
*** Aborted at 1634723518 (unix time) try "date -d @1634723518" if you are
using GNU date ***
PC: @ 0x113941a doris::SlotRef::get_string_val()
*** SIGSEGV (@0x0) received by PID 26168 (TID 0x7fc03b949700) from PID 0;
stack trace: ***
@ 0x1fd12e2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fc0f8d6a630 (unknown)
@ 0x113941a doris::SlotRef::get_string_val()
@ 0x1122df8 doris::ExprContext::get_value()
@ 0x11336a7 doris::ScalarFnCall::evaluate_children()
@ 0x11376ae doris::ScalarFnCall::interpret_eval<>()
@ 0x119c0c3 doris::EqDoubleValPred::get_boolean_val()
@ 0x18d7d82 doris::ExecNode::eval_conjuncts()
@ 0x1a3d191 doris::BaseScanner::fill_dest_tuple()
@ 0x1a2b84c doris::JsonScanner::get_next()
@ 0x19f1681 doris::BrokerScanNode::scanner_scan()
@ 0x19f21cc doris::BrokerScanNode::scanner_worker()
@ 0x2ff66e0 execute_native_thread_routine
@ 0x7fc0f8d62ea5 start_thread
@ 0x7fc0f87899fd __clone
```
### What You Expected?
works well
### How to Reproduce?
use preceding filter feature
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] e0c9 commented on pull request #7157: [enhancement] Optimize get_json_xx function
e0c9 commented on pull request #7157: URL: https://github.com/apache/incubator-doris/pull/7157#issuecomment-976200987 > > And if you change the signature of functions, it may cause incompatible issue. > > hi @e0c9 What about this problem? I modified it locally based on the old version. json_string's type has changed. `const std::string& json_string` -> `const std::string_view& json_string` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #7193: [fix](broker-load) BE may crash when using preceding filter in broker or routine load
morningman opened a new pull request #7193: URL: https://github.com/apache/incubator-doris/pull/7193 ## Proposed changes The broker scan node has two tuple descriptors: One is dest tuple and the other is src tuple. The src tuple is used to read the lines of the original file, and the dest tuple is used to save the converted lines. The preceding filter is executed on the src tuple, so src tuple descriptor should be used to initialize the filter expression ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #7192 ) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye commented on issue #7078: [Bug] ES Extern Table Crash
qidaye commented on issue #7078:
URL:
https://github.com/apache/incubator-doris/issues/7078#issuecomment-976209643
I can not reproduce this bug.
Here is my steps to reproduce:
1. Turn on ASAN before build BE, adding `export BUILD_TYPE=ASAN` in
`custom_env.sh`
2. Create index in ES
```
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null?pretty -d '{
"mappings" : {
"properties" : {
"id_int" : {
"type" : "long"
},
"id_int1000" : {
"type" : "long"
},
"id_tinyint" : {
"type" : "integer"
},
"id_smallint" : {
"type" : "integer"
},
"id_float" : {
"type" : "float"
},
"id_double" : {
"type" : "double"
},
"id_char" : {
"type" : "text"
},
"id_varchar" : {
"type" : "text"
},
"id_varchar_int" : {
"type" : "text"
},
"id_varchar_date" : {
"type" : "text"
},
"id_varchar_datetime" : {
"type" : "text"
},
"id_date" : {
"type" : "date",
"format": "-MM-dd HH:mm:ss||-MM-dd||epoch_millis"
},
"id_datetime" : {
"type" : "date",
"format": "-MM-dd HH:mm:ss||-MM-dd||epoch_millis"
},
"id_decimal" : {
"type" : "text"
},
"id_decimal10" : {
"type" : "text"
},
"id_boolean" : {
"type" : "boolean"
}
}
}
}'
```
3. Index docs to the index
```
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null/_doc/1?pretty -d '{"id_int":11,
"id_int1000":11, "id_tinyint":1, "id_smallint":1, "id_float":1.0,
"id_double":1.0, "id_char":"aa", "id_varchar":"aa", "id_varchar_int":"1",
"id_varchar_date":"2020-10-01", "id_varchar_datetime":"2020-10-10 10:10:10",
"id_date":"2020-10-10", "id_datetime":"2020-10-10 10:10:10",
"id_decimal":"222", "id_decimal10":"222", "id_boolean":true}'
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null/_doc/2?pretty -d '{"id_int":null,
"id_int1000":null, "id_tinyint":null, "id_smallint":null, "id_float":null,
"id_double":null, "id_char":null, "id_varchar":null, "id_varchar_int":null,
"id_varchar_date":null, "id_varchar_datetime":null, "id_date":null,
"id_datetime":null, "id_decimal":null, "id_decimal10":null, "id_boolean":null}'
```
4. Create external ES table in Doris
```sql
CREATE EXTERNAL TABLE external_es_table_without_null(
id_int int(11) NOT NULL COMMENT "",
id_int1000 int(11) NOT NULL COMMENT "",
id_tinyint tinyint(4) NOT NULL COMMENT "",
id_smallint smallint(6) NOT NULL COMMENT "",
id_float float NOT NULL COMMENT "",
id_double double NOT NULL COMMENT "",
id_char char(255) NOT NULL COMMENT "",
id_varchar varchar(65533) NOT NULL COMMENT "",
id_varchar_int varchar(65533) NOT NULL COMMENT "",
id_varchar_date varchar(65533) NOT NULL COMMENT "",
id_varchar_datetime varchar(65533) NOT NULL COMMENT "",
id_date date NOT NULL COMMENT "",
id_datatime datetime NOT NULL COMMENT "",
id_decimal varchar(65533) NOT NULL COMMENT "",
id_decimal10 varchar(65533) NOT NULL COMMENT "",
id_boolean boolean NOT NULL COMMENT ""
)
ENGINE=ELASTICSEARCH COMMENT "ELASTICSEARCH"
PROPERTIES (
"hosts" = "http://";,
"user" = "root",
"password" = "root",
"index" = "doris_external_data_not_null",
"type" = "_doc",
"transport" = "http",
"enable_docvalue_scan" = "true",
"max_docvalue_fields" = "20",
"enable_keyword_sniff" = "true"
);
```
5. Select ES table in Doris
```sql
select * from external_es_table_without_null as ref_0 where ( cast(id_int as
string) is NULL);
select * from external_es_table_without_null as ref_0 where ( if( cast( case
when ref_0.id_char > ref_0.id_varchar_int then false else false end as BOOLEAN
), cast(ref_0.id_tinyint as INT), cast(ref_0.id_boolean as INT) ) is NULL );
```
The result is show normal and BE dose not crash. If any step is wrong,
please let me know.
I have referenced the steps to reproduce the bug in [1]
1. https://github.com/StarRocks/starrocks/issues/1172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.o
[GitHub] [incubator-doris] qidaye edited a comment on issue #7078: [Bug] ES Extern Table Crash
qidaye edited a comment on issue #7078:
URL:
https://github.com/apache/incubator-doris/issues/7078#issuecomment-976209643
I can not reproduce this bug.
Here is my steps to reproduce:
1. Turn on ASAN before build BE, adding `export BUILD_TYPE=ASAN` in
`custom_env.sh`
2. Create index in ES
```
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null?pretty -d '{
"mappings" : {
"properties" : {
"id_int" : {
"type" : "long"
},
"id_int1000" : {
"type" : "long"
},
"id_tinyint" : {
"type" : "integer"
},
"id_smallint" : {
"type" : "integer"
},
"id_float" : {
"type" : "float"
},
"id_double" : {
"type" : "double"
},
"id_char" : {
"type" : "text"
},
"id_varchar" : {
"type" : "text"
},
"id_varchar_int" : {
"type" : "text"
},
"id_varchar_date" : {
"type" : "text"
},
"id_varchar_datetime" : {
"type" : "text"
},
"id_date" : {
"type" : "date",
"format": "-MM-dd HH:mm:ss||-MM-dd||epoch_millis"
},
"id_datetime" : {
"type" : "date",
"format": "-MM-dd HH:mm:ss||-MM-dd||epoch_millis"
},
"id_decimal" : {
"type" : "text"
},
"id_decimal10" : {
"type" : "text"
},
"id_boolean" : {
"type" : "boolean"
}
}
}
}'
```
3. Index docs to the index
```
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null/_doc/1?pretty -d '{"id_int":11,
"id_int1000":11, "id_tinyint":1, "id_smallint":1, "id_float":1.0,
"id_double":1.0, "id_char":"aa", "id_varchar":"aa", "id_varchar_int":"1",
"id_varchar_date":"2020-10-01", "id_varchar_datetime":"2020-10-10 10:10:10",
"id_date":"2020-10-10", "id_datetime":"2020-10-10 10:10:10",
"id_decimal":"222", "id_decimal10":"222", "id_boolean":true}'
curl -u root:root -sXPUT -H 'Content-Type:application/json'
http:///doris_external_data_not_null/_doc/2?pretty -d '{"id_int":null,
"id_int1000":null, "id_tinyint":null, "id_smallint":null, "id_float":null,
"id_double":null, "id_char":null, "id_varchar":null, "id_varchar_int":null,
"id_varchar_date":null, "id_varchar_datetime":null, "id_date":null,
"id_datetime":null, "id_decimal":null, "id_decimal10":null, "id_boolean":null}'
```
4. Create external ES table in Doris
```sql
CREATE EXTERNAL TABLE external_es_table_without_null(
id_int int(11) NOT NULL COMMENT "",
id_int1000 int(11) NOT NULL COMMENT "",
id_tinyint tinyint(4) NOT NULL COMMENT "",
id_smallint smallint(6) NOT NULL COMMENT "",
id_float float NOT NULL COMMENT "",
id_double double NOT NULL COMMENT "",
id_char char(255) NOT NULL COMMENT "",
id_varchar varchar(65533) NOT NULL COMMENT "",
id_varchar_int varchar(65533) NOT NULL COMMENT "",
id_varchar_date varchar(65533) NOT NULL COMMENT "",
id_varchar_datetime varchar(65533) NOT NULL COMMENT "",
id_date date NOT NULL COMMENT "",
id_datatime datetime NOT NULL COMMENT "",
id_decimal varchar(65533) NOT NULL COMMENT "",
id_decimal10 varchar(65533) NOT NULL COMMENT "",
id_boolean boolean NOT NULL COMMENT ""
)
ENGINE=ELASTICSEARCH COMMENT "ELASTICSEARCH"
PROPERTIES (
"hosts" = "http://";,
"user" = "root",
"password" = "root",
"index" = "doris_external_data_not_null",
"type" = "_doc",
"transport" = "http",
"enable_docvalue_scan" = "true",
"max_docvalue_fields" = "20",
"enable_keyword_sniff" = "true"
);
```
5. Select ES table in Doris
```sql
select * from external_es_table_without_null as ref_0 where ( cast(id_int as
string) is NULL);
select * from external_es_table_without_null as ref_0 where ( if( cast( case
when ref_0.id_char > ref_0.id_varchar_int then false else false end as BOOLEAN
), cast(ref_0.id_tinyint as INT), cast(ref_0.id_boolean as INT) ) is NULL );
```
The result is show normally and BE dose not crash. If any step is wrong,
please let me know.
I have referenced the steps to reproduce the bug in [1]
1. https://github.com/StarRocks/starrocks/issues/1172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra
[GitHub] [incubator-doris] qidaye commented on pull request #7079: Fix ES Extern Table Crash
qidaye commented on pull request #7079: URL: https://github.com/apache/incubator-doris/pull/7079#issuecomment-976211328 I can not reproduce this bug. I add my steps to reproduce in the issues. If my steps are wrong, please let me know. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] anneji-dev opened a new pull request #7194: Predicates contains invalid date should be throw exception
anneji-dev opened a new pull request #7194:
URL: https://github.com/apache/incubator-doris/pull/7194
## Proposed changes
When predicates contains invalid date should be throw exception, MYSQL
database is like this.
cases are as follows:
create table t3 (tc1 datetime,tc2 datetime);
insert into t3 values('2020-1-11 00:00:00','2020-1-11 00:00:00');
select tc1 from t3 where tc2='2020-10-34 20:06:20';
This performance is abnormal,should be like Mysql database throw exception
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and
performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature
there in detail
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at
d...@doris.apache.org by explaining why you chose the solution you did and what
alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] minghong opened a new pull request #7195: [fix][Bug] compile error (#7191)
minghong opened a new pull request #7195: URL: https://github.com/apache/incubator-doris/pull/7195 update compilation.md -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz opened a new issue #7196: [Feature] Refactored memory statistics framework MemTracker
xinyiZzz opened a new issue #7196: URL: https://github.com/apache/incubator-doris/issues/7196 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description https://shimo.im/docs/DT6JXDRkdTvdyV3G ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7195: [fix][Bug] compile error (#7191)
morningman commented on pull request #7195: URL: https://github.com/apache/incubator-doris/pull/7195#issuecomment-976219275 Hi @minghong , thanks, but it duplicated with #7174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7181: [Bug] Fix mem limit -1 in partition aggregate node
github-actions[bot] commented on pull request #7181: URL: https://github.com/apache/incubator-doris/pull/7181#issuecomment-976219616 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei opened a new pull request #7197: [Fix compile bug] could not find protobuf lib during compile
yiguolei opened a new pull request #7197: URL: https://github.com/apache/incubator-doris/pull/7197 Failed to compile thirdparty brpc package, because could not find protobuf lib -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7172: [Memory] fix BUG crash when ~BitShufflePageDecoder destroys uninitialized chunk
github-actions[bot] commented on pull request #7172: URL: https://github.com/apache/incubator-doris/pull/7172#issuecomment-976220066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7172: [Memory] fix BUG crash when ~BitShufflePageDecoder destroys uninitialized chunk
morningman merged pull request #7172: URL: https://github.com/apache/incubator-doris/pull/7172 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (ce7fa5d -> ad0d2b8)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from ce7fa5d [typo] Update multi-tenant.md (#7162) add ad0d2b8 [fix](memory) fix bug that ~BitShufflePageDecoder destroys uninitialized chunk (#7172) No new revisions were added by this update. Summary of changes: be/src/olap/rowset/segment_v2/bitshuffle_page.h | 4 +--- be/src/runtime/memory/chunk.h | 6 +++--- be/src/runtime/memory/chunk_allocator.cpp | 4 +++- 3 files changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7171: [Bug] ~BitShufflePageDecoder destroys uninitialized chunk
morningman closed issue #7171: URL: https://github.com/apache/incubator-doris/issues/7171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7190: [doc]Created commit-format-specification.md
github-actions[bot] commented on pull request #7190: URL: https://github.com/apache/incubator-doris/pull/7190#issuecomment-976221088 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7197: [Fix compile bug] could not find protobuf lib during compile
morningman commented on a change in pull request #7197:
URL: https://github.com/apache/incubator-doris/pull/7197#discussion_r754858875
##
File path: thirdparty/build-thirdparty.sh
##
@@ -502,6 +502,7 @@ build_brpc() {
rm -rf CMakeCache.txt CMakeFiles/
LDFLAGS="-L${TP_LIB_DIR} -static-libstdc++ -static-libgcc" \
${CMAKE_CMD} -G "${GENERATOR}" -DBUILD_SHARED_LIBS=0
-DCMAKE_INSTALL_PREFIX=$TP_INSTALL_DIR \
+-DCMAKE_LIBRARY_PATH="$TP_INSTALL_DIR/lib;$TP_INSTALL_DIR/lib64" \
Review comment:
```suggestion
-DCMAKE_LIBRARY_PATH="$TP_INSTALL_DIR/lib64" \
```
`lib` is just a soft link of `lib64`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz opened a new pull request #7198: [Memory] Use TCMalloc Hook to count the real Process and Query MemTracker
xinyiZzz opened a new pull request #7198: URL: https://github.com/apache/incubator-doris/pull/7198 ## Proposed changes 1. Add thread local variable ThreadStatus, save queryID, MemTracker, etc. 2. Use TCMalloc Hook to consume and release MemTracker in ThreadStatus. 3. Add real BE process level MemTracker, consume and release through TCMalloc Hook. 4. Add the real Query MemTracker, a queryID is unique in a BE process, consume and release through TCMalloc Hook. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #7196) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7151: Optimize bitmap function count, use roaring cardinality method
morningman commented on a change in pull request #7151:
URL: https://github.com/apache/incubator-doris/pull/7151#discussion_r754001296
##
File path: be/src/exprs/bitmap_function.cpp
##
@@ -333,10 +333,10 @@ BigIntVal BitmapFunctions::bitmap_count(FunctionContext*
ctx, const StringVal& s
// zero size means the src input is a agg object
if (src.len == 0) {
auto bitmap = reinterpret_cast(src.ptr);
-return {bitmap->cardinality()};
+return static_cast(bitmap->cardinality());
Review comment:
Is this more optimal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
