[GitHub] [incubator-doris] ZhikaiZuo opened a new pull request #6936: Fix some return logic error in init BE encoding_map
ZhikaiZuo opened a new pull request #6936:
URL: https://github.com/apache/incubator-doris/pull/6936
## Proposed changes
Checking _encoding_map in the original code to return in advance will cause
some EncodingInfoResolver constructors to write encoding methods that cannot be
pushed to default_encoding_type_map_ or value_seek_encoding_map_.
E.g:
EncodingInfoResolver::EncodingInfoResolver() {
_add_map();
_add_map();
...
}
The second line code is invilid.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [ ] Optimization. Including functional usability improvements and
performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature
there in detail
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at
d...@doris.apache.org by explaining why you chose the solution you did and what
alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] GoGoWen commented on a change in pull request #6932: fix schema change fail as memory allocation on row block sorting
GoGoWen commented on a change in pull request #6932:
URL: https://github.com/apache/incubator-doris/pull/6932#discussion_r736212628
##
File path: be/src/olap/schema_change.cpp
##
@@ -1184,6 +1196,16 @@ OLAPStatus
SchemaChangeWithSorting::process(RowsetReaderSharedPtr rowset_reader,
true)) {
LOG(WARNING) << "failed to allocate RowBlock.";
return OLAP_ERR_INPUT_PARAMETER_ERROR;
+}else {
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] GoGoWen commented on a change in pull request #6932: fix schema change fail as memory allocation on row block sorting
GoGoWen commented on a change in pull request #6932:
URL: https://github.com/apache/incubator-doris/pull/6932#discussion_r736213504
##
File path: be/src/olap/schema_change.cpp
##
@@ -800,6 +803,15 @@ void RowBlockAllocator::release(RowBlock* row_block) {
delete row_block;
}
+bool RowBlockAllocator::is_memory_enough_for_sorting(size_t num_rows, size_t
allocated_rows){
+if (num_rows <= allocated_rows) {
+return true;
+}
+size_t row_block_size = _row_len * (num_rows - allocated_rows);
+return _memory_allocated + row_block_size < _memory_limitation;
Review comment:
yes, u'r right, should be _mem_tracker->consumption(). fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wunan1210 opened a new pull request #6937: support use char like \x01 in flink-doris-sink column & line delimiter
wunan1210 opened a new pull request #6937:
URL: https://github.com/apache/incubator-doris/pull/6937
## Proposed changes
add a option to support use char like \x01 in flink-doris-sink column & line
delimiter
use like
```
tEnv.executeSql(
"CREATE TABLE doris_test_sink (" +
"name STRING," +
"age INT" +
") " +
"WITH (\n" +
" 'connector' = 'doris',\n" +
" 'fenodes' = 'FE_IP:8030',\n" +
" 'table.identifier' = 'db.table',\n" +
" 'username' = 'root',\n" +
" 'password' = '',\n" +
" 'sink.properties.format' = 'csv',\n" +
" 'sink.properties.escape_delimiters' = 'true', \n" +
" 'sink.properties.column_separator' = '\\x01', \n" +
" 'sink.properties.line_delimiter' = '\\x02'\n" +
")");
```
but i am not sure if it is needed ...
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
- [ ] Code refactor (Modify the code structure, format the code, etc...)
- [x] Optimization. Including functional usability improvements and
performance improvements.
- [ ] Dependency. Such as changes related to third-party components.
- [ ] Other.
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature
there in detail
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [ ] If these changes need document changes, I have updated the document
- [ ] Any dependent changes have been merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] DinoZhang opened a new pull request #6938: fix column length exception
DinoZhang opened a new pull request #6938: URL: https://github.com/apache/incubator-doris/pull/6938 ## Proposed changes fix #5007 ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng merged pull request #6922: [Spark-Doris-Connector] fixed some spark-doris-connector doc typo
wuyunfeng merged pull request #6922: URL: https://github.com/apache/incubator-doris/pull/6922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Spark-Doris-Connector] fixed some spark-doris-connector doc typo
This is an automated email from the ASF dual-hosted git repository.
wyf pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9d4e6d8 [Spark-Doris-Connector] fixed some spark-doris-connector doc
typo
9d4e6d8 is described below
commit 9d4e6d8362f1c65190076865982015e00adb7030
Author: luzhijing <82810928+luzhij...@users.noreply.github.com>
AuthorDate: Tue Oct 26 18:23:53 2021 +0800
[Spark-Doris-Connector] fixed some spark-doris-connector doc typo
---
docs/zh-CN/extending-doris/spark-doris-connector.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/zh-CN/extending-doris/spark-doris-connector.md
b/docs/zh-CN/extending-doris/spark-doris-connector.md
index 92d13d3..3ede5ac 100644
--- a/docs/zh-CN/extending-doris/spark-doris-connector.md
+++ b/docs/zh-CN/extending-doris/spark-doris-connector.md
@@ -52,7 +52,7 @@ Spark Doris Connector 可以支持通过 Spark 读取 Doris 中存储的数据
```bash
sh build.sh 3 ## spark 3.x版本，默认是3.1.2
-sh build.sh 2 ## soark 2.x版本，默认是2.3.4
+sh build.sh 2 ## spark 2.x版本，默认是2.3.4
```
编译成功后，会在 `output/` 目录下生成文件 `doris-spark-1.0.0-SNAPSHOT.jar`。将此文件复制到 `Spark` 的
`ClassPath` 中即可使用 `Spark-Doris-Connector`。例如，`Local` 模式运行的 `Spark`，将此文件放入
`jars/` 文件夹下。`Yarn`集群模式运行的`Spark`，则将此文件放入预部署包中。
@@ -221,4 +221,4 @@ kafkaSource.selectExpr("CAST(key AS STRING)", "CAST(value
as STRING)")
| TIME | DataTypes.DoubleType |
| HLL| Unsupported datatype |
-*
注：Connector中，将`DATE`和`DATETIME`映射为`String`。由于`Doris`底层存储引擎处理逻辑，直接使用时间类型时，覆盖的时间范围无法满足需求。所以使用
`String` 类型直接返回对应的时间可读文本。
\ No newline at end of file
+*
注：Connector中，将`DATE`和`DATETIME`映射为`String`。由于`Doris`底层存储引擎处理逻辑，直接使用时间类型时，覆盖的时间范围无法满足需求。所以使用
`String` 类型直接返回对应的时间可读文本。
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #6939: Fix String type column using zonemap to filter data maybe core dump
yangzhg opened a new pull request #6939: URL: https://github.com/apache/incubator-doris/pull/6939 ## Proposed changes Fix String type column using zonemap to filter data maybe core dump, because of not allocating memory before parsing string type zonemap ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6925: [Benchmark] Add star schema benchmark tools
github-actions[bot] commented on pull request #6925: URL: https://github.com/apache/incubator-doris/pull/6925#issuecomment-951852790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6939: Fix String type column using zonemap to filter data maybe core dump
github-actions[bot] commented on pull request #6939: URL: https://github.com/apache/incubator-doris/pull/6939#issuecomment-951872398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] nimuyuhan opened a new pull request #6942: [Function] add BE bitmap function bitmap_max
nimuyuhan opened a new pull request #6942: URL: https://github.com/apache/incubator-doris/pull/6942 ## Proposed changes Support bitmap_max. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix https://github.com/apache/incubator-doris/issues/6875) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #6939: Fix String type column using zonemap to filter data maybe core dump
yangzhg merged pull request #6939: URL: https://github.com/apache/incubator-doris/pull/6939 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Fix String type column using zonemap to filter data maybe core dump (#6939)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4f9b46d Fix String type column using zonemap to filter data maybe
core dump (#6939)
4f9b46d is described below
commit 4f9b46d40383c2526bfd1fd604cb95063ae65c7c
Author: Zhengguo Yang
AuthorDate: Wed Oct 27 09:25:38 2021 +0800
Fix String type column using zonemap to filter data maybe core dump (#6939)
Fix String type column using zonemap to filter data maybe core dump,
because of not allocating memory before parsing string type zonemap
---
be/src/olap/wrapper_field.cpp | 11 +--
be/src/olap/wrapper_field.h | 4
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/wrapper_field.cpp b/be/src/olap/wrapper_field.cpp
index 61f5f88..be908a2 100644
--- a/be/src/olap/wrapper_field.cpp
+++ b/be/src/olap/wrapper_field.cpp
@@ -17,6 +17,8 @@
#include "olap/wrapper_field.h"
+#include "olap/row_cursor.h"
+
namespace doris {
const size_t DEFAULT_STRING_LENGTH = 50;
@@ -25,8 +27,9 @@ WrapperField* WrapperField::create(const TabletColumn&
column, uint32_t len) {
bool is_string_type =
(column.type() == OLAP_FIELD_TYPE_CHAR || column.type() ==
OLAP_FIELD_TYPE_VARCHAR ||
column.type() == OLAP_FIELD_TYPE_HLL || column.type() ==
OLAP_FIELD_TYPE_OBJECT ||
-column.type() == OLAP_FIELD_TYPE_STRING);
-size_t max_length = column.type() == OLAP_FIELD_TYPE_STRING ?
OLAP_STRING_MAX_LENGTH : OLAP_VARCHAR_MAX_LENGTH;
+ column.type() == OLAP_FIELD_TYPE_STRING);
+size_t max_length = column.type() == OLAP_FIELD_TYPE_STRING ?
OLAP_STRING_MAX_LENGTH
+:
OLAP_VARCHAR_MAX_LENGTH;
if (is_string_type && len > max_length) {
OLAP_LOG_WARNING("length of string parameter is too long[len=%lu,
max_len=%lu].", len,
max_length);
@@ -89,6 +92,10 @@ WrapperField::WrapperField(Field* rep, size_t variable_len,
bool is_string_type)
_string_content.reset(new char[slice->size]);
slice->data = _string_content.get();
}
+if (_rep->type() == OLAP_FIELD_TYPE_STRING) {
+_long_text_buf = (char*)malloc(RowCursor::DEFAULT_TEXT_LENGTH *
sizeof(char));
+rep->set_long_text_buf(&_long_text_buf);
+}
}
WrapperField::WrapperField(Field* rep, const RowCursorCell& row_cursor_cell)
diff --git a/be/src/olap/wrapper_field.h b/be/src/olap/wrapper_field.h
index c14c019..3d22a7b 100644
--- a/be/src/olap/wrapper_field.h
+++ b/be/src/olap/wrapper_field.h
@@ -41,6 +41,9 @@ public:
virtual ~WrapperField() {
delete _rep;
delete[] _owned_buf;
+if (_long_text_buf) {
+delete _long_text_buf;
+}
}
// 将内部的value转成string输出
@@ -103,6 +106,7 @@ private:
bool _is_string_type;
char* _field_buf = nullptr;
char* _owned_buf = nullptr;
+char* _long_text_buf = nullptr;
//include fixed and variable length and null bytes
size_t _length;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Cache][Bug] Correct update cache timeout unit (#6888)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 29a4ff4 [Cache][Bug] Correct update cache timeout unit (#6888)
29a4ff4 is described below
commit 29a4ff4bbefd2359b0345275cf58d96c0708747a
Author: Lijia Liu
AuthorDate: Wed Oct 27 09:53:58 2021 +0800
[Cache][Bug] Correct update cache timeout unit (#6888)
Now FE update cache use MICROSECONDS as TimeUnit.
Replace it by MILLISECONDS.
---
fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheBeProxy.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheBeProxy.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheBeProxy.java
index d7e5eda..cb30cbb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheBeProxy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheBeProxy.java
@@ -53,7 +53,7 @@ public class CacheBeProxy extends CacheProxy {
try {
Future future =
BackendServiceProxy.getInstance()
.updateCache(address, request);
-InternalService.PCacheResponse response = future.get(timeoutMs,
TimeUnit.MICROSECONDS);
+InternalService.PCacheResponse response = future.get(timeoutMs,
TimeUnit.MILLISECONDS);
if (response.getStatus() == InternalService.PCacheStatus.CACHE_OK)
{
status.setStatus(new Status(TStatusCode.OK, "CACHE_OK"));
} else {
@@ -125,7 +125,7 @@ public class CacheBeProxy extends CacheProxy {
request =
request.toBuilder().setClearType(InternalService.PClearType.CLEAR_ALL).build();
LOG.info("clear all backend cache, backendId {}", backend.getId());
Future future =
BackendServiceProxy.getInstance().clearCache(address, request);
-InternalService.PCacheResponse response = future.get(timeoutMs,
TimeUnit.MICROSECONDS);
+InternalService.PCacheResponse response = future.get(timeoutMs,
TimeUnit.MILLISECONDS);
if (response.getStatus() == InternalService.PCacheStatus.CACHE_OK)
{
status.setStatus(new Status(TStatusCode.OK, "CACHE_OK"));
return true;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6888: correct update cache timeout unit
morningman merged pull request #6888: URL: https://github.com/apache/incubator-doris/pull/6888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Bug] Fix treat tuple_is_null_predicate is const expr cause core problem (#6919)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 77a954d [Bug] Fix treat tuple_is_null_predicate is const expr cause
core problem (#6919)
77a954d is described below
commit 77a954d02c84918eda657f77a80d1cc231692482
Author: HappenLee
AuthorDate: Wed Oct 27 09:54:25 2021 +0800
[Bug] Fix treat tuple_is_null_predicate is const expr cause core problem
(#6919)
Fix treat tuple_is_null_predicate is const expr cause core problem
---
be/src/exprs/tuple_is_null_predicate.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/be/src/exprs/tuple_is_null_predicate.h
b/be/src/exprs/tuple_is_null_predicate.h
index 2125887..58069d5 100644
--- a/be/src/exprs/tuple_is_null_predicate.h
+++ b/be/src/exprs/tuple_is_null_predicate.h
@@ -31,6 +31,9 @@ public:
return pool->add(new TupleIsNullPredicate(*this));
}
+bool is_constant() const override {
+return false;
+}
protected:
friend class Expr;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6913: 使用双引号将关键字from和to做两表关联时，条件判断为空的时候，会导致be节点挂掉
morningman closed issue #6913: URL: https://github.com/apache/incubator-doris/issues/6913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6919: [Bug] Fix treat tuple_is_null_predicate is const expr cause core problem
morningman merged pull request #6919: URL: https://github.com/apache/incubator-doris/pull/6919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6923: [Bug] fix Runtime filter can't find fragment-id when apply_filter called early
morningman merged pull request #6923: URL: https://github.com/apache/incubator-doris/pull/6923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6921: [Bug] Runtime filter can't find fragment-id when apply_filter called early
morningman closed issue #6921: URL: https://github.com/apache/incubator-doris/issues/6921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Bug] fix Runtime filter can't find fragment-id when apply_filter called early (#6923)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new d4249e4 [Bug] fix Runtime filter can't find fragment-id when
apply_filter called early (#6923)
d4249e4 is described below
commit d4249e4f213ca6db472d772a7b2c5146be40
Author: Pxl <952130...@qq.com>
AuthorDate: Wed Oct 27 09:54:52 2021 +0800
[Bug] fix Runtime filter can't find fragment-id when apply_filter called
early (#6923)
#6921
---
be/src/runtime/fragment_mgr.cpp | 14 +++---
be/src/runtime/fragment_mgr.h | 2 ++
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index f980e7b..a8ff209 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -610,6 +610,7 @@ Status FragmentMgr::exec_plan_fragment(const
TExecPlanFragmentParams& params, Fi
{
std::lock_guard lock(_lock);
_fragment_map.insert(std::make_pair(params.params.fragment_instance_id,
exec_state));
+_cv.notify_all();
}
auto st = _thread_pool->submit_func(
@@ -747,7 +748,7 @@ Status FragmentMgr::exec_external_plan_fragment(const
TScanOpenParams& params,
}
VLOG_QUERY << "BackendService execute open() TQueryPlanInfo: "
-<< apache::thrift::ThriftDebugString(t_query_plan_info);
+ << apache::thrift::ThriftDebugString(t_query_plan_info);
// assign the param used to execute PlanFragment
TExecPlanFragmentParams exec_fragment_params;
exec_fragment_params.protocol_version =
(PaloInternalServiceVersion::type)0;
@@ -811,14 +812,21 @@ Status FragmentMgr::apply_filter(const
PPublishFilterRequest* request, const cha
std::shared_ptr fragment_state;
{
-std::lock_guard lock(_lock);
+std::unique_lock lock(_lock);
+if (!_fragment_map.count(tfragment_instance_id)) {
+VLOG_NOTICE << "wait for fragment start execute, fragment-id:" <<
fragment_instance_id;
+_cv.wait_for(lock, std::chrono::milliseconds(1000),
+ [&] { return
_fragment_map.count(tfragment_instance_id); });
+}
+
auto iter = _fragment_map.find(tfragment_instance_id);
if (iter == _fragment_map.end()) {
-LOG(WARNING) << "unknown fragment-id:" << fragment_instance_id;
+VLOG_CRITICAL << "unknown fragment-id:" <<
fragment_instance_id;
return Status::InvalidArgument("fragment-id");
}
fragment_state = iter->second;
}
+
DCHECK(fragment_state != nullptr);
RuntimeFilterMgr* runtime_filter_mgr =
fragment_state->executor()->runtime_state()->runtime_filter_mgr();
diff --git a/be/src/runtime/fragment_mgr.h b/be/src/runtime/fragment_mgr.h
index 70233e1..ba56216 100644
--- a/be/src/runtime/fragment_mgr.h
+++ b/be/src/runtime/fragment_mgr.h
@@ -99,6 +99,8 @@ private:
std::mutex _lock;
+std::condition_variable _cv;
+
// Make sure that remove this before no data reference FragmentExecState
std::unordered_map>
_fragment_map;
// query id -> QueryFragmentsCtx
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Benchmark] Add star schema benchmark tools (#6925)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 00fe9de [Benchmark] Add star schema benchmark tools (#6925)
00fe9de is described below
commit 00fe9deaebe70c84a1207273388d58f99cb17cd0
Author: Mingyu Chen
AuthorDate: Wed Oct 27 09:55:36 2021 +0800
[Benchmark] Add star schema benchmark tools (#6925)
This CL mainly changes:
1. Add star schema benchmark tools in `tools/ssb-tools`, for user to easy
load and test with SSB data set.
2. Disable the segment cache for some read scenario such as compaction and
alter operation.(Fix #6924 )
3. Fix a bug that `max_segment_num_per_rowset` won't work(Fix #6926)
4. Enable `enable_batch_delete_by_default` by default.
---
be/src/agent/cgroups_mgr.cpp | 4 +-
be/src/common/config.h | 2 +-
be/src/exec/tablet_sink.cpp| 32 ++--
be/src/exec/tablet_sink.h | 12 +-
be/src/olap/delta_writer.cpp | 3 +
be/src/olap/rowset/beta_rowset_reader.cpp | 5 +-
be/src/olap/segment_loader.cpp | 24 +--
be/src/olap/segment_loader.h | 27 +++-
docs/.vuepress/sidebar/en.js | 3 +-
docs/.vuepress/sidebar/zh-CN.js| 3 +-
docs/en/administrator-guide/config/be_config.md| 2 +-
docs/en/best-practices/star-schema-benchmark.md| 163
docs/zh-CN/administrator-guide/config/be_config.md | 2 +-
docs/zh-CN/best-practices/star-schema-benchmark.md | 164 +
.../main/java/org/apache/doris/common/Config.java | 2 +-
tools/ssb-tools/build-ssb-dbgen.sh | 53 +++
tools/ssb-tools/create-tables.sql | 117 +++
tools/ssb-tools/doris-cluster.conf | 27
tools/ssb-tools/gen-ssb-data.sh| 112 ++
tools/ssb-tools/load-dimension-data.sh | 68 +
tools/ssb-tools/load-fact-data.sh | 126
tools/ssb-tools/queries/q1.1.sql | 24 +++
tools/ssb-tools/queries/q1.2.sql | 24 +++
tools/ssb-tools/queries/q1.3.sql | 25
tools/ssb-tools/queries/q2.1.sql | 26
tools/ssb-tools/queries/q2.2.sql | 27
tools/ssb-tools/queries/q2.3.sql | 26
tools/ssb-tools/queries/q3.1.sql | 28
tools/ssb-tools/queries/q3.2.sql | 28
tools/ssb-tools/queries/q3.3.sql | 30
tools/ssb-tools/queries/q3.4.sql | 30
tools/ssb-tools/queries/q4.1.sql | 30
tools/ssb-tools/queries/q4.2.sql | 31
tools/ssb-tools/queries/q4.3.sql | 29
34 files changed, 1259 insertions(+), 50 deletions(-)
diff --git a/be/src/agent/cgroups_mgr.cpp b/be/src/agent/cgroups_mgr.cpp
index edb274f..ffcbe21 100644
--- a/be/src/agent/cgroups_mgr.cpp
+++ b/be/src/agent/cgroups_mgr.cpp
@@ -325,8 +325,8 @@ AgentStatus CgroupsMgr::init_cgroups() {
_is_cgroups_init_success = true;
return AgentStatus::DORIS_SUCCESS;
} else {
-LOG(WARNING) << "Could not find a valid cgroups path for resource
isolation,"
- << "current value is " << _root_cgroups_path << ". ignore
it.";
+VLOG_NOTICE << "Could not find a valid cgroups path for resource
isolation,"
+<< "current value is " << _root_cgroups_path << ". ignore
it.";
_is_cgroups_init_success = false;
return AgentStatus::DORIS_ERROR;
}
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 37e71d9..42c43fb 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -629,7 +629,7 @@ CONF_Int32(send_batch_thread_pool_queue_size, "102400");
// so if there are too many segment in a rowset, the compaction process
// will run out of memory.
// When doing compaction, each segment may take at least 1MB buffer.
-CONF_mInt32(max_segment_num_per_rowset, "100");
+CONF_mInt32(max_segment_num_per_rowset, "200");
// The connection timeout when connecting to external table such as odbc table.
CONF_mInt32(external_table_connect_timeout_sec, "5");
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index d282058..c5c02d2 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -163,10 +163,7 @@ Status NodeChannel::open_wait() {
// add batch closure
_add_batch_closure =
ReusableClosure::create();
_add_batch_closure->addFailedHandler([this]() {
-std::stringstream ss;
-ss << name() << " add batch re
[GitHub] [incubator-doris] morningman closed issue #6926: [Bug] BE config max_segment_num_per_rowset doesn' t work
morningman closed issue #6926: URL: https://github.com/apache/incubator-doris/issues/6926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6925: [Benchmark] Add star schema benchmark tools
morningman merged pull request #6925: URL: https://github.com/apache/incubator-doris/pull/6925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6924: [Feature] [Optimize] Disable segment cache for non-query reader
morningman closed issue #6924: URL: https://github.com/apache/incubator-doris/issues/6924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Enhance] Add BackendHbResponse info (#6929)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new a4a7e64 [Enhance] Add BackendHbResponse info (#6929)
a4a7e64 is described below
commit a4a7e642b456df735dd108735f72a5b406b9b822
Author: dh-cloud <60729713+dh-cl...@users.noreply.github.com>
AuthorDate: Wed Oct 27 09:56:07 2021 +0800
[Enhance] Add BackendHbResponse info (#6929)
when be has excepiton, fe doesn't log the BackendHbResponse info, so we
can't know which be has exception
the exception log is：
`WARN (heartbeat mgr|31) [HeartbeatMgr.runAfterCatalogReady():141] get bad
heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException:
Connection refused (Connection refused)
`
so need add toString(), then fe can log the BackendHbResponse info
---
.../main/java/org/apache/doris/system/BackendHbResponse.java | 11 +++
1 file changed, 11 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
b/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
index 9db8790..1818c01 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/BackendHbResponse.java
@@ -104,5 +104,16 @@ public class BackendHbResponse extends HeartbeatResponse
implements Writable {
httpPort = in.readInt();
brpcPort = in.readInt();
}
+
+@Override
+public String toString() {
+StringBuilder sb = new StringBuilder();
+sb.append(super.toString());
+sb.append(", beId: ").append(beId);
+sb.append(", bePort: ").append(bePort);
+sb.append(", httpPort: ").append(httpPort);
+sb.append(", brpcPort: ").append(brpcPort);
+return sb.toString();
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6929: add BackendHbResponse info
morningman merged pull request #6929: URL: https://github.com/apache/incubator-doris/pull/6929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 opened a new pull request #6943: Docker 1.4.1 Compile Environment, First Compile Description
hf200012 opened a new pull request #6943: URL: https://github.com/apache/incubator-doris/pull/6943 Docker 1.4.1 Compile Environment, First Compile Description ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
