[GitHub] [incubator-doris] zhangstar333 closed pull request #8284: [fix] delete the repeat same code to compile
zhangstar333 closed pull request #8284: URL: https://github.com/apache/incubator-doris/pull/8284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 opened a new pull request #8285: [Doc]Add sync job fe configuration item description
hf200012 opened a new pull request #8285: URL: https://github.com/apache/incubator-doris/pull/8285 Add sync job fe configuration item description # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] mrhhsg opened a new issue #8286: [Bug] Compile error: duplicated destructor function in MinMaxFuncBase
mrhhsg opened a new issue #8286: URL: https://github.com/apache/incubator-doris/issues/8286 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` incubator-doris/be/src/exprs/minmax_predicate.h:38:13: error: 'virtual doris::MinMaxFuncBase::~MinMaxFuncBase()' cannot be overloaded with 'virtual doris::MinMaxFuncBase::~MinMaxFuncBase()' ``` ### What You Expected? compile success ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] mrhhsg opened a new pull request #8287: [Fix]Remove duplicated destructor function in MinMaxFuncBase
mrhhsg opened a new pull request #8287: URL: https://github.com/apache/incubator-doris/pull/8287 # Proposed changes Issue Number: close #8286 ## Problem Summary: Fix the compile error. ## Checklist(Required) 1. Does it affect the original behavior: No 2. Has unit tests been added: No Need 3. Has document been added or modified: No Need 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on a change in pull request #8285: [Doc]Add sync job fe configuration item description
jackwener commented on a change in pull request #8285: URL: https://github.com/apache/incubator-doris/pull/8285#discussion_r816587522 ## File path: docs/zh-CN/administrator-guide/config/fe_config.md ## @@ -2166,3 +2166,53 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清 是否为 Master FE 节点独有的配置项：true 数据大小阈值，用来判断副本的数据量是否太大 + +### enable_create_sync_job + +开启Mysql数据同步作业功能。默认是false，关闭此功能 + +默认值：false + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### sync_commit_interval_second + +提交事务的最大时间间隔。若超过了这个时间channel中还有数据没有提交，consumer会通知channel提交事务。 + +默认值：10（秒） + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### min_sync_commit_size + +提交事务需满足的最小event数量。若Fe接收到的event数量小于它，会继续等待下一批数据直到时间超过了`sync_commit_interval_second `为止。默认值是1个events，如果你想修改此配置，请确保此值小于canal端的`canal.instance.memory.buffer.size`配置（默认16384），否则在ack前Fe会尝试获取比store队列长度更多的event，导致store队列阻塞至超时为止。 Review comment: ```suggestion 提交事务需满足的最小 event 数量。若 Fe 接收到的 event 数量小于它，会继续等待下一批数据直到时间超过了 `sync_commit_interval_second ` 为止。默认值是 1 个 events，如果你想修改此配置，请确保此值小于 canal 端的 `canal.instance.memory.buffer.size` 配置（默认16384），否则在 ack 前Fe会尝试获取比 store 队列长度更多的 event，导致 store 队列阻塞至超时为止。 ``` ## File path: docs/zh-CN/administrator-guide/config/fe_config.md ## @@ -2166,3 +2166,53 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清 是否为 Master FE 节点独有的配置项：true 数据大小阈值，用来判断副本的数据量是否太大 + +### enable_create_sync_job + +开启Mysql数据同步作业功能。默认是false，关闭此功能 Review comment: ```suggestion 开启 MySQL 数据同步作业功能。默认是 false，关闭此功能 ``` ## File path: docs/zh-CN/administrator-guide/config/fe_config.md ## @@ -2166,3 +2166,53 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清 是否为 Master FE 节点独有的配置项：true 数据大小阈值，用来判断副本的数据量是否太大 + +### enable_create_sync_job + +开启Mysql数据同步作业功能。默认是false，关闭此功能 + +默认值：false + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### sync_commit_interval_second + +提交事务的最大时间间隔。若超过了这个时间channel中还有数据没有提交，consumer会通知channel提交事务。 + +默认值：10（秒） + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### min_sync_commit_size + +提交事务需满足的最小event数量。若Fe接收到的event数量小于它，会继续等待下一批数据直到时间超过了`sync_commit_interval_second `为止。默认值是1个events，如果你想修改此配置，请确保此值小于canal端的`canal.instance.memory.buffer.size`配置（默认16384），否则在ack前Fe会尝试获取比store队列长度更多的event，导致store队列阻塞至超时为止。 + +默认值：1 + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### min_bytes_sync_commit + +提交事务需满足的最小数据大小。若Fe接收到的数据大小小于它，会继续等待下一批数据直到时间超过了`sync_commit_interval_second`为止。默认值是15MB，如果你想修改此配置，请确保此值小于canal端的`canal.instance.memory.buffer.size`和`canal.instance.memory.buffer.memunit`的乘积（默认16MB），否则在ack前Fe会尝试获取比store空间更大的数据，导致store队列阻塞至超时为止。 + +默认值：15 * 1024 * 1024（15M） + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### max_bytes_sync_commit + + 数据同步作业线程池中的最大线程数量。此线程池整个FE中只有一个，用于处理FE中所有数据同步作业向BE发送数据的任务task，线程池的实现在`SyncTaskPool`类。 Review comment: ```suggestion 数据同步作业线程池中的最大线程数量。此线程池整个FE中只有一个，用于处理FE中所有数据同步作业向BE发送数据的任务 task，线程池的实现在 `SyncTaskPool` 类。 ``` ## File path: docs/zh-CN/administrator-guide/config/fe_config.md ## @@ -2166,3 +2166,53 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清 是否为 Master FE 节点独有的配置项：true 数据大小阈值，用来判断副本的数据量是否太大 + +### enable_create_sync_job + +开启Mysql数据同步作业功能。默认是false，关闭此功能 + +默认值：false + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### sync_commit_interval_second + +提交事务的最大时间间隔。若超过了这个时间channel中还有数据没有提交，consumer会通知channel提交事务。 Review comment: ```suggestion 提交事务的最大时间间隔。若超过了这个时间 channel 中还有数据没有提交，consumer 会通知 channel 提交事务。 ``` ## File path: docs/zh-CN/administrator-guide/config/fe_config.md ## @@ -2166,3 +2166,53 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清 是否为 Master FE 节点独有的配置项：true 数据大小阈值，用来判断副本的数据量是否太大 + +### enable_create_sync_job + +开启Mysql数据同步作业功能。默认是false，关闭此功能 + +默认值：false + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### sync_commit_interval_second + +提交事务的最大时间间隔。若超过了这个时间channel中还有数据没有提交，consumer会通知channel提交事务。 + +默认值：10（秒） + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### min_sync_commit_size + +提交事务需满足的最小event数量。若Fe接收到的event数量小于它，会继续等待下一批数据直到时间超过了`sync_commit_interval_second `为止。默认值是1个events，如果你想修改此配置，请确保此值小于canal端的`canal.instance.memory.buffer.size`配置（默认16384），否则在ack前Fe会尝试获取比store队列长度更多的event，导致store队列阻塞至超时为止。 + +默认值：1 + +是否可以动态配置：true + +是否为 Master FE 节点独有的配置项：true + +### min_bytes_sync_commit + +提交事务需满足的最小数据大小。若Fe接收到的数据大小小于它，会继续等待下一批数据直到时间超过了`sync_commit_interval_second`为止。默认值是15MB，如果你想修改此配置，请确保此值小于canal端的`canal.instance.memory.buffer.size`和`canal.instance.memory.buffer.memunit`的乘积（默认16MB），否则在ack前Fe会尝试获取比store空间更大的数据，导致store队列阻塞至超时为止。 Review comment: ```suggestion 提交事务需满足的最小数据大小。若 Fe 接收到的数据大小小于它，会继续等待下一批数据直到时间超过了 `sync_commit_interval_second` 为止。默认值是 15 MB，如果你想修改此配置，请确保此值小于 canal 端的 `canal.instance.memory.buffer.size` 和 `canal.instance.memory.buffer.memunit` 的乘积（默认 16 MB），否则在 ack 前 Fe 会尝试获取比 store 空间更大的数据，导致 store 队列阻塞至超时为止。 ``` -- This is an automated message from the Apache Git S
[GitHub] [incubator-doris] jackwener commented on a change in pull request #8277: [fix] fix a bug of encryption function with iv may return wrong result
jackwener commented on a change in pull request #8277:
URL: https://github.com/apache/incubator-doris/pull/8277#discussion_r816606898
##
File path: be/src/util/encryption_util.cpp
##
@@ -219,21 +230,30 @@ static int do_decrypt(EVP_CIPHER_CTX* cipher_ctx, const
EVP_CIPHER* cipher,
int EncryptionUtil::decrypt(EncryptionMode mode, const unsigned char* encrypt,
uint32_t encrypt_length, const unsigned char* key,
uint32_t key_length,
-const unsigned char* iv, bool padding, unsigned
char* decrypt_content) {
+const char* iv_str, bool padding, unsigned char*
decrypt_content) {
const EVP_CIPHER* cipher = get_evp_type(mode);
/* The encrypt key to be used for decryption */
unsigned char encrypt_key[ENCRYPTION_MAX_KEY_LENGTH / 8];
create_key(key, key_length, encrypt_key, mode);
-if (cipher == nullptr || (EVP_CIPHER_iv_length(cipher) > 0 && !iv)) {
+int iv_length = EVP_CIPHER_iv_length(cipher);
+if (cipher == nullptr || (iv_length > 0 && !iv_str)) {
return AES_BAD_DATA;
}
+char* init_vec = nullptr;
+std::string iv_default("DORISDORISDORIS_");
+
+if (iv_str) {
+init_vec = &iv_default[0];
+memcpy(init_vec, iv_str, strnlen(iv_str, EVP_MAX_IV_LENGTH));
+init_vec[iv_length] = '\0';
+}
EVP_CIPHER_CTX* cipher_ctx = EVP_CIPHER_CTX_new();
EVP_CIPHER_CTX_reset(cipher_ctx);
int length = 0;
-int ret = do_decrypt(cipher_ctx, cipher, encrypt, encrypt_length,
encrypt_key, iv, padding,
- decrypt_content, &length);
+int ret = do_decrypt(cipher_ctx, cipher, encrypt, encrypt_length,
encrypt_key,
+ (unsigned char*)init_vec, padding, decrypt_content,
&length);
Review comment:
reinterpret_cast may be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on pull request #8273: [Refactor] Remove agent status
jackwener commented on pull request #8273: URL: https://github.com/apache/incubator-doris/pull/8273#issuecomment-1055237191 UT didn't pass. I can't ensure the correctness, It needs to build local and run all unit tests; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei opened a new pull request #8288: [Refactor] remove pusher.cpp and related mock test code
yiguolei opened a new pull request #8288: URL: https://github.com/apache/incubator-doris/pull/8288 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #8287: [Fix]Remove duplicated destructor function in MinMaxFuncBase
yangzhg commented on pull request #8287: URL: https://github.com/apache/incubator-doris/pull/8287#issuecomment-1055285050 because of this pr will fix the compile error, I will merge it now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed issue #8286: [Bug] Compile error: duplicated destructor function in MinMaxFuncBase
yangzhg closed issue #8286: URL: https://github.com/apache/incubator-doris/issues/8286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (2b9b0fc -> 940efc6)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 2b9b0fc [Fix] Function percentile input null return null (#8238) add 940efc6 [Fix]Remove duplicated destructor function in MinMaxFuncBase (#8287) No new revisions were added by this update. Summary of changes: be/src/exprs/minmax_predicate.h | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #8287: [Fix]Remove duplicated destructor function in MinMaxFuncBase
yangzhg merged pull request #8287: URL: https://github.com/apache/incubator-doris/pull/8287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #8289: [refactor] remove types_test
yangzhg opened a new pull request #8289: URL: https://github.com/apache/incubator-doris/pull/8289 # Proposed changes 1. remove types_test, it will cause core dump in higher version gcc or clang, becuse of memory align, some code will be vectorize in higher gcc or clang 2. change string type length to 2GB instead of -1 3. modify unaccessable code ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new issue #8290: [Bug] [Vectorized] Exchange merge null first order is wrong
HappenLee opened a new issue #8290: URL: https://github.com/apache/incubator-doris/issues/8290 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version vectorized ### What's Wrong? null first order is wrong in exchange node merge ### What You Expected? right order with null first ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new pull request #8291: [Bug][Vectorized] Fix exchange node merge sort null first order wrong
HappenLee opened a new pull request #8291: URL: https://github.com/apache/incubator-doris/pull/8291 # Proposed changes Issue Number: close #8290 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (Yes) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8053: [docs] fix document date-time-functions typo
github-actions[bot] commented on pull request #8053: URL: https://github.com/apache/incubator-doris/pull/8053#issuecomment-1055340451 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8090: [refactor](optimize)refactor RowRef/RowRefList to decrease hash table memory size
morningman commented on pull request #8090: URL: https://github.com/apache/incubator-doris/pull/8090#issuecomment-1055340501 hi @zuochunwei , sorry for the later reply. Please rebase the code to solve the conflict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8235: [fix](planner) Convert format in RewriteFromUnixTimeRule
morningman commented on a change in pull request #8235:
URL: https://github.com/apache/incubator-doris/pull/8235#discussion_r816698786
##
File path: fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
##
@@ -1673,30 +1673,21 @@ public void testFromUnixTimeRewrite() throws Exception {
String sql = "select * from test1 where from_unixtime(query_time) >
'2021-03-02 10:01:28'";
String explainString =
UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, "EXPLAIN " + sql);
Assert.assertTrue(explainString.contains("PREDICATES: `query_time` <=
253402271999, `query_time` > 1614650488"));
+
//format -MM-dd HH:mm:ss
sql = "select * from test1 where from_unixtime(query_time, '-MM-dd
HH:mm:ss') > '2021-03-02 10:01:28'";
explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext,
"EXPLAIN " + sql);
Assert.assertTrue(explainString.contains("PREDICATES: `query_time` <=
253402271999, `query_time` > 1614650488"));
-//format -MM-dd HH:mm
Review comment:
Why remove these test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8291: [Bug][Vectorized] Fix exchange node merge sort null first order wrong
github-actions[bot] commented on pull request #8291: URL: https://github.com/apache/incubator-doris/pull/8291#issuecomment-1055349567 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
wangbo commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r816704629
##
File path: be/src/olap/row_block.cpp
##
@@ -89,8 +89,7 @@ void RowBlock::_compute_layout() {
_field_offset_in_memory.push_back(memory_size);
// All field has a nullbyte in memory
-if (column.type() == OLAP_FIELD_TYPE_VARCHAR || column.type() ==
OLAP_FIELD_TYPE_HLL ||
-column.type() == OLAP_FIELD_TYPE_CHAR || column.type() ==
OLAP_FIELD_TYPE_OBJECT ||column.type() == OLAP_FIELD_TYPE_STRING) {
+if (column.is_slice_stored()) {
Review comment:
Maybe ```is_variable_type``` be better than ```is_slice_stored``` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
wangbo commented on a change in pull request #8234: URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r816706217 ## File path: be/src/olap/tablet_schema.cpp ## @@ -246,6 +256,7 @@ uint32_t TabletColumn::get_field_length_by_type(TPrimitiveType::type type, uint3 return 4; case TPrimitiveType::DOUBLE: return 8; +case TPrimitiveType::QUANTILE_STATE: Review comment: Why ```QUANTILE_STATE``` share the same type length with ```Object Type```? Is QUANTILE variable length type or fixed length? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8279: [feature-wip][array-type] Refactor type info for nested array.
github-actions[bot] commented on pull request #8279: URL: https://github.com/apache/incubator-doris/pull/8279#issuecomment-1055420065 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234: URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r816755862 ## File path: be/src/olap/tablet_schema.cpp ## @@ -246,6 +256,7 @@ uint32_t TabletColumn::get_field_length_by_type(TPrimitiveType::type type, uint3 return 4; case TPrimitiveType::DOUBLE: return 8; +case TPrimitiveType::QUANTILE_STATE: Review comment: QUANTILE_STATE is variable length type, which is same to BITMAP. It only store `ptr`(8 byte) and `length`(8 byte) in the TupleRow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r816756606
##
File path: be/src/olap/row_block.cpp
##
@@ -89,8 +89,7 @@ void RowBlock::_compute_layout() {
_field_offset_in_memory.push_back(memory_size);
// All field has a nullbyte in memory
-if (column.type() == OLAP_FIELD_TYPE_VARCHAR || column.type() ==
OLAP_FIELD_TYPE_HLL ||
-column.type() == OLAP_FIELD_TYPE_CHAR || column.type() ==
OLAP_FIELD_TYPE_OBJECT ||column.type() == OLAP_FIELD_TYPE_STRING) {
+if (column.is_slice_stored()) {
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234: URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r816756313 ## File path: be/src/olap/tablet_schema.cpp ## @@ -246,6 +256,7 @@ uint32_t TabletColumn::get_field_length_by_type(TPrimitiveType::type type, uint3 return 4; case TPrimitiveType::DOUBLE: return 8; +case TPrimitiveType::QUANTILE_STATE: Review comment: QUANTILE_STATE is variable length type, which is same to BITMAP. It only store ptr(8 byte) and length(8 byte) in the TupleRow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zenoyang commented on a change in pull request #8267: [fix](vectorization)Some small fix for SegmentIter Vectorization
zenoyang commented on a change in pull request #8267:
URL: https://github.com/apache/incubator-doris/pull/8267#discussion_r816485202
##
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##
@@ -613,10 +613,7 @@ void SegmentIterator::_vec_init_lazy_materialization() {
_is_pred_column[cid] = true;
pred_column_ids.insert(cid);
-// for date type which can not be executed in a vectorized way,
using short circuit execution
-if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR ||
-type == OLAP_FIELD_TYPE_DECIMAL || type ==
OLAP_FIELD_TYPE_DATE ||
-predicate->is_in_predicate()) {
+if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR || predicate->is_in_predicate()) {
Review comment:
Why is there no OLAP_FIELD_TYPE_STRING type here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a change in pull request #8267: [fix](vectorization)Some small fix for SegmentIter Vectorization
wangbo commented on a change in pull request #8267:
URL: https://github.com/apache/incubator-doris/pull/8267#discussion_r816777518
##
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##
@@ -613,10 +613,7 @@ void SegmentIterator::_vec_init_lazy_materialization() {
_is_pred_column[cid] = true;
pred_column_ids.insert(cid);
-// for date type which can not be executed in a vectorized way,
using short circuit execution
-if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR ||
-type == OLAP_FIELD_TYPE_DECIMAL || type ==
OLAP_FIELD_TYPE_DATE ||
-predicate->is_in_predicate()) {
+if (type == OLAP_FIELD_TYPE_VARCHAR || type ==
OLAP_FIELD_TYPE_CHAR || predicate->is_in_predicate()) {
Review comment:
It's missed, I'll add it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8277: [fix] fix a bug of encryption function with iv may return wrong result
github-actions[bot] commented on pull request #8277: URL: https://github.com/apache/incubator-doris/pull/8277#issuecomment-1055460157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8277: [fix] fix a bug of encryption function with iv may return wrong result
morningman commented on pull request #8277: URL: https://github.com/apache/incubator-doris/pull/8277#issuecomment-1055461806 And do we need to change vectorized method too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8289: [refactor] remove types_test
morningman commented on a change in pull request #8289:
URL: https://github.com/apache/incubator-doris/pull/8289#discussion_r816785983
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarType.java
##
@@ -291,7 +291,8 @@ public static ScalarType createVarcharType(String lenStr) {
public static ScalarType createStringType() {
// length checked in analysis
ScalarType type = new ScalarType(PrimitiveType.STRING);
-type.len = -1;
+// 2GB - 4
+type.len = 2147483643;
Review comment:
Define this magic number some where. Maybe in Type.java?
And explain why minus 4 in comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8248: [fix] (rpc-udf) Fixed the problem that the query could not be interrupted if an error occurred in the rpc server duri
morningman commented on a change in pull request #8248:
URL: https://github.com/apache/incubator-doris/pull/8248#discussion_r815380025
##
File path: samples/doris-demo/README.md
##
@@ -35,6 +35,14 @@ Provide usage examples of Doris stream load, including file
and memory data impo
Provides the use of Spring to integrate Doris, and provides external data
access interfaces
+# rpc-udf-java-demo
+
+Provides an example of how to implements an Doris udf function service in java
+
+# rpc-udf-cpp-demo
+
+Provides an example of how to implements an Doris udf function service in java
Review comment:
```suggestion
Provides an example of how to implements an Doris udf function service in
CPP
```
##
File path: docs/zh-CN/extending-doris/udf/remote-user-defined-function.md
##
@@ -24,7 +24,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Remote User Defined Function Service
+# 远程UDF
Review comment:
可以在这个文章中列出 demo 的代码位置，以方便用户查看。
不然用户不太可能去 doris-demo 文档中查找。
##
File path: be/src/service/function_service.h
##
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "common/status.h"
+#include "gen_cpp/function_service.pb.h"
+#include "runtime/cache/result_cache.h"
+#include "runtime/decimalv2_value.h"
+#include "util/priority_thread_pool.hpp"
+
+namespace brpc {
+class Controller;
+}
+
+namespace doris {
+
+class ExecEnv;
+
+class PFunctionServiceImpl : public PFunctionService {
Review comment:
What is this class for? A demo?
##
File path: be/src/runtime/fragment_mgr.cpp
##
@@ -250,15 +247,18 @@ Status FragmentExecState::cancel_before_execute() {
return Status::OK();
}
-Status FragmentExecState::cancel(const PPlanFragmentCancelReason& reason) {
-std::lock_guard l(_status_lock);
-RETURN_IF_ERROR(_exec_status);
-if (reason == PPlanFragmentCancelReason::LIMIT_REACH) {
-_executor.set_is_report_on_cancel(false);
-}
-_executor.cancel();
-if (_pipe != nullptr) {
-_pipe->cancel(PPlanFragmentCancelReason_Name(reason));
+Status FragmentExecState::cancel(const PPlanFragmentCancelReason& reason,
const std::string& msg) {
+if (!_cancelled) {
+_cancelled = true;
Review comment:
No need to protect with `_status_lock`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8123: [improvement] optimize compaction too slow replica process, will set to ok if the replica compaction is done
morningman commented on a change in pull request #8123:
URL: https://github.com/apache/incubator-doris/pull/8123#discussion_r816838058
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
##
@@ -46,10 +46,12 @@
SCHEMA_CHANGE,
CLONE,
ALTER, // replica is under rollup or schema change
-DECOMMISSION; // replica is ready to be deleted
+DECOMMISSION, // replica is ready to be deleted
+COMPACTION_TOO_SLOW; // replica version count is too large
+
public boolean canLoad() {
-return this == NORMAL || this == SCHEMA_CHANGE || this == ALTER;
+return this == NORMAL || this == SCHEMA_CHANGE || this == ALTER ||
this == COMPACTION_TOO_SLOW;
Review comment:
This is meaningless.
Because every time you set the state to `COMPACTION_TOO_SLOW`, you also set
replica `isBad` to `true`.
I think we can only set replica's state to `COMPACTION_TOO_SLOW`, and change
logic in `replica.isAlive()`.
##
File path: fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
##
@@ -647,7 +647,10 @@ private void handleReplicaMissing(TabletSchedCtx
tabletCtx, AgentBatchTask batch
RootPathLoadStatistic destPath = chooseAvailableDestPath(tabletCtx,
tag, false /* not for colocate */);
Preconditions.checkNotNull(destPath);
tabletCtx.setDest(destPath.getBeId(), destPath.getPathHash());
-
+// check compaction too slow file is recovered
+if (tabletCtx.compactionRecovered()) {
Review comment:
why not check it before `chooseAvailableDestPath()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8288: [Refactor] remove pusher.cpp and related mock test code
github-actions[bot] commented on pull request #8288: URL: https://github.com/apache/incubator-doris/pull/8288#issuecomment-1055527192 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8280: [Bug][Vectorized] Fix core dump of mutable block different of block
github-actions[bot] commented on pull request #8280: URL: https://github.com/apache/incubator-doris/pull/8280#issuecomment-1055528755 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8282: [Bug] fix string type length limit error && add runtime filter decimal support
morningman commented on a change in pull request #8282:
URL: https://github.com/apache/incubator-doris/pull/8282#discussion_r816845580
##
File path: be/src/vec/core/block.cpp
##
@@ -720,12 +720,9 @@ Status Block::serialize(PBlock* pblock, size_t*
uncompressed_bytes, size_t* comp
// serialize data values
allocated_buf->resize(content_uncompressed_size);
char* buf = allocated_buf->data();
-char* start_buf = buf;
for (const auto& c : *this) {
buf = c.type->serialize(*(c.column), buf);
}
-CHECK(content_uncompressed_size == (buf - start_buf))
Review comment:
why remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8273: [Refactor] Remove agent status
morningman commented on pull request #8273: URL: https://github.com/apache/incubator-doris/pull/8273#issuecomment-1055533379 > Details ut has been fixed. you need to rebase the master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8273: [Refactor] Remove agent status
morningman commented on a change in pull request #8273:
URL: https://github.com/apache/incubator-doris/pull/8273#discussion_r816850436
##
File path: be/src/agent/pusher.cpp
##
@@ -47,16 +47,15 @@ Pusher::Pusher(OLAPEngine* engine, const TPushReq& push_req)
Pusher::~Pusher() {}
-AgentStatus Pusher::init() {
-AgentStatus status = DORIS_SUCCESS;
+Status Pusher::init() {
+Status status = Status::OK();
Review comment:
Remove this line and define it at line 74
##
File path: be/src/agent/pusher.cpp
##
@@ -73,7 +72,7 @@ AgentStatus Pusher::init() {
// Set download param
string tmp_file_dir;
status = _get_tmp_file_dir(olap_table->storage_root_path_name(),
&tmp_file_dir);
-if (status != DORIS_SUCCESS) {
+if (status != Status::OK()) {
LOG(WARNING) << "get local path failed. tmp file dir: " <<
tmp_file_dir;
Review comment:
print `st.get_error_msg()`
##
File path: be/src/agent/task_worker_pool.cpp
##
@@ -1541,11 +1523,11 @@ void TaskWorkerPool::_move_dir_thread_callback() {
TStatus task_status;
// TODO: move dir
-AgentStatus status =
+Status status =
_move_dir(move_dir_req.tablet_id, move_dir_req.schema_hash,
move_dir_req.src,
move_dir_req.job_id, true /* TODO */, &error_msgs);
-if (status != DORIS_SUCCESS) {
+if (status != Status::OK()) {
Review comment:
use `status.ok()`.
Better to modify all other places.
##
File path: be/src/agent/task_worker_pool.cpp
##
@@ -1581,7 +1563,7 @@ AgentStatus TaskWorkerPool::_move_dir(const TTabletId
tablet_id, const TSchemaHa
LOG(INFO) << "failed to get tablet. tablet_id:" << tablet_id
<< ", schema hash:" << schema_hash;
error_msgs->push_back("failed to get tablet");
-return DORIS_TASK_REQUEST_ERROR;
+return Status::InvalidArgument("Could not find tablet");
Review comment:
If using Status, we don't need `error_msgs` anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8256: [Enhancement](routine-load) Support routine load task succeed with empty data consumed
morningman commented on pull request #8256: URL: https://github.com/apache/incubator-doris/pull/8256#issuecomment-107609 Does this affect the overall behavior of the job? Because I understand that "cancel" is only internal logic and does not affect the continuation of the routine load job? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8245: fix restore with partitions cause some query fail
github-actions[bot] commented on pull request #8245: URL: https://github.com/apache/incubator-doris/pull/8245#issuecomment-1055561548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-spark-connector] zhaomin1423 opened a new pull request #9: set \\N value when the value of filed is NULL
zhaomin1423 opened a new pull request #9: URL: https://github.com/apache/incubator-doris-spark-connector/pull/9 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-spark-connector] zhaomin1423 closed pull request #9: set \\N value when the value of filed is NULL
zhaomin1423 closed pull request #9: URL: https://github.com/apache/incubator-doris-spark-connector/pull/9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a change in pull request #8273: [Refactor] Remove agent status
yiguolei commented on a change in pull request #8273:
URL: https://github.com/apache/incubator-doris/pull/8273#discussion_r817258663
##
File path: be/src/agent/pusher.cpp
##
@@ -47,16 +47,15 @@ Pusher::Pusher(OLAPEngine* engine, const TPushReq& push_req)
Pusher::~Pusher() {}
-AgentStatus Pusher::init() {
-AgentStatus status = DORIS_SUCCESS;
+Status Pusher::init() {
+Status status = Status::OK();
Review comment:
I already removed pusher.cpp file in next pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a change in pull request #8273: [Refactor] Remove agent status
yiguolei commented on a change in pull request #8273:
URL: https://github.com/apache/incubator-doris/pull/8273#discussion_r817258851
##
File path: be/src/agent/pusher.cpp
##
@@ -73,7 +72,7 @@ AgentStatus Pusher::init() {
// Set download param
string tmp_file_dir;
status = _get_tmp_file_dir(olap_table->storage_root_path_name(),
&tmp_file_dir);
-if (status != DORIS_SUCCESS) {
+if (status != Status::OK()) {
LOG(WARNING) << "get local path failed. tmp file dir: " <<
tmp_file_dir;
Review comment:
I have already removed pusher.cpp file in next pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8053: [docs] fix document date-time-functions typo
morningman merged pull request #8053: URL: https://github.com/apache/incubator-doris/pull/8053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (940efc6 -> 5cc62fa)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 940efc6 [Fix]Remove duplicated destructor function in MinMaxFuncBase
(#8287)
add 5cc62fa [docs] fix document date-time-functions typo (#8053)
No new revisions were added by this update.
Summary of changes:
docs/.vuepress/sidebar/en.js | 4 +++-
docs/.vuepress/sidebar/zh-CN.js | 2 +-
.../sql-functions/date-time-functions/convert_tz.md | 12 +---
.../sql-functions/date-time-functions/curdate.md | 7 ---
.../sql-functions/date-time-functions/curtime.md | 12 +---
.../sql-functions/date-time-functions/str_to_date.md | 4 ++--
.../sql-functions/date-time-functions/convert_tz.md | 10 --
.../sql-functions/date-time-functions/curdate.md | 8
8 files changed, 28 insertions(+), 31 deletions(-)
copy docs/{zh-CN =>
en}/sql-reference/sql-functions/date-time-functions/convert_tz.md (90%)
copy docs/{zh-CN =>
en}/sql-reference/sql-functions/date-time-functions/curtime.md (92%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] junjie4585 closed issue #8171: [Bug] binlog load 方式同步数据，报 errCode = 2, detailMessage = transaction not found异常
junjie4585 closed issue #8171: URL: https://github.com/apache/incubator-doris/issues/8171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] junjie4585 commented on issue #8171: [Bug] binlog load 方式同步数据，报 errCode = 2, detailMessage = transaction not found异常
junjie4585 commented on issue #8171: URL: https://github.com/apache/incubator-doris/issues/8171#issuecomment-1056061584 列映射问题，显示指定列映射才能正常导入 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8156: [Transactions][Stmt] Support view transactions info for specified status by `SHOW TRANSACTION` stmt
morningman merged pull request #8156: URL: https://github.com/apache/incubator-doris/pull/8156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (5cc62fa -> 236105d)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 5cc62fa [docs] fix document date-time-functions typo (#8053) add 236105d [feature][show-transaction] Support view transactions info for specified status by `SHOW TRANSACTION` stmt (#8156) No new revisions were added by this update. Summary of changes: .../Data Manipulation/SHOW TRANSACTION.md | 12 +-- .../Data Manipulation/SHOW TRANSACTION.md | 15 ++--- .../apache/doris/analysis/ShowTransactionStmt.java | 19 +++- .../java/org/apache/doris/qe/ShowExecutor.java | 20 +++-- .../doris/transaction/DatabaseTransactionMgr.java | 25 ++ .../doris/transaction/GlobalTransactionMgr.java| 5 + 6 files changed, 83 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
morningman merged pull request #8247: URL: https://github.com/apache/incubator-doris/pull/8247 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [improvement] Upgrade MySQL version to 5.7.37 to reduce unnecessary CVE issues (#8247)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new d5b6428 [improvement] Upgrade MySQL version to 5.7.37 to reduce
unnecessary CVE issues (#8247)
d5b6428 is described below
commit d5b6428c6df4952e4262b882ab579aa4ee8e6b4e
Author: aopangzi <99486640+aopan...@users.noreply.github.com>
AuthorDate: Wed Mar 2 10:16:02 2022 +0800
[improvement] Upgrade MySQL version to 5.7.37 to reduce unnecessary CVE
issues (#8247)
---
docs/en/administrator-guide/variables.md | 12 +
docs/zh-CN/administrator-guide/variables.md| 12 +
.../apache/doris/mysql/MysqlHandshakePacket.java | 2 +-
.../java/org/apache/doris/qe/GlobalVariable.java | 4 ++-
.../java/org/apache/doris/qe/SessionVariable.java | 30 +-
.../doris/mysql/MysqlHandshakePacketTest.java | 2 +-
6 files changed, 58 insertions(+), 4 deletions(-)
diff --git a/docs/en/administrator-guide/variables.md
b/docs/en/administrator-guide/variables.md
index b03a85d..e6d6e22 100644
--- a/docs/en/administrator-guide/variables.md
+++ b/docs/en/administrator-guide/variables.md
@@ -373,6 +373,18 @@ Translated with www.DeepL.com/Translator (free version)
Used for compatibility with MySQL clients. No practical effect.
+* `tx_read_only`
+
+Used for compatibility with MySQL clients. No practical effect.
+
+* `transaction_read_only`
+
+Used for compatibility with MySQL clients. No practical effect.
+
+* `transaction_isolation`
+
+Used for compatibility with MySQL clients. No practical effect.
+
* `version`
Used for compatibility with MySQL clients. No practical effect.
diff --git a/docs/zh-CN/administrator-guide/variables.md
b/docs/zh-CN/administrator-guide/variables.md
index 7506abe..219e860 100644
--- a/docs/zh-CN/administrator-guide/variables.md
+++ b/docs/zh-CN/administrator-guide/variables.md
@@ -367,6 +367,18 @@ SELECT /*+ SET_VAR(query_timeout = 1,
enable_partition_cache=true) */ sleep(3);
用于兼容 MySQL 客户端。无实际作用。
+* `tx_read_only`
+
+用于兼容 MySQL 客户端。无实际作用。
+
+* `transaction_read_only`
+
+用于兼容 MySQL 客户端。无实际作用。
+
+* `transaction_isolation`
+
+用于兼容 MySQL 客户端。无实际作用。
+
* `version`
用于兼容 MySQL 客户端。无实际作用。
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java
index ed7aec9..856b2e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java
@@ -23,7 +23,7 @@ public class MysqlHandshakePacket extends MysqlPacket {
// Version of handshake packet, since MySQL 3.21.0, Handshake of protocol
10 is used
private static final int PROTOCOL_VERSION = 10;
// JDBC uses this version to check which protocol the server support
-private static final String SERVER_VERSION = "5.1.73";
+public static final String SERVER_VERSION = "5.7.37";
// 33 stands for UTF-8 character set
private static final int CHARACTER_SET = 33;
// use default capability for all
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
index b6612a2..9a0e817 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
@@ -20,6 +20,8 @@ package org.apache.doris.qe;
import org.apache.doris.common.Version;
import org.apache.doris.common.util.TimeUtils;
+import org.apache.doris.mysql.MysqlHandshakePacket;
+
import com.google.common.collect.Lists;
import java.lang.reflect.Field;
@@ -47,7 +49,7 @@ public final class GlobalVariable {
Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH;
@VariableMgr.VarAttr(name = VERSION, flag = VariableMgr.READ_ONLY)
-public static String version = "5.1.0";
+public static String version = MysqlHandshakePacket.SERVER_VERSION;
// 0: table names are stored as specified and comparisons are case
sensitive.
// 1: table names are stored in lowercase on disk and comparisons are not
case sensitive.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 1f55f39..c15f717 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -48,6 +48,9 @@ public class SessionVariable implements Serializable,
Writable {
public static final String RESOURCE_VARIABLE = "resource_group";
public static final String AUTO_COMMIT = "autocommit";
public static final String
[incubator-doris] branch master updated (d5b6428 -> 315bfe2)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from d5b6428 [improvement] Upgrade MySQL version to 5.7.37 to reduce unnecessary CVE issues (#8247) add 315bfe2 Revert "[chore](dependency) upgrade-grpc-version (#8218)" (#8250) No new revisions were added by this update. Summary of changes: fe/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8250: [revert] "[improvement]upgrade-grpc-version"
morningman merged pull request #8250: URL: https://github.com/apache/incubator-doris/pull/8250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](fold-constant)(hive) fix constant-folding in order by and optimize logs in hive client (#8268)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new f41316a [fix](fold-constant)(hive) fix constant-folding in order by
and optimize logs in hive client (#8268)
f41316a is described below
commit f41316a3ec4015b317006078978f4c860c5d8832
Author: qiye
AuthorDate: Wed Mar 2 10:17:17 2022 +0800
[fix](fold-constant)(hive) fix constant-folding in order by and optimize
logs in hive client (#8268)
fix: #7509
1. fix order by clause constant folding
2. optimize Hive exception message
3. change hive file status log type
---
.../java/org/apache/doris/analysis/SelectStmt.java | 7 ---
.../doris/catalog/HiveMetaStoreClientHelper.java | 20 ++--
.../java/org/apache/doris/planner/HiveScanNode.java | 2 +-
3 files changed, 15 insertions(+), 14 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 502a7fd..779800c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -1450,7 +1450,7 @@ public class SelectStmt extends QueryStmt {
* ORDER BY `b` DESC
* ```
* Aliases information of groupBy and orderBy clauses is
recorded in `QueryStmt.aliasSMap`.
- * The select clause has it's own alias info in
`SelectListItem.alias`.
+ * The select clause has its own alias info in
`SelectListItem.alias`.
*
* Aliases expr in the `group by` and `order by` clauses
are not analyzed, i.e. `Expr.isAnalyzed=false`
* Subsequent constant folding will analyze the unanalyzed
Expr before collecting the constant
@@ -1471,7 +1471,7 @@ public class SelectStmt extends QueryStmt {
}
}
if (orderByElements != null) {
-for (OrderByElement orderByElem : orderByElements) {
+for (OrderByElement orderByElem : orderByElementsAfterAnalyzed) {
// same as above
if (containAlias(orderByElem.getExpr())) {
continue;
@@ -1548,7 +1548,7 @@ public class SelectStmt extends QueryStmt {
}
}
if (orderByElements != null) {
-for (OrderByElement orderByElem : orderByElements) {
+for (OrderByElement orderByElem : orderByElementsAfterAnalyzed) {
Expr expr = orderByElem.getExpr();
if (expr.getId() == null) {
orderByElem.setExpr(expr);
@@ -1556,6 +1556,7 @@ public class SelectStmt extends QueryStmt {
orderByElem.setExpr(rewrittenExprMap.get(expr.getId().toString()));
}
}
+orderByElements = (ArrayList)
orderByElementsAfterAnalyzed;
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
index 5dcfd12..8ca9d7b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
@@ -126,7 +126,7 @@ public class HiveMetaStoreClientHelper {
hivemetastoreclient = new HiveMetaStoreClient(hiveConf);
} catch (MetaException e) {
LOG.warn("Create HiveMetaStoreClient failed: {}", e.getMessage());
-throw new DdlException("Create HiveMetaStoreClient failed");
+throw new DdlException("Create HiveMetaStoreClient failed: " +
e.getMessage());
}
return hivemetastoreclient;
}
@@ -144,7 +144,7 @@ public class HiveMetaStoreClientHelper {
return client.tableExists(dbName, tblName);
} catch (TException e) {
LOG.warn("Hive metastore thrift exception: {}", e.getMessage());
-throw new DdlException("Connect hive metastore failed.");
+throw new DdlException("Connect hive metastore failed. Error: " +
e.getMessage());
} finally {
dropClient(client);
}
@@ -179,7 +179,7 @@ public class HiveMetaStoreClientHelper {
SerializationUtilities.serializeExpressionToKryo(hivePartitionPredicate), null,
(short) -1, hivePartitions);
} catch (TException e) {
LOG.warn("Hive metastore thrift exception: {}",
e.getMessage());
-throw new DdlException("Connect hive metastore failed.");
+throw new DdlException("Connect hive metastore failed. Error:
" + e.getMessage());
} finally {
[GitHub] [incubator-doris] morningman closed issue #7509: [Bug] fold constant error in order by clause
morningman closed issue #7509: URL: https://github.com/apache/incubator-doris/issues/7509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8268: [fix](fold-constant)(hive) fix constant-folding in order by and optimize logs in hive client
morningman merged pull request #8268: URL: https://github.com/apache/incubator-doris/pull/8268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8271: make options of build.sh and run-be-ut.sh work
morningman merged pull request #8271: URL: https://github.com/apache/incubator-doris/pull/8271 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (f41316a -> 77b21fb)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from f41316a [fix](fold-constant)(hive) fix constant-folding in order by and optimize logs in hive client (#8268) add 77b21fb [chore] make options of build.sh and run-be-ut.sh work (#8271) No new revisions were added by this update. Summary of changes: build.sh | 3 +-- run-be-ut.sh | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8274: [Docs] Fix a typo in downloads.md
morningman merged pull request #8274: URL: https://github.com/apache/incubator-doris/pull/8274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (77b21fb -> efe88be)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 77b21fb [chore] make options of build.sh and run-be-ut.sh work (#8271) add efe88be [docs] Fix a typo in downloads.md (#8274) No new revisions were added by this update. Summary of changes: docs/zh-CN/downloads/downloads.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8291: [Bug][Vectorized] Fix exchange node merge sort null first order wrong
morningman merged pull request #8291: URL: https://github.com/apache/incubator-doris/pull/8291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8290: [Bug] [Vectorized] Exchange merge null first order is wrong
morningman closed issue #8290: URL: https://github.com/apache/incubator-doris/issues/8290 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix][Vectorized] Fix exchange node merge sort null first order wrong (#8291)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new b241bc4 [fix][Vectorized] Fix exchange node merge sort null first
order wrong (#8291)
b241bc4 is described below
commit b241bc4e9dfe5893a4292fe7671cf33ff9629718
Author: HappenLee
AuthorDate: Wed Mar 2 10:19:06 2022 +0800
[fix][Vectorized] Fix exchange node merge sort null first order wrong
(#8291)
---
be/src/vec/core/sort_cursor.h | 2 +-
be/src/vec/runtime/vsorted_run_merger.cpp | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/core/sort_cursor.h b/be/src/vec/core/sort_cursor.h
index d9712b4..54557f1 100644
--- a/be/src/vec/core/sort_cursor.h
+++ b/be/src/vec/core/sort_cursor.h
@@ -118,7 +118,7 @@ struct ReceiveQueueSortCursorImpl : public SortCursorImpl {
desc.resize(ordering_expr.size());
for (int i = 0; i < desc.size(); i++) {
desc[i].direction = is_asc_order[i] ? 1 : -1;
-desc[i].nulls_direction = nulls_first[i] ? -1 : 1;
+desc[i].nulls_direction = nulls_first[i] ? -desc[i].direction :
desc[i].direction;
}
_is_eof = !has_next_block();
}
diff --git a/be/src/vec/runtime/vsorted_run_merger.cpp
b/be/src/vec/runtime/vsorted_run_merger.cpp
index 3332085..0d0ecc0 100644
--- a/be/src/vec/runtime/vsorted_run_merger.cpp
+++ b/be/src/vec/runtime/vsorted_run_merger.cpp
@@ -41,7 +41,7 @@ VSortedRunMerger::VSortedRunMerger(const
std::vector& ordering_e
}
Status VSortedRunMerger::prepare(const vector& input_runs, bool
parallel) {
-for (const auto &supplier : input_runs) {
+for (const auto& supplier : input_runs) {
_cursors.emplace_back(supplier, _ordering_expr, _is_asc_order,
_nulls_first);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch dev-1.0.0 updated (2be9380 -> 5cff344)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch dev-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 2be9380 [fix][chore](insert)(fe) Fix analysis error of insert stmt and modify grpc-netty dependency (#8265) new 24a8836 [Fix] Function percentile input null return null (#8238) new 2274629 [improvement][vec] better memequal impl to speed up string compare (#8229) new 4a7667d [Fix]Remove duplicated destructor function in MinMaxFuncBase (#8287) new 5cff344 [fix][Vectorized] Fix exchange node merge sort null first order wrong (#8291) The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/exec/mysql_scanner.cpp | 1 - be/src/exprs/minmax_predicate.h| 3 +- be/src/runtime/string_value.hpp| 4 + be/src/util/counts.h | 2 +- be/src/vec/common/string_ref.h | 107 +++-- be/src/vec/core/sort_cursor.h | 2 +- be/src/vec/functions/function_json.cpp | 8 +- be/src/vec/functions/function_string.cpp | 18 ++-- be/src/vec/functions/function_timestamp.cpp| 9 +- be/src/vec/functions/function_totype.h | 4 +- be/src/vec/runtime/vsorted_run_merger.cpp | 2 +- be/test/vec/function/function_string_test.cpp | 19 .../aggregate-functions/percentile.md | 8 ++ .../aggregate-functions/percentile.md | 7 ++ 14 files changed, 97 insertions(+), 97 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 03/04: [Fix]Remove duplicated destructor function in MinMaxFuncBase (#8287)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch dev-1.0.0 in repository https://gitbox.apache.org/repos/asf/incubator-doris.git commit 4a7667d266ff5b666b3fba1fc16f0a4d3ecd1f7b Author: Mr.Hu AuthorDate: Tue Mar 1 18:38:09 2022 +0800 [Fix]Remove duplicated destructor function in MinMaxFuncBase (#8287) --- be/src/exprs/minmax_predicate.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/be/src/exprs/minmax_predicate.h b/be/src/exprs/minmax_predicate.h index 1004560..3a9ff5b 100644 --- a/be/src/exprs/minmax_predicate.h +++ b/be/src/exprs/minmax_predicate.h @@ -35,8 +35,6 @@ public: virtual Status assign(void* min_data, void* max_data) = 0; // merge from other minmax_func virtual Status merge(MinMaxFuncBase* minmax_func, ObjectPool* pool) = 0; -virtual ~MinMaxFuncBase() = default; -; }; template - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 01/04: [Fix] Function percentile input null return null (#8238)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 24a8836c5f375ef259d905c04200ddff359d6909
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Tue Mar 1 14:42:48 2022 +0800
[Fix] Function percentile input null return null (#8238)
---
be/src/exec/mysql_scanner.cpp | 1 -
be/src/exprs/minmax_predicate.h | 1 +
be/src/util/counts.h | 2 +-
be/src/vec/functions/function_json.cpp| 8
be/src/vec/functions/function_string.cpp | 18 +-
be/src/vec/functions/function_timestamp.cpp | 9 +++--
be/src/vec/functions/function_totype.h| 4 ++--
be/test/vec/function/function_string_test.cpp | 19 +++
.../sql-functions/aggregate-functions/percentile.md | 8
.../sql-functions/aggregate-functions/percentile.md | 7 +++
10 files changed, 58 insertions(+), 19 deletions(-)
diff --git a/be/src/exec/mysql_scanner.cpp b/be/src/exec/mysql_scanner.cpp
index a6be664..65f6e1e 100644
--- a/be/src/exec/mysql_scanner.cpp
+++ b/be/src/exec/mysql_scanner.cpp
@@ -41,7 +41,6 @@ MysqlScanner::~MysqlScanner() {
if (_my_conn) {
mysql_close(_my_conn);
_my_conn = nullptr;
-mysql_library_end();
}
}
diff --git a/be/src/exprs/minmax_predicate.h b/be/src/exprs/minmax_predicate.h
index 61f7f1f..1004560 100644
--- a/be/src/exprs/minmax_predicate.h
+++ b/be/src/exprs/minmax_predicate.h
@@ -25,6 +25,7 @@ namespace doris {
// only used in Runtime Filter
class MinMaxFuncBase {
public:
+virtual ~MinMaxFuncBase() = default;
virtual void insert(const void* data) = 0;
virtual bool find(void* data) = 0;
virtual bool is_empty() = 0;
diff --git a/be/src/util/counts.h b/be/src/util/counts.h
index 6a3d103..dc1cd3d 100644
--- a/be/src/util/counts.h
+++ b/be/src/util/counts.h
@@ -109,7 +109,7 @@ public:
doris_udf::DoubleVal terminate(double quantile) const {
if (_counts.empty()) {
-return doris_udf::DoubleVal();
+return doris_udf::DoubleVal::null();
}
std::vector> elems(_counts.begin(),
_counts.end());
diff --git a/be/src/vec/functions/function_json.cpp
b/be/src/vec/functions/function_json.cpp
index 466bc7b..ee4b6c9 100644
--- a/be/src/vec/functions/function_json.cpp
+++ b/be/src/vec/functions/function_json.cpp
@@ -224,7 +224,7 @@ struct GetJsonNumberType {
using ReturnType = typename NumberType::ReturnType;
using ColumnType = typename NumberType::ColumnType;
using Container = typename ColumnType::Container;
-static void vector_vector(const ColumnString::Chars& ldata,
+static void vector_vector(FunctionContext* context, const
ColumnString::Chars& ldata,
const ColumnString::Offsets& loffsets,
const ColumnString::Chars& rdata,
const ColumnString::Offsets& roffsets,
Container& res,
@@ -314,9 +314,9 @@ struct GetJsonString {
using ColumnType = ColumnString;
using Chars = ColumnString::Chars;
using Offsets = ColumnString::Offsets;
-static void vector_vector(const Chars& ldata, const Offsets& loffsets,
const Chars& rdata,
- const Offsets& roffsets, Chars& res_data,
Offsets& res_offsets,
- NullMap& null_map) {
+static void vector_vector(FunctionContext* context, const Chars& ldata,
const Offsets& loffsets,
+ const Chars& rdata, const Offsets& roffsets,
Chars& res_data,
+ Offsets& res_offsets, NullMap& null_map) {
size_t input_rows_count = loffsets.size();
res_offsets.resize(input_rows_count);
diff --git a/be/src/vec/functions/function_string.cpp
b/be/src/vec/functions/function_string.cpp
index 94ed9c9..926e8ed 100644
--- a/be/src/vec/functions/function_string.cpp
+++ b/be/src/vec/functions/function_string.cpp
@@ -485,9 +485,9 @@ struct AesEncryptImpl {
using Offsets = ColumnString::Offsets;
using ReturnType = DataTypeString;
using ColumnType = ColumnString;
-static void vector_vector(const Chars& ldata, const Offsets& loffsets,
const Chars& rdata,
- const Offsets& roffsets, Chars& res_data,
Offsets& res_offsets,
- NullMap& null_map_data) {
+static void vector_vector(FunctionContext* context, const Chars& ldata,
const Offsets& loffsets,
+ const Chars& rdata, const Offsets& roffsets,
Chars& res_data,
+ Offsets& res_offsets, NullMap& null_map_data) {
DCHECK_EQ(loffsets.size(), roffsets.size());
size_t input_rows_count = loffsets.size();
[incubator-doris] 02/04: [improvement][vec] better memequal impl to speed up string compare (#8229)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 22746292fd73fa0a9fae40a8dde8eafd6a2a1f3c
Author: zbtzbtzbt <35688959+zbtzbt...@users.noreply.github.com>
AuthorDate: Tue Mar 1 11:25:12 2022 +0800
[improvement][vec] better memequal impl to speed up string compare (#8229)
like #8214
faster string compare operator in vec engine.
---
be/src/runtime/string_value.hpp | 4 ++
be/src/vec/common/string_ref.h | 107 +---
2 files changed, 37 insertions(+), 74 deletions(-)
diff --git a/be/src/runtime/string_value.hpp b/be/src/runtime/string_value.hpp
index c44115d..aac9e3a 100644
--- a/be/src/runtime/string_value.hpp
+++ b/be/src/runtime/string_value.hpp
@@ -22,6 +22,7 @@
#include "runtime/string_value.h"
#include "util/cpu_info.h"
+#include "vec/common/string_ref.h"
#ifdef __SSE4_2__
#include "util/sse_util.hpp"
#endif
@@ -88,6 +89,9 @@ inline bool StringValue::eq(const StringValue& other) const {
if (this->len != other.len) {
return false;
}
+#if defined(__SSE2__)
+return memequalSSE2Wide(this->ptr, other.ptr, this->len);
+#endif
return string_compare(this->ptr, this->len, other.ptr, other.len,
this->len) == 0;
}
diff --git a/be/src/vec/common/string_ref.h b/be/src/vec/common/string_ref.h
index 5dd146e..8ecbe07 100644
--- a/be/src/vec/common/string_ref.h
+++ b/be/src/vec/common/string_ref.h
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
// This file is copied from
-// https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/StringRef.h
+// https://github.com/ClickHouse/ClickHouse/blob/master/base/base/StringRef.h
// and modified by Doris
#pragma once
@@ -98,6 +98,32 @@ inline bool compareSSE2x4(const char* p1, const char* p2) {
}
inline bool memequalSSE2Wide(const char* p1, const char* p2, size_t size) {
+/** The order of branches and the trick with overlapping comparisons
+ * are the same as in memcpy implementation.
+ * See the comments in
+ *
https://github.com/ClickHouse/ClickHouse/blob/master/base/glibc-compatibility/memcpy/memcpy.h
+ */
+
+if (size <= 16) {
+if (size >= 8) {
+/// Chunks of [8,16] bytes.
+return unaligned_load(p1) ==
unaligned_load(p2) &&
+ unaligned_load(p1 + size - 8) ==
unaligned_load(p2 + size - 8);
+} else if (size >= 4) {
+/// Chunks of [4,7] bytes.
+return unaligned_load(p1) ==
unaligned_load(p2) &&
+ unaligned_load(p1 + size - 4) ==
unaligned_load(p2 + size - 4);
+} else if (size >= 2) {
+/// Chunks of [2,3] bytes.
+return unaligned_load(p1) ==
unaligned_load(p2) &&
+ unaligned_load(p1 + size - 2) ==
unaligned_load(p2 + size - 2);
+} else if (size >= 1) {
+/// A single byte.
+return *p1 == *p2;
+}
+return true;
+}
+
while (size >= 64) {
if (compareSSE2x4(p1, p2)) {
p1 += 64;
@@ -107,74 +133,14 @@ inline bool memequalSSE2Wide(const char* p1, const char*
p2, size_t size) {
return false;
}
-switch ((size % 64) / 16) {
-case 3:
-if (!compareSSE2(p1 + 32, p2 + 32)) return false;
-[[fallthrough]];
-case 2:
-if (!compareSSE2(p1 + 16, p2 + 16)) return false;
-[[fallthrough]];
-case 1:
-if (!compareSSE2(p1, p2)) return false;
-[[fallthrough]];
-case 0:
-break;
-}
-
-p1 += (size % 64) / 16 * 16;
-p2 += (size % 64) / 16 * 16;
-
-switch (size % 16) {
-case 15:
-if (p1[14] != p2[14]) return false;
-[[fallthrough]];
-case 14:
-if (p1[13] != p2[13]) return false;
-[[fallthrough]];
-case 13:
-if (p1[12] != p2[12]) return false;
-[[fallthrough]];
-case 12:
-if (unaligned_load(p1 + 8) == unaligned_load(p2 +
8))
-goto l8;
-else
-return false;
-case 11:
-if (p1[10] != p2[10]) return false;
-[[fallthrough]];
-case 10:
-if (p1[9] != p2[9]) return false;
-[[fallthrough]];
-case 9:
-if (p1[8] != p2[8]) return false;
-l8:
-[[fallthrough]];
-case 8:
-return unaligned_load(p1) == unaligned_load(p2);
-case 7:
-if (p1[6] != p2[6]) return false;
-[[fallthrough]];
-case 6:
-if (p1[5] != p2[5]) return false;
-[[fallthrough]];
-case 5:
-if (p1[4] != p2[4]) return false;
-[[fallthrough]];
-case 4:
-return unaligned_load(p1) == unaligned_load(p2);
-case 3:
-if (p1[2] != p2[2]) return false;
-[[fallthrough]];
-case 2:
-return unaligned_load(p1) == unaligned
[incubator-doris] 04/04: [fix][Vectorized] Fix exchange node merge sort null first order wrong (#8291)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5cff344f8587a6a8b49bf1bc826bd7ef283161a5
Author: HappenLee
AuthorDate: Wed Mar 2 10:19:06 2022 +0800
[fix][Vectorized] Fix exchange node merge sort null first order wrong
(#8291)
---
be/src/vec/core/sort_cursor.h | 2 +-
be/src/vec/runtime/vsorted_run_merger.cpp | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/core/sort_cursor.h b/be/src/vec/core/sort_cursor.h
index d9712b4..54557f1 100644
--- a/be/src/vec/core/sort_cursor.h
+++ b/be/src/vec/core/sort_cursor.h
@@ -118,7 +118,7 @@ struct ReceiveQueueSortCursorImpl : public SortCursorImpl {
desc.resize(ordering_expr.size());
for (int i = 0; i < desc.size(); i++) {
desc[i].direction = is_asc_order[i] ? 1 : -1;
-desc[i].nulls_direction = nulls_first[i] ? -1 : 1;
+desc[i].nulls_direction = nulls_first[i] ? -desc[i].direction :
desc[i].direction;
}
_is_eof = !has_next_block();
}
diff --git a/be/src/vec/runtime/vsorted_run_merger.cpp
b/be/src/vec/runtime/vsorted_run_merger.cpp
index 3332085..0d0ecc0 100644
--- a/be/src/vec/runtime/vsorted_run_merger.cpp
+++ b/be/src/vec/runtime/vsorted_run_merger.cpp
@@ -41,7 +41,7 @@ VSortedRunMerger::VSortedRunMerger(const
std::vector& ordering_e
}
Status VSortedRunMerger::prepare(const vector& input_runs, bool
parallel) {
-for (const auto &supplier : input_runs) {
+for (const auto& supplier : input_runs) {
_cursors.emplace_back(supplier, _ordering_expr, _is_asc_order,
_nulls_first);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a change in pull request #8282: [Bug] fix string type length limit error && add runtime filter decimal support
BiteThet commented on a change in pull request #8282:
URL: https://github.com/apache/incubator-doris/pull/8282#discussion_r817302755
##
File path: be/src/vec/core/block.cpp
##
@@ -720,12 +720,9 @@ Status Block::serialize(PBlock* pblock, size_t*
uncompressed_bytes, size_t* comp
// serialize data values
allocated_buf->resize(content_uncompressed_size);
char* buf = allocated_buf->data();
-char* start_buf = buf;
for (const auto& c : *this) {
buf = c.type->serialize(*(c.column), buf);
}
-CHECK(content_uncompressed_size == (buf - start_buf))
Review comment:
When data type is HLL, content_uncompressed_size maybe larger than real
size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #8202: [improvment] show export support label like
EmmyMiao87 commented on a change in pull request #8202:
URL: https://github.com/apache/incubator-doris/pull/8202#discussion_r817310040
##
File path: fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
##
@@ -147,7 +155,9 @@ private ExportJob createJob(long jobId, ExportStmt stmt)
throws Exception {
continue;
}
Review comment:
```suggestion
If ( label not null or empty) {
if (isLabelUseLike) {
check pattern;
} else {
check job label;
}
}
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
##
@@ -126,11 +129,16 @@ private ExportJob createJob(long jobId, ExportStmt stmt)
throws Exception {
// NOTE: jobid and states may both specified, or only one of them, or
neither
public List> getExportJobInfosByIdOrState(
-long dbId, long jobId, String label, Set
states,
-ArrayList orderByPairs, long limit) {
+long dbId, long jobId, String label, boolean isLabelUseLike,
Set states,
+ArrayList orderByPairs, long limit) throws
AnalysisException {
long resultNum = limit == -1L ? Integer.MAX_VALUE : limit;
LinkedList> exportJobInfos = new
LinkedList>();
+PatternMatcher matcher = null;
+if(isLabelUseLike){
Review comment:
Pay attention about code format ~
##
File path: fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
##
@@ -147,7 +155,9 @@ private ExportJob createJob(long jobId, ExportStmt stmt)
throws Exception {
continue;
}
-if (!Strings.isNullOrEmpty(label) && !jobLabel.equals(label)) {
+if (!Strings.isNullOrEmpty(label) &&
Review comment:
The above way of writing is more readable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #8289: [refactor] remove types_test
HappenLee commented on a change in pull request #8289: URL: https://github.com/apache/incubator-doris/pull/8289#discussion_r817311314 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarType.java ## @@ -69,6 +66,12 @@ // Keep consistent with backend ColumnType::CHAR_INLINE_LENGTH public static final int CHAR_INLINE_LENGTH = 128; +// Max length of String types, in be storage layer store string length +// using int32, the max length is 2GB, the first 4 bytes store the length +// so the max available length is 2GB - 4 + Review comment: no need the space？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BradySuen commented on issue #6080: docker编译出错 [205/678] Building CXX object src/http/CMakeFiles/Webserver.dir/action/mini_load.cpp.o
BradySuen commented on issue #6080: URL: https://github.com/apache/incubator-doris/issues/6080#issuecomment-1056137227 ditto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] levy5307 commented on a change in pull request #8289: [refactor] remove types_test
levy5307 commented on a change in pull request #8289: URL: https://github.com/apache/incubator-doris/pull/8289#discussion_r817329249 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarType.java ## @@ -69,6 +66,12 @@ // Keep consistent with backend ColumnType::CHAR_INLINE_LENGTH public static final int CHAR_INLINE_LENGTH = 128; +// Max length of String types, in be storage layer store string length +// using int32, the max length is 2GB, the first 4 bytes store the length +// so the max available length is 2GB - 4 + +public static final int MAX_STRING_LENGTH = 0x7fff - 4; Review comment: I think `public static final int MAX_STRING_LENGTH = 2 << 30 - 1 - 4;` is more clear here. And you should explain why did you minus 1 here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] levy5307 commented on a change in pull request #8289: [refactor] remove types_test
levy5307 commented on a change in pull request #8289: URL: https://github.com/apache/incubator-doris/pull/8289#discussion_r817329249 ## File path: fe/fe-core/src/main/java/org/apache/doris/catalog/ScalarType.java ## @@ -69,6 +66,12 @@ // Keep consistent with backend ColumnType::CHAR_INLINE_LENGTH public static final int CHAR_INLINE_LENGTH = 128; +// Max length of String types, in be storage layer store string length +// using int32, the max length is 2GB, the first 4 bytes store the length +// so the max available length is 2GB - 4 + +public static final int MAX_STRING_LENGTH = 0x7fff - 4; Review comment: I think `public static final int MAX_STRING_LENGTH = 2 << 30 - 1 - 4;` is more clear. And you should explain why did you minus 1 here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] levy5307 commented on a change in pull request #8259: [Feature] Support Changing the bucketing mode of the table from Hash Distribution to Random Distribution
levy5307 commented on a change in pull request #8259:
URL: https://github.com/apache/incubator-doris/pull/8259#discussion_r817364339
##
File path:
fe/fe-core/src/main/java/org/apache/doris/catalog/HashDistributionInfo.java
##
@@ -146,4 +146,9 @@ public String toString() {
return builder.toString();
}
+
+public RandomDistributionInfo toRandomDistributionInfo() {
+RandomDistributionInfo randomDistributionInfo = new
RandomDistributionInfo(bucketNum);
Review comment:
```
RandomDistributionInfo randomDistributionInfo = new
RandomDistributionInfo(bucketNum);
return randomDistributionInfo;
```
==>
```
return new RandomDistributionInfo(bucketNum);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8123: [improvement] optimize compaction too slow replica process, will set to ok if the replica compaction is done
morningman commented on a change in pull request #8123:
URL: https://github.com/apache/incubator-doris/pull/8123#discussion_r817368650
##
File path: fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
##
@@ -479,6 +479,29 @@ public int getTabletOrderIdx() {
return tabletOrderIdx;
}
+public boolean compactionRecovered() {
+Replica chosenReplica = null;
+long maxVersionCount = -1;
+long minVersionCount = Integer.MAX_VALUE;
+for (Replica replica : tablet.getReplicas()) {
+if (replica.getVersionCount() > maxVersionCount) {
+maxVersionCount = replica.getVersionCount();
+chosenReplica = replica;
+}
+if (replica.getVersionCount() < minVersionCount) {
+minVersionCount = replica.getVersionCount();
+}
+}
+boolean recoverd = false;
+for (Replica replica : tablet.getReplicas()) {
+if (replica.isBad() && replica.tooSlow() &&
!chosenReplica.equals(replica)) {
Review comment:
`replica.isBad()` can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8123: [improvement] optimize compaction too slow replica process, will set to ok if the replica compaction is done
morningman commented on a change in pull request #8123:
URL: https://github.com/apache/incubator-doris/pull/8123#discussion_r817370825
##
File path: fe/fe-core/src/main/java/org/apache/doris/clone/TabletScheduler.java
##
@@ -1023,17 +1042,15 @@ private boolean handleReplicaTooSlow(TabletSchedCtx
tabletCtx) throws SchedExcep
}
}
-if (chosenReplica != null && !chosenReplica.equals(minReplica) &&
minReplica.isAlive()) {
-try {
-
Catalog.getCurrentCatalog().setReplicaStatus(tabletCtx.getTabletId(),
chosenReplica.getBackendId(),
-Replica.ReplicaStatus.BAD);
-throw new SchedException(Status.FINISHED, "set slow replica as
bad");
-} catch (MetaNotFoundException e) {
-LOG.warn("set slow replica bad failed:", e);
-return false;
-}
+if (chosenReplica != null && !chosenReplica.equals(minReplica) &&
minReplica.isAlive()
+&& aliveReplicaCount >= (replicaCount / 2 + 1)) {
Review comment:
If there the replica num is 2, then we can not handle this situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] levy5307 commented on a change in pull request #8259: [Feature] Support Changing the bucketing mode of the table from Hash Distribution to Random Distribution
levy5307 commented on a change in pull request #8259:
URL: https://github.com/apache/incubator-doris/pull/8259#discussion_r817378136
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##
@@ -7043,12 +7036,12 @@ public void replayBackendTabletsInfo(BackendTabletsInfo
backendTabletsInfo) {
public void convertDistributionType(Database db, OlapTable tbl) throws
DdlException {
tbl.writeLockOrDdlException();
try {
-if (!tbl.convertRandomDistributionToHashDistribution()) {
-throw new DdlException("Table " + tbl.getName() + " is not
random distributed");
+if (!tbl.convertHashDistributionToRandomDistribution()) {
Review comment:
Could you add some unit tests for your changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8237: [feature-wip][array-type] Refactor the type info for nested array.
morningman closed issue #8237: URL: https://github.com/apache/incubator-doris/issues/8237 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8279: [feature-wip][array-type] Refactor type info for nested array.
morningman merged pull request #8279: URL: https://github.com/apache/incubator-doris/pull/8279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8240: [feature-wip][array-type] Refactor type info for nested array.
morningman commented on pull request #8240: URL: https://github.com/apache/incubator-doris/pull/8240#issuecomment-1056344993 close this PR, #8279 will be cherry-picked to array-type branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #8240: [feature-wip][array-type] Refactor type info for nested array.
morningman closed pull request #8240: URL: https://github.com/apache/incubator-doris/pull/8240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (b241bc4 -> b40e914)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from b241bc4 [fix][Vectorized] Fix exchange node merge sort null first order wrong (#8291) add b40e914 [feature-wip][array-type] Refactor type info for nested array. (#8279) No new revisions were added by this update. Summary of changes: be/src/olap/aggregate_func.h | 4 +- be/src/olap/column_block.h | 2 +- be/src/olap/column_vector.cpp | 10 ++-- be/src/olap/column_vector.h| 12 ++--- be/src/olap/field.h| 4 +- be/src/olap/push_handler.cpp | 2 +- be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 2 +- .../olap/rowset/segment_v2/bitmap_index_reader.h | 4 +- .../olap/rowset/segment_v2/bitmap_index_writer.cpp | 12 ++--- .../olap/rowset/segment_v2/bitmap_index_writer.h | 2 +- .../rowset/segment_v2/bloom_filter_index_reader.h | 4 +- .../segment_v2/bloom_filter_index_writer.cpp | 8 +-- .../rowset/segment_v2/bloom_filter_index_writer.h | 2 +- be/src/olap/rowset/segment_v2/column_reader.cpp| 4 +- be/src/olap/rowset/segment_v2/column_reader.h | 7 +-- be/src/olap/rowset/segment_v2/column_writer.cpp| 2 +- .../rowset/segment_v2/indexed_column_reader.cpp| 2 +- .../olap/rowset/segment_v2/indexed_column_reader.h | 4 +- .../rowset/segment_v2/indexed_column_writer.cpp| 4 +- .../olap/rowset/segment_v2/indexed_column_writer.h | 4 +- be/src/olap/rowset/segment_v2/segment.cpp | 2 +- be/src/olap/rowset/segment_v2/zone_map_index.cpp | 4 +- be/src/olap/schema_change.cpp | 2 +- be/src/olap/types.cpp | 57 ++ be/src/olap/types.h| 16 +++--- be/test/olap/column_vector_test.cpp| 11 ++--- .../rowset/segment_v2/binary_dict_page_test.cpp| 4 +- .../rowset/segment_v2/binary_prefix_page_test.cpp | 2 +- .../olap/rowset/segment_v2/bitmap_index_test.cpp | 2 +- .../bloom_filter_index_reader_writer_test.cpp | 2 +- .../segment_v2/column_reader_writer_test.cpp | 8 +-- .../olap/rowset/segment_v2/encoding_info_test.cpp | 4 +- be/test/olap/schema_change_test.cpp| 42 be/test/olap/storage_types_test.cpp| 7 ++- be/test/tools/benchmark_tool.cpp | 2 +- 35 files changed, 138 insertions(+), 122 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch array-type updated (d256206 -> 04ad3d2)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch array-type in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. omit d256206 [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089) omit 37becad [feature-wip](array-type) Create table with nested array type. (#8003) add 2b9b0fc [Fix] Function percentile input null return null (#8238) add 940efc6 [Fix]Remove duplicated destructor function in MinMaxFuncBase (#8287) add 5cc62fa [docs] fix document date-time-functions typo (#8053) add 236105d [feature][show-transaction] Support view transactions info for specified status by `SHOW TRANSACTION` stmt (#8156) add d5b6428 [improvement] Upgrade MySQL version to 5.7.37 to reduce unnecessary CVE issues (#8247) add 315bfe2 Revert "[chore](dependency) upgrade-grpc-version (#8218)" (#8250) add f41316a [fix](fold-constant)(hive) fix constant-folding in order by and optimize logs in hive client (#8268) add 77b21fb [chore] make options of build.sh and run-be-ut.sh work (#8271) add efe88be [docs] Fix a typo in downloads.md (#8274) add b241bc4 [fix][Vectorized] Fix exchange node merge sort null first order wrong (#8291) add b40e914 [feature-wip][array-type] Refactor type info for nested array. (#8279) new ac6048a [feature-wip](array-type) Create table with nested array type. (#8003) new 04ad3d2 [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d256206) \ N -- N -- N refs/heads/array-type (04ad3d2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/exec/mysql_scanner.cpp | 1 - be/src/exprs/minmax_predicate.h| 3 +- be/src/olap/aggregate_func.h | 4 +- be/src/olap/column_block.h | 2 +- be/src/olap/column_vector.cpp | 10 ++-- be/src/olap/column_vector.h| 12 ++--- be/src/olap/field.h| 4 +- be/src/olap/push_handler.cpp | 2 +- be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 2 +- .../olap/rowset/segment_v2/bitmap_index_reader.h | 4 +- .../olap/rowset/segment_v2/bitmap_index_writer.cpp | 12 ++--- .../olap/rowset/segment_v2/bitmap_index_writer.h | 2 +- .../rowset/segment_v2/bloom_filter_index_reader.h | 4 +- .../segment_v2/bloom_filter_index_writer.cpp | 8 +-- .../rowset/segment_v2/bloom_filter_index_writer.h | 2 +- be/src/olap/rowset/segment_v2/column_reader.cpp| 4 +- be/src/olap/rowset/segment_v2/column_reader.h | 7 +-- be/src/olap/rowset/segment_v2/column_writer.cpp| 2 +- .../rowset/segment_v2/indexed_column_reader.cpp| 2 +- .../olap/rowset/segment_v2/indexed_column_reader.h | 4 +- .../rowset/segment_v2/indexed_column_writer.cpp| 4 +- .../olap/rowset/segment_v2/indexed_column_writer.h | 4 +- be/src/olap/rowset/segment_v2/segment.cpp | 2 +- be/src/olap/rowset/segment_v2/zone_map_index.cpp | 4 +- be/src/olap/schema_change.cpp | 2 +- be/src/olap/types.cpp | 57 ++ be/src/olap/types.h| 16 +++--- be/src/util/counts.h | 2 +- be/src/vec/core/sort_cursor.h | 2 +- be/src/vec/functions/function_json.cpp | 8 +-- be/src/vec/functions/function_string.cpp | 18 +++ be/src/vec/functions/function_timestamp.cpp| 9 +++- be/src/vec/functions/function_totype.h | 4 +- be/src/vec/runtime/vsorted_run_merger.cpp | 2 +- be/test/olap/column_vector_test.cpp| 11 ++--- .../rowset/segment_v2/binary_dict_page_test.cpp| 4 +- .../rowset/segment_v2/binary_prefix_page_test.cpp | 2 +- .../olap/rowset/segment_v2/bitmap_index_test.cpp | 2 +- .../bloom_filter_index_reader_writer_test.cpp | 2 +- .../segment_v2/column_reader_writer_test.cpp | 8 +-- .../olap/rowset/segment_v2/encoding_info_test.cpp | 4 +- be/t
[incubator-doris] 01/02: [feature-wip](array-type) Create table with nested array type. (#8003)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit ac6048ad436b951f20d04751a6d817f2466ead66
Author: Adonis Ling
AuthorDate: Wed Feb 16 12:44:30 2022 +0800
[feature-wip](array-type) Create table with nested array type. (#8003)
```
create table array_type_table(k1 INT, k2 Array>) duplicate key
(k1)
distributed by hash(k1) buckets 1 properties('replication_num' = '1');
```
---
be/src/olap/tablet_meta.cpp | 9 -
.../org/apache/doris/analysis/CreateTableStmt.java | 16
.../main/java/org/apache/doris/analysis/TypeDef.java | 10 --
.../java/org/apache/doris/catalog/ArrayType.java | 9 +
.../main/java/org/apache/doris/catalog/Column.java | 10 --
.../src/main/java/org/apache/doris/catalog/Type.java | 20 +++-
.../org/apache/doris/catalog/CreateTableTest.java| 13 +
7 files changed, 41 insertions(+), 46 deletions(-)
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index 6e6195d..73632a1 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -128,11 +128,6 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t
partition_id, int64_t tablet_id
}
}
}
-
-if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
-ColumnPB* children_column = column->add_children_columns();
-_init_column_from_tcolumn(0, tcolumn.children_column[0],
children_column);
-}
}
schema->set_next_column_unique_id(next_unique_id);
@@ -210,6 +205,10 @@ void TabletMeta::_init_column_from_tcolumn(uint32_t
unique_id, const TColumn& tc
if (tcolumn.__isset.is_bloom_filter_column) {
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
}
+if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
+ColumnPB* children_column = column->add_children_columns();
+_init_column_from_tcolumn(0, tcolumn.children_column[0],
children_column);
+}
}
OLAPStatus TabletMeta::create_from_file(const string& file_path) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index ed2b689..5fb937e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -17,8 +17,11 @@
package org.apache.doris.analysis;
+import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import org.apache.commons.collections.CollectionUtils;
import org.apache.doris.catalog.AggregateType;
-import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Index;
@@ -35,12 +38,6 @@ import org.apache.doris.common.util.PrintableMap;
import org.apache.doris.external.elasticsearch.EsUtil;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
-
-import com.google.common.base.Strings;
-import com.google.common.collect.Lists;
-import com.google.common.collect.Sets;
-
-import org.apache.commons.collections.CollectionUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -372,11 +369,6 @@ public class CreateTableStmt extends DdlStmt {
columnDef.analyze(engineName.equals("olap"));
if (columnDef.getType().isArrayType()) {
-ArrayType tp = (ArrayType) columnDef.getType();
-if (!tp.getItemType().getPrimitiveType().isIntegerType() &&
-!tp.getItemType().getPrimitiveType().isCharFamily()) {
-throw new AnalysisException("Array column just support
INT/VARCHAR sub-type");
-}
if (columnDef.getAggregateType() != null &&
columnDef.getAggregateType() != AggregateType.NONE) {
throw new AnalysisException("Array column can't support
aggregation " + columnDef.getAggregateType());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
index 6bce9b5..d81d633 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
@@ -92,8 +92,10 @@ public class TypeDef implements ParseNode {
throw new AnalysisException("Unsupported data type: " + type.toSql());
}
if (type.isArrayType()) {
-ScalarType itemType = (ScalarType) ((ArrayType) type).getItemType();
-analyzeNestedType(itemType);
+Type itemType = ((ArrayType) type).getItem
[incubator-doris] 02/02: [feature-wip][array-type] Support ArrayLiteral in SQL. (#8089)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch array-type
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 04ad3d2a57b1042b78935a3906a444ec7824a32a
Author: Adonis Ling
AuthorDate: Fri Feb 18 10:59:57 2022 +0800
[feature-wip][array-type] Support ArrayLiteral in SQL. (#8089)
Please refer to #8074
---
be/src/olap/column_vector.cpp | 21 ++--
be/src/olap/rowset/segment_v2/column_reader.cpp| 8 +-
be/src/olap/rowset/segment_v2/column_reader.h | 13 +++
be/src/olap/rowset/segment_v2/column_writer.cpp| 10 +-
be/src/olap/rowset/segment_v2/column_writer.h | 5 +-
be/src/runtime/collection_value.cpp| 16 ++-
be/src/runtime/mysql_result_writer.cpp | 6 +-
be/src/runtime/raw_value.cpp | 1 +
be/src/runtime/row_batch.cpp | 10 +-
be/src/runtime/tuple.cpp | 29 +++---
.../segment_v2/column_reader_writer_test.cpp | 24 +
fe/fe-core/src/main/cup/sql_parser.cup | 14 +++
.../org/apache/doris/analysis/ArrayLiteral.java| 28 +++--
.../org/apache/doris/analysis/CreateTableStmt.java | 10 +-
.../main/java/org/apache/doris/analysis/Expr.java | 3 +-
.../java/org/apache/doris/catalog/ArrayType.java | 4 +
.../main/java/org/apache/doris/catalog/Column.java | 13 ++-
.../main/java/org/apache/doris/catalog/Type.java | 4 +-
.../org/apache/doris/analysis/ColumnDefTest.java | 18 +++-
.../apache/doris/analysis/InsertArrayStmtTest.java | 114 +
.../org/apache/doris/utframe/UtFrameUtils.java | 9 ++
21 files changed, 306 insertions(+), 54 deletions(-)
diff --git a/be/src/olap/column_vector.cpp b/be/src/olap/column_vector.cpp
index 3340628..0237b62 100644
--- a/be/src/olap/column_vector.cpp
+++ b/be/src/olap/column_vector.cpp
@@ -210,14 +210,19 @@ void ArrayColumnVectorBatch::prepare_for_read(size_t
start_idx, size_t size, boo
DCHECK(start_idx + size <= capacity());
for (size_t i = 0; i < size; ++i) {
if (!is_null_at(start_idx + i)) {
-_data[start_idx + i] = CollectionValue(
-
_elements->mutable_cell_ptr(*(_offsets->scalar_cell_ptr(start_idx + i))),
-*(_offsets->scalar_cell_ptr(start_idx + i + 1)) -
-*(_offsets->scalar_cell_ptr(start_idx + i)),
-item_has_null,
-_elements->is_nullable() ?
const_cast(&_elements->null_signs()[*(
-
_offsets->scalar_cell_ptr(start_idx + i))])
- : nullptr);
+auto next_offset = *(_offsets->scalar_cell_ptr(start_idx + i + 1));
+auto offset = *(_offsets->scalar_cell_ptr(start_idx + i));
+uint32_t length = next_offset - offset;
+if (length == 0) {
+_data[start_idx + i] = CollectionValue(length);
+} else {
+_data[start_idx + i] = CollectionValue(
+_elements->mutable_cell_ptr(offset),
+length,
+item_has_null,
+_elements->is_nullable() ?
const_cast(&_elements->null_signs()[offset])
+ : nullptr);
+}
}
}
}
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp
b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 20d2918..9530d80 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -127,7 +127,7 @@ Status ColumnReader::init() {
"Bad file $0: invalid column index type $1",
_path_desc.filepath, index_meta.type()));
}
}
-if (_ordinal_index_meta == nullptr) {
+if (!is_empty() && _ordinal_index_meta == nullptr) {
return Status::Corruption(strings::Substitute(
"Bad file $0: missing ordinal index for column $1",
_path_desc.filepath, _meta.column_id()));
}
@@ -339,6 +339,10 @@ Status ColumnReader::seek_at_or_before(ordinal_t ordinal,
OrdinalPageIndexIterat
}
Status ColumnReader::new_iterator(ColumnIterator** iterator) {
+if (is_empty()) {
+*iterator = new EmptyFileColumnIterator();
+return Status::OK();
+}
if (is_scalar_type((FieldType)_meta.type())) {
*iterator = new FileColumnIterator(this);
return Status::OK();
@@ -427,7 +431,7 @@ Status ArrayFileColumnIterator::next_batch(size_t* n,
ColumnBlockView* dst, bool
// read item
size_t item_size = array_batch->get_item_size(dst->current_offset(), *n);
-if (item_size > 0) {
+if (item_size >= 0) {
bool item_has_null = false;
ColumnVectorBatch* item_vector_batch = array_batch->elements();
diff --git a/be/src/olap/rowset/segment
[GitHub] [incubator-doris] zuochunwei opened a new pull request #8292: (improving) refactor column::insert_many_dict_data
zuochunwei opened a new pull request #8292: URL: https://github.com/apache/incubator-doris/pull/8292 # Proposed changes i have tested this changes by adding global atomic timer: clock_gettime(CLOCK_MONOTONIC, &ts); before vs after modifying, test sql ssb Q2.1 SELECT sum(lo_revenue), year(lo_orderdate) AS year, p_brand FROM lineorder_flat WHERE p_category = 'MFGR#12' AND s_region = 'AMERICA' GROUP BY year, p_brand ORDER BY year, p_brand; result: PredicateColumnType::insert_many_dict_data() 11970424864 vs 8907241614 ColumnString::insert_many_dict_data 3044568165 vs 2467506128 Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8292: (improving) refactor column::insert_many_dict_data
spaces-X commented on a change in pull request #8292: URL: https://github.com/apache/incubator-doris/pull/8292#discussion_r817407427 ## File path: be/src/vec/columns/column_string.h ## @@ -32,6 +32,8 @@ #include "vec/common/sip_hash.h" #include "vec/core/field.h" +#include "../../util/mem_util.hpp" Review comment: Maybe it's better to use an absolute path? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on pull request #8292: (improving) refactor column::insert_many_dict_data
wangbo commented on pull request #8292: URL: https://github.com/apache/incubator-doris/pull/8292#issuecomment-1056449402 Please add ```BlockLoadTime``` Timer for performance improvement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangbo edited a comment on pull request #8292: (improving) refactor column::insert_many_dict_data
wangbo edited a comment on pull request #8292: URL: https://github.com/apache/incubator-doris/pull/8292#issuecomment-1056449402 Please add ```BlockLoadTime``` Timer for performance improvement. I don't know what does ```11970424864 vs 8907241614``` means. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #8293: [fix](fe-ut) Fix FE unit test
morningman opened a new pull request #8293: URL: https://github.com/apache/incubator-doris/pull/8293 # Proposed changes Issue Number: close #xxx ## Problem Summary: Fix following ut: 1. GlobalTransactionMgrTest 2. BackupJobTest 3. ReplicaTest 4. SparkLoadJobTest Also remove old FE Meta version ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gensiss opened a new issue #8294: [Feature] StarRocks都有可视化部署了，我们能不能搞一下
Gensiss opened a new issue #8294: URL: https://github.com/apache/incubator-doris/issues/8294 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description _No response_ ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
