[GitHub] [incubator-doris] wayofeng opened a new pull request #7982: Translate the Chinese comments
wayofeng opened a new pull request #7982: URL: https://github.com/apache/incubator-doris/pull/7982 # Proposed changes Translate the Chinese comments of file /be/src/common/config.h ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aubdiy closed issue #7891: [Bug]
aubdiy closed issue #7891: URL: https://github.com/apache/incubator-doris/issues/7891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aubdiy commented on issue #7891: [Bug]
aubdiy commented on issue #7891: URL: https://github.com/apache/incubator-doris/issues/7891#issuecomment-1032378045 user docker compile, is ok . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7891: [Bug]
morningman commented on issue #7891: URL: https://github.com/apache/incubator-doris/issues/7891#issuecomment-1032380515 you can simply remove the `/data/sae-dp-files/tmp/apache-doris-0.15.0-incubating-src/thirdparty/installed/bin/lz4c` before executing `build-thirdparty.sh` again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aubdiy opened a new issue #7983: [Bug] Broker Load 从 hdfs3.3.1 导入数据到 doris 中, 日志提示Could not initialize class org.apache.hadoop.util.StringUtils
aubdiy opened a new issue #7983:
URL: https://github.com/apache/incubator-doris/issues/7983
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
0.15.0
### What's Wrong?
Broker Load 从 hdfs3.3.1 导入数据到 doris 中, 日志提示
```
receive a ping request, request detail:
TBrokerPingBrokerRequest(version:VERSION_ONE, clientId:172.31.3.146)
[INFO ] 2022-02-08 08:07:10,947
method:org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.listPath(HDFSBrokerServiceImpl.java:67)
received a list path request, request detail:
TBrokerListPathRequest(version:VERSION_ONE,
path:hdfs://test.internal:8020/tmp/data.csv, isRecursive:false,
properties:{_DORIS_STORAGE_TYPE_=BROKER})
[INFO ] 2022-02-08 08:07:10,948
method:org.apache.doris.broker.hdfs.FileSystemManager.getDistributedFileSystem(FileSystemManager.java:244)
create file system for new path: hdfs://test.internal:8020/tmp/data.csv
Exception in thread "pool-2-thread-13" java.lang.NoClassDefFoundError: Could
not initialize class org.apache.hadoop.util.StringUtils
at
org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
at
org.apache.doris.broker.hdfs.FileSystemManager.getDistributedFileSystem(FileSystemManager.java:360)
at
org.apache.doris.broker.hdfs.FileSystemManager.getFileSystem(FileSystemManager.java:152)
at
org.apache.doris.broker.hdfs.FileSystemManager.listPath(FileSystemManager.java:427)
at
org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.listPath(HDFSBrokerServiceImpl.java:74)
at
org.apache.doris.thrift.TPaloBrokerService$Processor$listPath.getResult(TPaloBrokerService.java:815)
at
org.apache.doris.thrift.TPaloBrokerService$Processor$listPath.getResult(TPaloBrokerService.java:795)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[INFO ] 2022-02-08 08:07:10,963
method:org.apache.doris.broker.hdfs.HDFSBrokerServiceImpl.listPath(HDFSBrokerServiceImpl.java:67)
received a list path request, request detail:
TBrokerListPathRequest(version:VERSION_ONE,
path:hdfs://test.internal:8020/tmp/data.csv, isRecursive:false,
properties:{_DORIS_STORAGE_TYPE_=BROKER})
```
### What You Expected?
希望能够正常导入
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aubdiy commented on issue #7983: [Bug] Broker Load 从 hdfs3.3.1 导入数据到 doris 中, 日志提示Could not initialize class org.apache.hadoop.util.StringUtils
aubdiy commented on issue #7983:
URL:
https://github.com/apache/incubator-doris/issues/7983#issuecomment-1032392489
问题已经修复.
修复过程:
1. 进入 fs_brokers/apache_hdfs_broker
2. 执行编译, 编译成功后将 `output/apache_hdfs_broker/lib/apache_hdfs_broker.jar` 保留代用,
然后删除掉 `output` 目录
3. 修改 pom.xml 文件中 hadoop 相关依赖, 将 2.7.3 改为 3.3.1
4. 修改 pom.xml 文件, 删除 hadoop-hdfs 包, 增加 hadoop-hdfs-client 包(3.3.1) 版本
5. 修改 pom.xml 文件中 hadoop-aws 相关依赖, 将 2.7.3 改为 3.3.1
6. 修改
`fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java:[636,15]`
代码, 将 ' } catch (IOException e) {' 替换为 ' } catch (Exception e) {' , 否则编译会提示
```
[ERROR]
/root/incubator-doris-DORIS-0.15.0-release/apache-doris-0.15.0-incubating-src/fs_brokers/apache_hdfs_broker/src/main/java/org/apache/doris/broker/hdfs/FileSystemManager.java:[636,15]
exception java.io.IOException is never thrown in body of corresponding try
statement
```
7. 执行编译 sh build.sh
8. 编译成功后, 将第2 步中的 `apache_hdfs_broker.jar` 覆盖
`output/apache_hdfs_broker/lib/apache_hdfs_broker.jar`
9. 重新部署 hdf_broker , 可以正常导入
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary opened a new pull request #7984: [Docs] add rpc function document
stalary opened a new pull request #7984: URL: https://github.com/apache/incubator-doris/pull/7984 ## Problem Summary: add rpc function document ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on pull request #7984: [Docs] add rpc function document
stalary commented on pull request #7984: URL: https://github.com/apache/incubator-doris/pull/7984#issuecomment-1032422036 At present, I have only written the Chinese document, and the English document will be supplemented after review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] stalary commented on pull request #7984: [Docs] add rpc function document
stalary commented on pull request #7984: URL: https://github.com/apache/incubator-doris/pull/7984#issuecomment-1032424543 TBR @yangzhg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] dataroaring opened a new pull request #7985: [doc] remove useless word 'To' in materialized view
dataroaring opened a new pull request #7985: URL: https://github.com/apache/incubator-doris/pull/7985 # Remove an useless word in doc Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (Yes) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 opened a new issue #7986: Apache Doris Manager 2022 Roadmap
hf200012 opened a new issue #7986: URL: https://github.com/apache/incubator-doris/issues/7986 - Doris stock cluster takeover - Existing cluster takeover (implementing takeover of installed clusters, including cluster configuration modification, startup, monitoring, etc.) - Doris visual upgrade support - Cluster node list management - Cluster overview view: such as node book, FE, BE, etc., running time, resource usage, number of running Load jobs, etc. - The current cluster list UI is optimized, refer to Ambari, change it to the machine list, and display the roles by tab (FE, BE, Broker) - You can click on the node to view the information of this node, including the relevant information of the node machine, installed services, node configuration information (BE, FE), node monitoring information (memory, disk, CPU, network, load, etc.) - On a specific node, the services on this node can be started and stopped, etc. - Can visualize elastic scaling clusters - Doris plugin support 1. Automatic installation and deployment of the audit log plug-in, realize the automatic installation of the audit log plug-in, and modify the configuration information of the plug-in at the same time. 2. Create a plug-in database, create a data table, and present the installation information of the plug-in, mainly the database and table information where the plug-in collects audit logs. 3. Give the query and display page of the audit log - Gradually weaken the current Doris web UI - Management of Doris cluster users, roles and permissions can be achieved through Doris Manager - Provides a SQL development console with associative input - Provides a visual display of SQL Profile, similar to HUE, which helps users analyze SQL performance - The SQL console supports saving queries for subsequent reuse after use, saving query history, etc. - The above function is also applicable to the analysis of the sql list on the current Web UI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #7987: [Feature] [Vectorized] support locate function
BiteThet opened a new issue #7987:
URL: https://github.com/apache/incubator-doris/issues/7987
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Description
support function locate in vectorized engine.
### Use case
MySQL [(none)]> SELECT LOCATE('', '', 1);
+---+
| locate('', '', 1) |
+---+
| 1 |
+---+
1 row in set (0.613 sec)
MySQL [(none)]> SELECT LOCATE('bar', 'foobarbar', 5);
+---+
| locate('bar', 'foobarbar', 5) |
+---+
| 7 |
+---+
1 row in set (0.015 sec)
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #7988: [Feature] [Vectorized] support locate function
BiteThet opened a new pull request #7988: URL: https://github.com/apache/incubator-doris/pull/7988 # Proposed changes Issue Number: close #7987 @HappenLee ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #7988: [Feature] [Vectorized] support locate function
HappenLee commented on a change in pull request #7988:
URL: https://github.com/apache/incubator-doris/pull/7988#discussion_r801532798
##
File path: be/src/vec/functions/function_string.h
##
@@ -1142,4 +1154,88 @@ struct MoneyFormatDecimalImpl {
}
};
+class FunctionStringLocatePos : public IFunction {
+public:
+static constexpr auto name = "locate";
+static FunctionPtr create() { return
std::make_shared(); }
+String get_name() const override { return name; }
+size_t get_number_of_arguments() const override { return 3; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+return std::make_shared();
+}
+
+DataTypes get_variadic_argument_types() {
+return {std::make_shared(),
std::make_shared(),
+std::make_shared()};
+}
+
+bool use_default_implementation_for_constants() const override { return
true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+auto col_substr =
+
block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+auto col_str =
+
block.get_by_position(arguments[1]).column->convert_to_full_column_if_const();
+auto col_pos =
+
block.get_by_position(arguments[2]).column->convert_to_full_column_if_const();
+
+ColumnInt32::MutablePtr col_res = ColumnInt32::create();
+
+auto& vec_pos = reinterpret_cast(col_pos.get())->get_data();
+auto& vec_res = col_res->get_data();
+vec_res.resize(input_rows_count);
+
+for (int i = 0; i < input_rows_count; ++i) {
+vec_res[i] = locate_pos(col_substr->get_data_at(i).to_string_val(),
+col_str->get_data_at(i).to_string_val(),
vec_pos[i]);
+}
+
+block.replace_by_position(result, std::move(col_res));
+return Status::OK();
+}
+
+private:
+int locate_pos(StringVal substr, StringVal str, int start_pos) {
+if (substr.len == 0) {
+if (start_pos <= 0) {
+return 0;
+} else if (start_pos == 1) {
+return 1;
+} else if (start_pos > str.len) {
+return 0;
+} else {
+return start_pos;
+}
+}
+// Hive returns 0 for *start_pos <= 0,
+// but throws an exception for *start_pos > str->len.
+// Since returning 0 seems to be Hive's error condition, return 0.
+std::vector index;
+size_t char_len = get_char_len(str, &index);
+if (start_pos > char_len) {
+return 0;
+}
+StringValue substr_sv = StringValue::from_string_val(substr);
+StringSearch search(&substr_sv);
+// Input start_pos starts from 1.
+StringValue adjusted_str(reinterpret_cast(str.ptr) +
index[start_pos - 1],
+ str.len - index[start_pos - 1]);
+int32_t match_pos = search.search(&adjusted_str);
+if (match_pos >= 0) {
+// Hive returns the position in the original string starting from
1.
+size_t char_len = 0;
Review comment:
abtruct the function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #7989: [fix] (grouping set) fix Unexpected exception: bitIndex < 0: -1
yangzhg opened a new pull request #7989: URL: https://github.com/apache/incubator-doris/pull/7989 # Proposed changes Issue Number: close #7971 ## Problem Summary: Add check if grouping function column in all in group by list ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6806: Release note 0.15.0
morningman closed issue #6806: URL: https://github.com/apache/incubator-doris/issues/6806 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7985: [doc] remove useless word 'To' in materialized view
github-actions[bot] commented on pull request #7985: URL: https://github.com/apache/incubator-doris/pull/7985#issuecomment-1032759334 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7982: Translate the Chinese comments
github-actions[bot] commented on pull request #7982: URL: https://github.com/apache/incubator-doris/pull/7982#issuecomment-1032763553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7955: [docs]Added http interface return example to obtain the specified table str…
github-actions[bot] commented on pull request #7955: URL: https://github.com/apache/incubator-doris/pull/7955#issuecomment-1032765183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch array-type created (now 3048ce8)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch array-type in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. at 3048ce8 [improvement][refactor](vec) Refactor serde of vec block and using brpc attachment (#7939) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Wahno opened a new issue #7990: [Bug] Spark connector not support STRING type
Wahno opened a new issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache doris 0.15.0 ### What's Wrong? When the STRING type is in the doris table structure, using the spark connector to import data will throw an exception: Unsupported type STRING. ### What You Expected? Support STRING type ### How to Reproduce? Create a table containing a STRING type, and use the Spark connector to import data to this table. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #7981: [feature-wip](iceberg) Step2: add table creation strict mode and support refresh iceberg table or db.
morningman commented on a change in pull request #7981:
URL: https://github.com/apache/incubator-doris/pull/7981#discussion_r802258821
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
##
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.CreateDbStmt;
+import org.apache.doris.analysis.CreateTableStmt;
+import org.apache.doris.analysis.DropDbStmt;
+import org.apache.doris.analysis.DropTableStmt;
+import org.apache.doris.analysis.RefreshDbStmt;
+import org.apache.doris.analysis.RefreshTableStmt;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.UserException;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Map;
+
+// Manager for refresh database and table action
+public class RefreshManager {
+private static final Logger LOG =
LogManager.getLogger(RefreshManager.class);
+
+public void handleRefreshTable(RefreshTableStmt stmt) throws UserException
{
+String dbName = stmt.getDbName();
+String tableName = stmt.getTblName();
+Catalog catalog = Catalog.getCurrentCatalog();
+
+// 0. check table type
+Database db = catalog.getDbOrDdlException(dbName);
+Table table = db.getTableNullable(tableName);
+if (!(table instanceof IcebergTable)) {
+throw new DdlException("Only support refresh Iceberg table.");
+}
+
+// 1. get iceberg properties
+Map icebergProperties = ((IcebergTable)
table).getIcebergProperties();
+icebergProperties.put(IcebergProperty.ICEBERG_TABLE, ((IcebergTable)
table).getIcebergTbl());
+icebergProperties.put(IcebergProperty.ICEBERG_DATABASE,
((IcebergTable) table).getIcebergDb());
+
+// 2. drop old table
+DropTableStmt dropTableStmt = new DropTableStmt(true,
stmt.getTableName(), true);
+catalog.dropTable(dropTableStmt);
+
+// 3. create new table
+CreateTableStmt createTableStmt = new CreateTableStmt(true, true,
+stmt.getTableName(), "ICEBERG", icebergProperties, "");
+catalog.createTable(createTableStmt);
+
+LOG.info("Successfully refresh table: {} from db: {}", tableName,
dbName);
+}
+
+public void handleRefreshDb(RefreshDbStmt stmt) throws DdlException {
+String dbName = stmt.getDbName();
+Catalog catalog = Catalog.getCurrentCatalog();
+
+Database db = catalog.getDbOrDdlException(dbName);
+// 0. get db properties
+Map dbProperties =
db.getDbProperties().getProperties();
+// build iceberg properties
+db.getDbProperties().addAndBuildProperties(dbProperties);
+// 1. check database type
Review comment:
Why calling `addAndBuildProperties` here?
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java
##
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.analysis.CreateDbStmt;
+import org.apache.doris.analysis.CreateTableStmt;
+import org.apache.doris.analysis.DropDbStmt;
+import org.apache.doris.analysis.DropTableStmt;
+import org.apache.doris.analysis.RefreshDbStmt;
+import org.apache.doris.analysis.RefreshTableStmt;
+import org.apache.doris.common.DdlException;
+import org.apach
[GitHub] [incubator-doris] HappenLee closed issue #4394: [Proposal] Support Bucket Shuffle Join for Doris
HappenLee closed issue #4394: URL: https://github.com/apache/incubator-doris/issues/4394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7990: [Bug] Spark connector not support STRING type
hf200012 commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-109726 Which version of your connector is？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #7792: [Feature][Vectorized] support function topn && remove some unused code
morningman closed issue #7792: URL: https://github.com/apache/incubator-doris/issues/7792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7793: [Feature](Vectorization) support function topn && remove some unused code
morningman merged pull request #7793: URL: https://github.com/apache/incubator-doris/pull/7793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature](vectorization) support function topn && remove some unused code (#7793)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 0553ce2 [feature](vectorization) support function topn && remove some unused code (#7793) 0553ce2 is described below commit 0553ce294449bcc1f025e22928c154d71c1cfe5d Author: Pxl <952130...@qq.com> AuthorDate: Wed Feb 9 13:05:31 2022 +0800 [feature](vectorization) support function topn && remove some unused code (#7793) --- be/src/exprs/topn_function.cpp | 40 be/src/vec/CMakeLists.txt | 1 + .../vec/aggregate_functions/aggregate_function.h | 39 +--- .../aggregate_functions/aggregate_function_avg.h | 8 +- .../aggregate_function_bitmap.h| 18 +- .../aggregate_functions/aggregate_function_count.h | 32 ++- .../aggregate_function_distinct.h | 12 +- .../aggregate_function_hll_union_agg.h | 15 +- .../aggregate_function_min_max.h | 27 ++- .../aggregate_function_nothing.h | 4 +- .../aggregate_functions/aggregate_function_null.h | 20 +- .../aggregate_function_simple_factory.cpp | 6 +- .../aggregate_function_stddev.h| 8 +- .../aggregate_functions/aggregate_function_sum.h | 24 +-- .../aggregate_function_topn.cpp| 63 ++ .../aggregate_functions/aggregate_function_topn.h | 233 + .../aggregate_functions/aggregate_function_uniq.h | 14 +- .../aggregate_function_window.h| 10 +- be/test/exprs/topn_function_test.cpp | 98 +++-- be/test/vec/aggregate_functions/agg_test.cpp | 46 +++- .../java/org/apache/doris/catalog/FunctionSet.java | 93 +++- 21 files changed, 498 insertions(+), 313 deletions(-) diff --git a/be/src/exprs/topn_function.cpp b/be/src/exprs/topn_function.cpp index e956e5e..544f980 100644 --- a/be/src/exprs/topn_function.cpp +++ b/be/src/exprs/topn_function.cpp @@ -91,50 +91,10 @@ StringVal TopNFunctions::topn_finalize(FunctionContext* ctx, const StringVal& sr return result; } -template void TopNFunctions::topn_update(FunctionContext*, const BooleanVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const TinyIntVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const SmallIntVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const IntVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const BigIntVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const FloatVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const DoubleVal&, const IntVal&, - StringVal*); template void TopNFunctions::topn_update(FunctionContext*, const StringVal&, const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const DateTimeVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const LargeIntVal&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const DecimalV2Val&, const IntVal&, - StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const BooleanVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const TinyIntVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const SmallIntVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const IntVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const BigIntVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const FloatVal&, const IntVal&, - const IntVal&, StringVal*); -template void TopNFunctions::topn_update(FunctionContext*, const DoubleVal&, co
[incubator-doris] branch master updated: [refactor](storage) VGenericIterator to reuse Schema (#7858)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new db20e1f [refactor](storage) VGenericIterator to reuse Schema (#7858)
db20e1f is described below
commit db20e1f323744cd2fab91f120df644a4d47106be
Author: zuochunwei
AuthorDate: Wed Feb 9 13:06:03 2022 +0800
[refactor](storage) VGenericIterator to reuse Schema (#7858)
1. reuse Schema to avoid copying, because clone Schema will generate a lot
of sub Field object
2. call interface provided by Block to reduce code lines
---
be/src/olap/row_block2.h | 2 +-
be/src/olap/rowset/beta_rowset_reader.cpp | 6 +--
be/src/olap/rowset/beta_rowset_reader.h| 1 +
.../rowset/segment_v2/empty_segment_iterator.h | 2 +-
be/src/olap/rowset/segment_v2/segment_iterator.h | 3 +-
be/src/vec/olap/vgeneric_iterators.cpp | 61 --
6 files changed, 29 insertions(+), 46 deletions(-)
diff --git a/be/src/olap/row_block2.h b/be/src/olap/row_block2.h
index b98ab95..7f2b79d 100644
--- a/be/src/olap/row_block2.h
+++ b/be/src/olap/row_block2.h
@@ -111,7 +111,7 @@ public:
private:
Status _copy_data_to_column(int cid, vectorized::MutableColumnPtr&
mutable_column_ptr);
-Schema _schema;
+const Schema& _schema;
size_t _capacity;
// _column_vector_batches[cid] == null if cid is not in `_schema`.
// memory are not allocated from `_pool` because we don't wan't to
reallocate them in clear()
diff --git a/be/src/olap/rowset/beta_rowset_reader.cpp
b/be/src/olap/rowset/beta_rowset_reader.cpp
index 263a4cc..3aed8eb 100644
--- a/be/src/olap/rowset/beta_rowset_reader.cpp
+++ b/be/src/olap/rowset/beta_rowset_reader.cpp
@@ -55,7 +55,7 @@ OLAPStatus BetaRowsetReader::init(RowsetReaderContext*
read_context) {
_stats = _context->stats;
}
// SegmentIterator will load seek columns on demand
-Schema schema(_context->tablet_schema->columns(),
*(_context->return_columns));
+_schema = std::make_unique(_context->tablet_schema->columns(),
*(_context->return_columns));
// convert RowsetReaderContext to StorageReadOptions
StorageReadOptions read_options;
@@ -102,7 +102,7 @@ OLAPStatus BetaRowsetReader::init(RowsetReaderContext*
read_context) {
std::vector> seg_iterators;
for (auto& seg_ptr : _segment_cache_handle.get_segments()) {
std::unique_ptr iter;
-auto s = seg_ptr->new_iterator(schema, read_options, _parent_tracker,
&iter);
+auto s = seg_ptr->new_iterator(*_schema, read_options,
_parent_tracker, &iter);
if (!s.ok()) {
LOG(WARNING) << "failed to create iterator[" << seg_ptr->id() <<
"]: " << s.to_string();
return OLAP_ERR_ROWSET_READER_INIT;
@@ -131,7 +131,7 @@ OLAPStatus BetaRowsetReader::init(RowsetReaderContext*
read_context) {
_iterator.reset(final_iterator);
// init input block
-_input_block.reset(new RowBlockV2(schema,
+_input_block.reset(new RowBlockV2(*_schema,
std::min(1024, read_context->batch_size), _parent_tracker));
if (!read_context->is_vec) {
diff --git a/be/src/olap/rowset/beta_rowset_reader.h
b/be/src/olap/rowset/beta_rowset_reader.h
index add0c31..997ab12 100644
--- a/be/src/olap/rowset/beta_rowset_reader.h
+++ b/be/src/olap/rowset/beta_rowset_reader.h
@@ -58,6 +58,7 @@ public:
RowsetTypePB type() const override { return RowsetTypePB::BETA_ROWSET; }
private:
+std::unique_ptr _schema;
RowsetReaderContext* _context;
BetaRowsetSharedPtr _rowset;
diff --git a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
b/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
index 3e1a4f9..0c186ed 100644
--- a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
@@ -35,7 +35,7 @@ public:
Status next_batch(vectorized::Block* block) override;
private:
-Schema _schema;
+const Schema& _schema;
};
} // namespace segment_v2
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.h
b/be/src/olap/rowset/segment_v2/segment_iterator.h
index 0577526..2eae13e 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.h
@@ -103,8 +103,7 @@ private:
class BitmapRangeIterator;
std::shared_ptr _segment;
-// TODO(zc): rethink if we need copy it
-Schema _schema;
+const Schema& _schema;
// _column_iterators.size() == _schema.num_columns()
// _column_iterators[cid] == nullptr if cid is not in _schema
std::vector _column_iterators;
diff --git a/be/src/vec/olap/vgeneric_iterators.cpp
b/be/src/vec/olap/vgeneric_iterators.cpp
index 3bee64b..f0f148d 100644
--- a/be/src/vec/olap/vgeneric_iterators.cpp
+++ b/be/src/vec/olap
[GitHub] [incubator-doris] morningman merged pull request #7858: [refactor](storage) VGenericIterator to reuse Schema
morningman merged pull request #7858: URL: https://github.com/apache/incubator-doris/pull/7858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7963: [fix] fix ltrim result may incorrect in some case
morningman merged pull request #7963: URL: https://github.com/apache/incubator-doris/pull/7963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (5029ef4 -> eeaf672)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 5029ef4 [fix] fix ltrim result may incorrect in some case (#7963) add eeaf672 (fix)[lateral-view] Solve the problem of not recognizing the lateral view on the view (#7968) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/analysis/InlineViewRef.java| 7 +-- .../org/apache/doris/planner/TableFunctionPlanTest.java | 16 +++- 2 files changed, 12 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #7968: (fix)[Lateral view] Solve the problem of not recognizing the lateral view on the view
morningman merged pull request #7968: URL: https://github.com/apache/incubator-doris/pull/7968 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix] fix ltrim result may incorrect in some case (#7963)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5029ef4 [fix] fix ltrim result may incorrect in some case (#7963)
5029ef4 is described below
commit 5029ef46c92fdba10a35136fa62eb7ac2d425d54
Author: Zhengguo Yang
AuthorDate: Wed Feb 9 13:06:37 2022 +0800
[fix] fix ltrim result may incorrect in some case (#7963)
fix ltrim result may incorrect in some case
according to https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
Built-in Function: int __builtin_cl/tz (unsigned int x)
If x is 0, the result is undefined.
So we handle the case of 0 separately
this function return different between gcc and clang when x is 0
---
be/CMakeLists.txt| 3 +-
be/src/exprs/v_string_functions.h| 219 ---
be/src/vec/functions/function_string.cpp | 17 ++-
be/test/exprs/string_functions_test.cpp | 56
4 files changed, 39 insertions(+), 256 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 6bf9813..2d50c67 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -674,7 +674,6 @@ else()
${DORIS_DEPENDENCIES}
${WL_START_GROUP}
${X86_DEPENDENCIES}
-${WL_END_GROUP}
)
endif()
@@ -690,6 +689,8 @@ if (WITH_MYSQL)
)
endif()
+set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES} ${WL_END_GROUP})
+
message(STATUS "DORIS_DEPENDENCIES is ${DORIS_DEPENDENCIES}")
# Add all external dependencies. They should come after the palo libs.
diff --git a/be/src/exprs/v_string_functions.h
b/be/src/exprs/v_string_functions.h
deleted file mode 100644
index 3fd9845..000
--- a/be/src/exprs/v_string_functions.h
+++ /dev/null
@@ -1,219 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#ifndef BE_V_STRING_FUNCTIONS_H
-#define BE_V_STRING_FUNCTIONS_H
-
-#include
-#include
-#include "runtime/string_value.hpp"
-
-#ifdef __SSE2__
-#include
-#endif
-
-namespace doris {
-class VStringFunctions {
-public:
-#ifdef __SSE2__
-/// n equals to 16 chars length
-static constexpr auto REGISTER_SIZE = sizeof(__m128i);
-#endif
-public:
-static StringVal rtrim(const StringVal& str) {
-if (str.is_null || str.len == 0) {
-return str;
-}
-auto begin = 0;
-auto end = str.len - 1;
-#ifdef __SSE2__
-char blank = ' ';
-const auto pattern = _mm_set1_epi8(blank);
-while (end - begin + 1 >= REGISTER_SIZE) {
-const auto v_haystack = _mm_loadu_si128(reinterpret_cast(str.ptr + end + 1 - REGISTER_SIZE));
-const auto v_against_pattern = _mm_cmpeq_epi8(v_haystack, pattern);
-const auto mask = _mm_movemask_epi8(v_against_pattern);
-int offset = __builtin_clz(~(mask << REGISTER_SIZE));
-/// means not found
-if (offset == 0)
-{
-return StringVal(str.ptr + begin, end - begin + 1);
-} else {
-end -= offset;
-}
-}
-#endif
-while (end >= begin && str.ptr[end] == ' ') {
---end;
-}
-if (end < 0) {
-return StringVal("");
-}
-return StringVal(str.ptr + begin, end - begin + 1);
-}
-
-static StringVal ltrim(const StringVal& str) {
-if (str.is_null || str.len == 0) {
-return str;
-}
-auto begin = 0;
-auto end = str.len - 1;
-#ifdef __SSE2__
-char blank = ' ';
-const auto pattern = _mm_set1_epi8(blank);
-while (end - begin + 1 >= REGISTER_SIZE) {
-const auto v_haystack = _mm_loadu_si128(reinterpret_cast(str.ptr + begin));
-const auto v_against_pattern = _mm_cmpeq_epi8(v_haystack, pattern);
-const auto mask = _mm_movemask_epi8(v_against_pattern);
-const auto offset = __builtin_ctz(mask ^ 0x);
-/// means not found
-if (offset == 0)
-{
-return StringVal(str.ptr + b
[GitHub] [incubator-doris] morningman merged pull request #7975: [improvement](log) Add query id info in error log for easy tracking
morningman merged pull request #7975: URL: https://github.com/apache/incubator-doris/pull/7975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [improvement](log) Add query id info in error log for easy tracking (#7975)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new df2c756 [improvement](log) Add query id info in error log for easy
tracking (#7975)
df2c756 is described below
commit df2c7563b08e94801daeb4370a1627886dbc3d34
Author: Mingyu Chen
AuthorDate: Wed Feb 9 13:07:28 2022 +0800
[improvement](log) Add query id info in error log for easy tracking (#7975)
This PR #7936 change some FE log level to debug, so that when error
happens, it is not easy to find out
which SQL cause the error.
So I add stmt id and query id in error log, so that user can use these
identifiers to find SQL in fe.audit.log
---
.../java/org/apache/doris/analysis/QueryStmt.java | 78 ++
.../org/apache/doris/mysql/nio/AcceptListener.java | 3 +-
.../java/org/apache/doris/qe/ConnectContext.java | 5 ++
.../java/org/apache/doris/qe/StmtExecutor.java | 18 ++---
.../org/apache/doris/planner/QueryPlanTest.java| 36 +++---
5 files changed, 96 insertions(+), 44 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
index db0abfd..4888ab7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/QueryStmt.java
@@ -23,11 +23,14 @@ import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
import org.apache.doris.rewrite.ExprRewriter;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
+
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
+
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -177,8 +180,8 @@ public abstract class QueryStmt extends StatementBase {
private void analyzeLimit(Analyzer analyzer) throws AnalysisException {
// TODO chenhao
if (limitElement.getOffset() > 0 && !hasOrderByClause()) {
- throw new AnalysisException("OFFSET requires an ORDER BY clause: " +
- limitElement.toSql().trim());
+throw new AnalysisException("OFFSET requires an ORDER BY clause: "
+
+limitElement.toSql().trim());
}
limitElement.analyze(analyzer);
}
@@ -213,7 +216,7 @@ public abstract class QueryStmt extends StatementBase {
List tblRefs = Lists.newArrayList();
collectTableRefs(tblRefs);
-for (TableRef tblRef: tblRefs) {
+for (TableRef tblRef : tblRefs) {
if (absoluteRef == null && !tblRef.isRelative()) absoluteRef =
tblRef;
/*if (tblRef.isCorrelated()) {
*
@@ -313,19 +316,21 @@ public abstract class QueryStmt extends StatementBase {
// are ignored.
if (!hasLimit() && !hasOffset() && (!analyzer.isRootAnalyzer() ||
fromInsert)) {
evaluateOrderBy = false;
-// Return a warning that the order by was ignored.
-StringBuilder strBuilder = new StringBuilder();
-strBuilder.append("Ignoring ORDER BY clause without LIMIT or
OFFSET: ");
-strBuilder.append("ORDER BY ");
-strBuilder.append(orderByElements.get(0).toSql());
-for (int i = 1; i < orderByElements.size(); ++i) {
-strBuilder.append(", ").append(orderByElements.get(i).toSql());
+if (LOG.isDebugEnabled()) {
+// Return a warning that the order by was ignored.
+StringBuilder strBuilder = new StringBuilder();
+strBuilder.append("Ignoring ORDER BY clause without LIMIT or
OFFSET: ");
+strBuilder.append("ORDER BY ");
+strBuilder.append(orderByElements.get(0).toSql());
+for (int i = 1; i < orderByElements.size(); ++i) {
+strBuilder.append(",
").append(orderByElements.get(i).toSql());
+}
+strBuilder.append(".\nAn ORDER BY appearing in a view,
subquery, union operand, ");
+strBuilder.append("or an insert/ctas statement has no effect
on the query result ");
+strBuilder.append("unless a LIMIT and/or OFFSET is used in
conjunction ");
+strBuilder.append("with the ORDER BY.");
+LOG.debug(strBuilder.toString());
}
-strBuilder.append(".\nAn ORDER BY appearing in a view, subquery,
union operand, ");
-strBuilder.append("or an insert/ctas statement has no effect on
the query result ");
-strBuilder.append("unless a LIMIT and/or OFFSET is used in
conjunction ");
-strBuilder.append("with
[GitHub] [incubator-doris] Wahno commented on issue #7990: [Bug] Spark connector not support STRING type
Wahno commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033388261 2.3.4-2.11-1.0.0-SNAPSHOT，compiled latest code from the master branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7990: [Bug] Spark connector not support STRING type
hf200012 commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033389012 org.apache.doris doris-spark-connector 2.3.4-2.11-SNAPSHOT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 removed a comment on issue #7990: [Bug] Spark connector not support STRING type
hf200012 removed a comment on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033389012 org.apache.doris doris-spark-connector 2.3.4-2.11-SNAPSHOT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Wahno commented on issue #7990: [Bug] Spark connector not support STRING type
Wahno commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033389575 Yes. You can check getCatalystType method from SchemaUtils class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7990: [Bug] Spark connector not support STRING type
hf200012 commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033392025 > 2.3.4-2.11-1.0.0-SNAPSHOT，compiled latest code from the master branch. ``` org.apache.doris doris-spark-connector 2.3.4-2.11-SNAPSHOT ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Wahno commented on issue #7990: [Bug] Spark connector not support STRING type
Wahno commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033393023 Maybe we should add a mapping of STRING type and DataTypes.StringType. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7990: [Bug] Spark connector not support STRING type
hf200012 commented on issue #7990: URL: https://github.com/apache/incubator-doris/issues/7990#issuecomment-1033397007 > Maybe we should add a mapping of STRING type and DataTypes.StringType. I saw that this place needs to be added. What is your usage method? It may not be completely modified last time. You are welcome to submit a PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #7989: [fix] (grouping set) fix Unexpected exception: bitIndex < 0: -1
morningman commented on pull request #7989: URL: https://github.com/apache/incubator-doris/pull/7989#issuecomment-1033413207 Please add ut in QueryPlanTest for this case -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Wahno opened a new pull request #7993: [fix](spark connector) fix spark connector unsupport STRING type.
Wahno opened a new pull request #7993: URL: https://github.com/apache/incubator-doris/pull/7993 # Proposed changes Issue Number: close #7990 ## Problem Summary: fix spark connector unsupport STRING type. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (Yes) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7993: [fix](spark connector) fix spark connector unsupport STRING type.
github-actions[bot] commented on pull request #7993: URL: https://github.com/apache/incubator-doris/pull/7993#issuecomment-1033440666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yinzhijian opened a new pull request #7994: [Feature](ThreadPool) Support thread pool per disk for scanners
yinzhijian opened a new pull request #7994: URL: https://github.com/apache/incubator-doris/pull/7994 # Proposed changes Issue Number: close #7581 ## Problem Summary: Support thread pool per disk for scanners to prevent pool performance from some high ioutil disks happening key point: 1. each disk has a thread pool for scanners 2. whenever a thread pool of one disk runs out of local work, tasks can be retrieved from other threads(disks). This is done round-robin. performance testing: vec version: 25% faster than single thread pool in a high io util disk test case normal version: 8% faster than single thread pool in a high io util disk test case ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
