[GitHub] [incubator-doris] yangzhg commented on pull request #9990: Add opentelemetry related third-party library.
yangzhg commented on PR #9990: URL: https://github.com/apache/incubator-doris/pull/9990#issuecomment-1149543668 json-3.10.1 should not compile test files opentelemetry-cpp-1.2.0 should not download file while compile -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9505: [Refactor] Use file factory to replace create file reader/writer
yiguolei merged PR #9505: URL: https://github.com/apache/incubator-doris/pull/9505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Refactor] Use file factory to replace create file reader/writer (#9505)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 94089b9192 [Refactor] Use file factory to replace create file
reader/writer (#9505)
94089b9192 is described below
commit 94089b919211403de7e6d8e664715d44ab4fd4f3
Author: HappenLee
AuthorDate: Wed Jun 8 15:07:39 2022 +0800
[Refactor] Use file factory to replace create file reader/writer (#9505)
1. Simplify code logic and improve abstraction
2. Fix the mem leak of raw pointer
Co-authored-by: lihaopeng
---
be/src/exec/CMakeLists.txt | 22 ++--
be/src/exec/arrow/arrow_reader.cpp | 2 +-
be/src/exec/arrow/orc_reader.cpp | 2 +-
be/src/exec/arrow/parquet_reader.cpp | 6 +-
be/src/exec/broker_scanner.cpp | 94 +++
be/src/exec/broker_scanner.h | 4 +-
be/src/exec/json_scanner.cpp | 73 ++--
be/src/exec/json_scanner.h | 5 +-
be/src/exec/orc_scanner.cpp| 54 ++---
be/src/exec/parquet_scanner.cpp| 49 ++--
be/src/exec/parquet_writer.cpp | 11 +-
be/src/exec/plain_binary_line_reader.cpp | 2 +-
be/src/exec/plain_text_line_reader.cpp | 2 +-
be/src/io/CMakeLists.txt | 46 +++
be/src/{exec => io}/broker_reader.cpp | 2 +-
be/src/{exec => io}/broker_reader.h| 2 +-
be/src/{exec => io}/broker_writer.cpp | 2 +-
be/src/{exec => io}/broker_writer.h| 2 +-
be/src/{exec => io}/buffered_reader.cpp| 2 +-
be/src/{exec => io}/buffered_reader.h | 2 +-
be/src/io/file_factory.cpp | 132 +
be/src/io/file_factory.h | 77
be/src/{exec => io}/file_reader.h | 0
be/src/{exec => io}/file_writer.h | 0
be/src/{exec => io}/hdfs_file_reader.cpp | 2 +-
be/src/{exec => io}/hdfs_file_reader.h | 2 +-
be/src/{exec => io}/hdfs_reader_writer.cpp | 11 +-
be/src/{exec => io}/hdfs_reader_writer.h | 6 +-
be/src/{exec => io}/hdfs_writer.cpp| 2 +-
be/src/{exec => io}/hdfs_writer.h | 2 +-
be/src/{exec => io}/local_file_reader.cpp | 2 +-
be/src/{exec => io}/local_file_reader.h| 2 +-
be/src/{exec => io}/local_file_writer.cpp | 20 +++-
be/src/{exec => io}/local_file_writer.h| 7 +-
be/src/{exec => io}/s3_reader.cpp | 2 +-
be/src/{exec => io}/s3_reader.h| 2 +-
be/src/{exec => io}/s3_writer.cpp | 2 +-
be/src/{exec => io}/s3_writer.h| 2 +-
be/src/runtime/export_sink.cpp | 53 ++---
be/src/runtime/file_result_writer.cpp | 40 ++-
be/src/runtime/file_result_writer.h| 2 +-
be/src/runtime/routine_load/kafka_consumer_pipe.h | 2 +-
be/src/runtime/stream_load/stream_load_pipe.h | 2 +-
be/src/util/broker_load_error_hub.cpp | 2 +-
be/src/util/broker_storage_backend.cpp | 4 +-
be/src/vec/exec/varrow_scanner.cpp | 44 +--
be/src/vec/exec/vjson_scanner.cpp | 2 +-
be/test/exec/broker_reader_test.cpp| 2 +-
be/test/exec/broker_scan_node_test.cpp | 2 +-
be/test/exec/broker_scanner_test.cpp | 2 +-
be/test/exec/buffered_reader_test.cpp | 4 +-
be/test/exec/hdfs_file_reader_test.cpp | 4 +-
be/test/exec/json_scanner_test.cpp | 2 +-
be/test/exec/json_scanner_with_jsonpath_test.cpp | 2 +-
be/test/exec/multi_bytes_separator_test.cpp| 2 +-
be/test/exec/orc_scanner_test.cpp | 2 +-
be/test/exec/parquet_scanner_test.cpp | 2 +-
be/test/exec/plain_text_line_reader_bzip_test.cpp | 2 +-
be/test/exec/plain_text_line_reader_gzip_test.cpp | 2 +-
.../exec/plain_text_line_reader_lz4frame_test.cpp | 2 +-
be/test/exec/plain_text_line_reader_lzop_test.cpp | 2 +-
.../plain_text_line_reader_uncompressed_test.cpp | 2 +-
be/test/exec/s3_reader_test.cpp| 4 +-
be/test/vec/exec/vbroker_scan_node_test.cpp| 2 +-
be/test/vec/exec/vbroker_scanner_test.cpp | 2 +-
be/test/vec/exec/vjson_scanner_test.cpp| 2 +-
be/test/vec/exec/vorc_scanner_test.cpp | 2 +-
be/test/vec/exec/vparquet_scanner_test.cpp | 2 +-
68 files changed, 427 insertions(+), 429 deletions(-)
diff --git
[GitHub] [incubator-doris] BePPPower closed pull request #9754: [feature] datetime column type support auto-initialized with default …
BePPPower closed pull request #9754: [feature] datetime column type support auto-initialized with default … URL: https://github.com/apache/incubator-doris/pull/9754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10009: [enhance] improve dict in-predicate evaluate
github-actions[bot] commented on PR #10009: URL: https://github.com/apache/incubator-doris/pull/10009#issuecomment-1149552885 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10009: [enhance] improve dict in-predicate evaluate
github-actions[bot] commented on PR #10009: URL: https://github.com/apache/incubator-doris/pull/10009#issuecomment-1149552926 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new issue, #10011: [Enhancement] Support build doris with libhdfs3 on ARM CPUs
yangzhg opened a new issue, #10011: URL: https://github.com/apache/incubator-doris/issues/10011 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Support build doris with libhdfs3 on ARM CPUs， libhdfs3 is not support build on arm， I have folk the libhdfs3 from apache hawq， and support arm through sse2neon， the repo is https://github.com/yangzhg/libhdfs3 ### Solution _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10008: [test]Fix test case infinite loop
github-actions[bot] commented on PR #10008: URL: https://github.com/apache/incubator-doris/pull/10008#issuecomment-1149553933 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #10008: [test]Fix test case infinite loop
github-actions[bot] commented on PR #10008: URL: https://github.com/apache/incubator-doris/pull/10008#issuecomment-1149553961 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caoliang-web opened a new pull request, #10012: [doc]restore adds common errors
caoliang-web opened a new pull request, #10012: URL: https://github.com/apache/incubator-doris/pull/10012 ## Problem Summary: 1.restore adds common errors ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9848: [DefaultConfig]enable query vectorization and storage vectorization and storage low cardinality opt
yiguolei merged PR #9848: URL: https://github.com/apache/incubator-doris/pull/9848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [DefaultConfigChange]enable query vectorization and storage vectorization and storage low cardinality optimization by default (#9848)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new d9bbf67b9e [DefaultConfigChange]enable query vectorization and storage
vectorization and storage low cardinality optimization by default (#9848)
d9bbf67b9e is described below
commit d9bbf67b9eefabe938209da362385b12b3637d42
Author: yiguolei <676222...@qq.com>
AuthorDate: Wed Jun 8 15:29:43 2022 +0800
[DefaultConfigChange]enable query vectorization and storage vectorization
and storage low cardinality optimization by default (#9848)
Co-authored-by: yiguolei
---
be/src/common/config.h| 4 ++--
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index fe40be22a8..fd37ca0b48 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -235,9 +235,9 @@ CONF_Int32(index_page_cache_percentage, "10");
// whether to disable page cache feature in storage
CONF_Bool(disable_storage_page_cache, "false");
-CONF_Bool(enable_storage_vectorization, "false");
+CONF_Bool(enable_storage_vectorization, "true");
-CONF_Bool(enable_low_cardinality_optimize, "false");
+CONF_Bool(enable_low_cardinality_optimize, "true");
// be policy
// whether disable automatic compaction task
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 6d18557b98..53392e9901 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -410,7 +410,7 @@ public class SessionVariable implements Serializable,
Writable {
@VariableMgr.VarAttr(name = RUNTIME_FILTER_MAX_IN_NUM)
private int runtimeFilterMaxInNum = 1024;
@VariableMgr.VarAttr(name = ENABLE_VECTORIZED_ENGINE)
-public boolean enableVectorizedEngine = false;
+public boolean enableVectorizedEngine = true;
@VariableMgr.VarAttr(name = ENABLE_PARALLEL_OUTFILE)
public boolean enableParallelOutfile = false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 opened a new pull request, #10013: [Feature] Support outfile on vectorized engine
Gabriel39 opened a new pull request, #10013: URL: https://github.com/apache/incubator-doris/pull/10013 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10007: [Bugfix](load) fix streamload failure due to false unhealthy replica in concurrent stream load
yiguolei commented on code in PR #10007:
URL: https://github.com/apache/incubator-doris/pull/10007#discussion_r892021391
##
be/src/agent/task_worker_pool.cpp:
##
@@ -707,6 +708,13 @@ void
TaskWorkerPool::_publish_version_worker_thread_callback() {
std::this_thread::sleep_for(std::chrono::seconds(1));
}
}
+if (res == OLAP_ERR_ROWSET_VERSION_NOT_CONTINUOUS) {
+std::unique_lock
worker_thread_lock(_worker_thread_lock);
+// push back to task queue if publish version not continuous
+// make sure publish version task run in order
+_tasks.push_back(agent_task_req);
Review Comment:
Maybe there will be a dead loop. If current tablet is a bad tablet(it really
has a missed version), you will push the task to task queue and not publish it.
Clone process will failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 opened a new issue, #10014: [Feature] Support outfile on vectorized engine
Gabriel39 opened a new issue, #10014: URL: https://github.com/apache/incubator-doris/issues/10014 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Support outfile on vectorized engine ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] mrhhsg opened a new issue, #10015: [Enhancement] Cache bitshuffle page decoded data to improve performance
mrhhsg opened a new issue, #10015: URL: https://github.com/apache/incubator-doris/issues/10015 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Even if the page cache is hit, decoding the bitshuffle page data still consumes a lot of CPU performance, so the decoded data needs to be cached. https://user-images.githubusercontent.com/1179834/172563804-d3d08bfc-c1e5-46d6-940f-a37e9a3de30c.png";> ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] mrhhsg opened a new pull request, #10016: [enhancement]Cache parsed page data for bitshuffle encoding
mrhhsg opened a new pull request, #10016: URL: https://github.com/apache/incubator-doris/pull/10016 # Proposed changes Issue Number: close #10015 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] gj-zhang commented on a diff in pull request #33: [FIX] fix flink date and timestamp type not mapping.
gj-zhang commented on code in PR #33:
URL:
https://github.com/apache/incubator-doris-flink-connector/pull/33#discussion_r892043741
##
flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/RowDataSerializer.java:
##
@@ -33,36 +34,32 @@
import static
org.apache.doris.flink.sink.writer.LoadConstants.DORIS_DELETE_SIGN;
import static org.apache.doris.flink.sink.writer.LoadConstants.JSON;
import static org.apache.doris.flink.sink.writer.LoadConstants.NULL_VALUE;
-import static org.apache.flink.table.data.RowData.createFieldGetter;
/**
* Serializer for RowData.
*/
public class RowDataSerializer implements DorisRecordSerializer {
String[] fieldNames;
-RowData.FieldGetter[] fieldGetters;
String type;
private ObjectMapper objectMapper;
private final String fieldDelimiter;
private final boolean enableDelete;
+private final DorisRowConverter rowConverter;
-private RowDataSerializer(String[] fieldNames, DataType[] dataTypes,
String type, String fieldDelimiter, boolean enableDelete) {
+private RowDataSerializer(String[] fieldNames, RowType rowType, String
type, String fieldDelimiter, boolean enableDelete) {
Review Comment:
yes. it be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] gj-zhang commented on a diff in pull request #33: [FIX] fix flink date and timestamp type not mapping.
gj-zhang commented on code in PR #33:
URL:
https://github.com/apache/incubator-doris-flink-connector/pull/33#discussion_r892043410
##
flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/converter/DorisRowConverter.java:
##
@@ -125,4 +181,59 @@ protected DeserializationConverter
createConverter(LogicalType type) {
throw new UnsupportedOperationException("Unsupported type:" +
type);
}
}
+
+protected SerializationConverter createExternalConverter(LogicalType type)
{
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Kikyou1997 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
Kikyou1997 commented on code in PR #9993:
URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892077031
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanContext.java:
##
@@ -0,0 +1,49 @@
+package org.apache.doris.nereids.trees.plans;
+
+import org.apache.doris.analysis.Analyzer;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.common.IdGenerator;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.planner.PlanFragmentId;
+import org.apache.doris.planner.PlanNodeId;
+
+import java.util.List;
+
+public class PlanContext {
+private List planFragmentList;
+
+private Analyzer analyzer;
+
+private final IdGenerator fragmentIdGenerator =
PlanFragmentId.createGenerator();
+
+private final IdGenerator nodeIdGenerator =
PlanNodeId.createGenerator();
+
+public List getPlanFragmentList() {
+return planFragmentList;
+}
+
+public TupleDescriptor generateTupleDesc() {
+return analyzer.getDescTbl().createTupleDescriptor();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 commented on a diff in pull request #10016: [enhancement]Cache parsed page data for bitshuffle encoding
Gabriel39 commented on code in PR #10016:
URL: https://github.com/apache/incubator-doris/pull/10016#discussion_r892117835
##
be/src/olap/rowset/segment_v2/column_reader.h:
##
@@ -298,10 +298,10 @@ class FileColumnIterator final : public ColumnIterator {
ParsedPage _page;
// keep dict page decoder
-std::unique_ptr _dict_decoder;
+std::shared_ptr _dict_decoder;
// keep dict page handle to avoid released
-PageHandle _dict_page_handle;
+std::shared_ptr _dict_page_handle;
Review Comment:
Could this variable be deleted safely? I notice that this variable will be
set to a new value each time so I think it's unnecessary to use as a member
variable.
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -679,22 +702,29 @@ Status FileColumnIterator::_read_data_page(const
OrdinalPageIndexIterator& iter)
Slice dict_data;
PageFooterPB dict_footer;
_opts.type = INDEX_PAGE;
+_dict_page_handle = std::make_shared();
RETURN_IF_ERROR(_reader->read_page(_opts,
_reader->get_dict_page_pointer(),
- &_dict_page_handle,
&dict_data, &dict_footer,
- _compress_codec.get()));
+ _dict_page_handle.get(),
&dict_data,
+ &dict_footer,
_compress_codec.get()));
// ignore dict_footer.dict_page_footer().encoding() due to only
// PLAIN_ENCODING is supported for dict page right now
-_dict_decoder =
std::make_unique(dict_data);
+_dict_decoder =
std::make_shared(dict_data);
RETURN_IF_ERROR(_dict_decoder->init());
-auto* pd_decoder =
(BinaryPlainPageDecoder*)_dict_decoder.get();
+auto* pd_decoder = _dict_decoder.get();
_dict_word_info.reset(new StringRef[pd_decoder->_num_elems]);
pd_decoder->get_dict_word_info(_dict_word_info.get());
+_page.dict_page_handle = _dict_page_handle;
}
-dict_page_decoder->set_dict_decoder(_dict_decoder.get(),
_dict_word_info.get());
+dict_page_decoder->set_dict_decoder(_dict_decoder,
_dict_word_info);
}
}
+
+if (_need_cache_parsed_page) {
+ParsedPage* cache_page = new ParsedPage(_page);
Review Comment:
Why we need to clone a cache page? Looks like we have already cached this
page and don't need to duplicate another one
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -468,6 +468,12 @@ FileColumnIterator::FileColumnIterator(ColumnReader*
reader) : _reader(reader) {
Status FileColumnIterator::init(const ColumnIteratorOptions& opts) {
_opts = opts;
RETURN_IF_ERROR(get_block_compression_codec(_reader->get_compression(),
_compress_codec));
+auto encoding = _reader->encoding_info()->encoding();
+_need_cache_parsed_page =
Review Comment:
should we add a new config to enable or disable this feature here? If budget
for page cache is poor, I'm worried that it will lead to serious performance
fallback as we cache a totally parsed page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zxealous opened a new pull request, #10017: [style] Fix the code style defined by class member variables in query…
zxealous opened a new pull request, #10017: URL: https://github.com/apache/incubator-doris/pull/10017 …_statistics # Proposed changes fix code style in QueryStatistics ## Problem Summary: The definition of class member variables in Doris BE usually starts with an underscore, but in the two classes NodeStatistics and QueryStatistics, class member variables are not defined in this form. Should this be modified to this form? ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9977: [doc](website)Add Doris new official website code and documents
morningman merged PR #9977: URL: https://github.com/apache/incubator-doris/pull/9977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10016: [enhancement]Cache parsed page data for bitshuffle encoding
yiguolei commented on code in PR #10016:
URL: https://github.com/apache/incubator-doris/pull/10016#discussion_r892186809
##
be/src/olap/rowset/segment_v2/binary_dict_page.cpp:
##
@@ -313,5 +315,23 @@ Status BinaryDictPageDecoder::next_batch(size_t* n,
ColumnBlockView* dst) {
return Status::OK();
}
+BinaryDictPageDecoder* BinaryDictPageDecoder::clone_for_cache() const {
Review Comment:
doris's code style is Status
BinaryDictPageDecoder::clone_for_cache(std::unique_ptr< BinaryDictPageDecoder
>& BinaryDictPageDecoder)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10016: [enhancement]Cache parsed page data for bitshuffle encoding
yiguolei commented on code in PR #10016:
URL: https://github.com/apache/incubator-doris/pull/10016#discussion_r892190895
##
be/src/olap/rowset/segment_v2/page_decoder.h:
##
@@ -102,6 +102,8 @@ class PageDecoder {
bool has_remaining() const { return current_index() < count(); }
+virtual PageDecoder* clone_for_cache() const = 0;
Review Comment:
add a default implementation here, just return __builtin_unreachable, and
other page decoder's impelementation need not implement a default
__builtin_unreachable again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10016: [enhancement]Cache parsed page data for bitshuffle encoding
yiguolei commented on code in PR #10016:
URL: https://github.com/apache/incubator-doris/pull/10016#discussion_r892256439
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -679,22 +702,29 @@ Status FileColumnIterator::_read_data_page(const
OrdinalPageIndexIterator& iter)
Slice dict_data;
PageFooterPB dict_footer;
_opts.type = INDEX_PAGE;
+_dict_page_handle = std::make_shared();
Review Comment:
这里如果不改动，那么相当于每个iterator 会读取一次dict page，然后生成新的word info 与 dict decoder，
这个代价很低，但是我们的代码结构会很简单，我觉得这次先可以采取这个策略。
##
be/src/olap/rowset/segment_v2/binary_dict_page.cpp:
##
@@ -313,5 +315,23 @@ Status BinaryDictPageDecoder::next_batch(size_t* n,
ColumnBlockView* dst) {
return Status::OK();
}
+BinaryDictPageDecoder* BinaryDictPageDecoder::clone_for_cache() const {
+BinaryDictPageDecoder* new_one = new BinaryDictPageDecoder(_data,
_options);
+new_one->_parsed = true;
+new_one->_encoding_type = _encoding_type;
+if (_encoding_type == DICT_ENCODING) {
+DCHECK(_bit_shuffle_ptr != nullptr);
+ColumnVectorBatch::create(0, false,
get_scalar_type_info(), nullptr,
Review Comment:
_batch is only used in non-vec engine, I think we could ignore it. And we
could add a check in column_reader only open parsed page cache during
vec-engine.
##
be/src/olap/rowset/segment_v2/page_decoder.h:
##
@@ -102,6 +102,8 @@ class PageDecoder {
bool has_remaining() const { return current_index() < count(); }
+virtual PageDecoder* clone_for_cache() const = 0;
Review Comment:
可能 unreachable 是不对的，他只是一个hint， 我觉得我们得直接 LOG(FATAL)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request, #10018: [enhancement] support build libhdfs in arm cpus
yangzhg opened a new pull request, #10018: URL: https://github.com/apache/incubator-doris/pull/10018 # Proposed changes Issue Number: close #10011 ## Problem Summary: Supports native hdfs functionality on arm cpu This pr mainly upgrades libdfs3 and supports running on arm，and make libhdfs3 with kerberos as default ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (Yes) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9972: [feature] datetime column type support auto-initialized with default …
yiguolei commented on code in PR #9972:
URL: https://github.com/apache/incubator-doris/pull/9972#discussion_r892010434
##
regression-test/suites/correctness/test_current_timestamp.groovy:
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+ // or more contributor license agreements. See the NOTICE file
+ // distributed with this work for additional information
+ // regarding copyright ownership. The ASF licenses this file
+ // to you under the Apache License, Version 2.0 (the
+ // "License"); you may not use this file except in compliance
+ // with the License. You may obtain a copy of the License at
+ //
+ // http://www.apache.org/licenses/LICENSE-2.0
+ //
+ // Unless required by applicable law or agreed to in writing,
+ // software distributed under the License is distributed on an
+ // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ // KIND, either express or implied. See the License for the
+ // specific language governing permissions and limitations
+ // under the License.
+
+suite("test_current_timestamp", "correctness") {
+def tableName = "test_current_timestamp"
+
+sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """
+CREATE TABLE ${tableName}
+(
+id TINYINT,
+name CHAR(10) NOT NULL DEFAULT "zs",
+dt_0 DATETIME,
+dt_1 DATETIME DEFAULT CURRENT_TIMESTAMP
+)
+COMMENT "test current_timestamp table"
+DISTRIBUTED BY HASH(id)
+PROPERTIES("replication_num" = "1");
+"""
+
+// test insert into.
+sql " insert into ${tableName} (id,name,dt_0) values
(1,'aa',current_timestamp()); "
+sql " insert into ${tableName} (id,name,dt_0) values
(2,'bb',current_timestamp()); "
+sql " insert into ${tableName} (id,name,dt_0) values
(3,'cc',current_timestamp()); "
+sql " insert into ${tableName} (id,name,dt_0) values
(4,'dd',current_timestamp()); "
+
+qt_insert_into """ select count(*) from ${tableName} where dt_0 = dt_1; """
Review Comment:
Do not compare date time, compare to_Date(dt_0) = to_date(dt_1), because
date time is very accurate and the value maybe different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9972: [feature] datetime column type support auto-initialized with default …
github-actions[bot] commented on PR #9972: URL: https://github.com/apache/incubator-doris/pull/9972#issuecomment-1149828737 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9972: [feature] datetime column type support auto-initialized with default …
github-actions[bot] commented on PR #9972: URL: https://github.com/apache/incubator-doris/pull/9972#issuecomment-1149828774 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
HappenLee commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892117337
##
be/src/olap/reader.h:
##
@@ -108,7 +108,7 @@ class TabletReader {
virtual ~TabletReader();
// Initialize TabletReader with tablet, data version and fetch range.
-virtual Status init(const ReaderParams& read_params);
+virtual Status init(const ReaderParams& read_params, bool is_alter_table =
false);
Review Comment:
why not put the `is_alter_table` to `read_params`.
if `is_alter_table` is `true` the `read_params.reader_type ==
READER_ALTER_TABLE` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 opened a new pull request, #10019: [Enhancement](Nereids) generate pattern by operator
924060929 opened a new pull request, #10019: URL: https://github.com/apache/incubator-doris/pull/10019 # Proposed changes generate pattern by operator ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: No 2. Has unit tests been added: No Need 3. Has document been added or modified: No Need 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
HappenLee commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892290286
##
be/src/olap/schema_change.h:
##
@@ -70,6 +59,8 @@ class RowBlockChanger {
// delete handler for filtering data which use specified in DELETE_DATA
const DeleteHandler* _delete_handler = nullptr;
+DescriptorTbl _desc_tbl;
Review Comment:
why add `_desc_tbl` seems only use in construct function?
##
be/src/olap/schema_change.h:
##
@@ -94,20 +85,62 @@ class SchemaChange {
SchemaChange() : _filtered_rows(0), _merged_rows(0) {}
virtual ~SchemaChange() = default;
-virtual Status process(RowsetReaderSharedPtr rowset_reader, RowsetWriter*
new_rowset_builder,
- TabletSharedPtr tablet, TabletSharedPtr
base_tablet) = 0;
-
-void add_filtered_rows(uint64_t filtered_rows) { _filtered_rows +=
filtered_rows; }
-
-void add_merged_rows(uint64_t merged_rows) { _merged_rows += merged_rows; }
+virtual Status process(RowsetReaderSharedPtr rowset_reader, RowsetWriter*
rowset_writer,
+ TabletSharedPtr new_tablet, TabletSharedPtr
base_tablet) {
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+RETURN_WITH_WARN_IF_ERROR(
+rowset_writer->flush(),
+Status::OLAPInternalError(OLAP_ERR_INPUT_PARAMETER_ERROR),
+fmt::format("create empty version for schema change
failed. version= {}-{}",
+rowset_writer->version().first,
rowset_writer->version().second));
+
+return Status::OK();
+}
+
+_filtered_rows = 0;
+_merged_rows = 0;
+
+RETURN_IF_ERROR(_inner_process(rowset_reader, rowset_writer,
new_tablet, base_tablet));
+_add_filtered_rows(rowset_reader->filtered_rows());
+
+// Check row num changes
+if (config::row_nums_check) {
Review Comment:
`if (config::row_nums_check && !_check_row_nums(rowset_reader,
*rowset_writer))`
##
be/src/olap/schema_change.cpp:
##
@@ -807,22 +807,32 @@ bool
RowBlockAllocator::is_memory_enough_for_sorting(size_t num_rows, size_t all
RowBlockMerger::RowBlockMerger(TabletSharedPtr tablet) : _tablet(tablet) {}
-RowBlockMerger::~RowBlockMerger() {}
+RowBlockMerger::~RowBlockMerger() = default;
bool RowBlockMerger::merge(const std::vector& row_block_arr,
RowsetWriter* rowset_writer,
uint64_t* merged_rows) {
uint64_t tmp_merged_rows = 0;
RowCursor row_cursor;
std::unique_ptr mem_pool(new MemPool("RowBlockMerger"));
std::unique_ptr agg_object_pool(new ObjectPool());
+
+auto merge_error = [&]() -> bool {
+while (_heap.size() > 0) {
Review Comment:
`!_heap.empty()` same in line 839
##
be/src/olap/schema_change.cpp:
##
@@ -917,37 +918,33 @@ void RowBlockMerger::_pop_heap() {
return;
Review Comment:
useless return
##
be/src/olap/schema_change.cpp:
##
@@ -92,19 +91,20 @@ class RowBlockMerger {
std::priority_queue _heap;
};
-RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema) {
+RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema,
DescriptorTbl desc_tbl)
+: _desc_tbl(desc_tbl) {
_schema_mapping.resize(tablet_schema.num_columns());
}
RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema,
- const DeleteHandler* delete_handler) {
+ const DeleteHandler* delete_handler,
DescriptorTbl desc_tbl)
+: _desc_tbl(desc_tbl) {
_schema_mapping.resize(tablet_schema.num_columns());
_delete_handler = delete_handler;
}
RowBlockChanger::~RowBlockChanger() {
-SchemaMapping::iterator it = _schema_mapping.begin();
-for (; it != _schema_mapping.end(); ++it) {
+for (auto it = _schema_mapping.begin(); it != _schema_mapping.end(); ++it)
{
SAFE_DELETE(it->default_value);
Review Comment:
recheck the logic? maybe we should do the work in `~ColumnMapping` and
`RowBlockChanger::~RowBlockChanger() = default`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9998: [feature-wip](array-type) support array in mysql_result_writer
github-actions[bot] commented on PR #9998: URL: https://github.com/apache/incubator-doris/pull/9998#issuecomment-1149890059 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9998: [feature-wip](array-type) support array in mysql_result_writer
github-actions[bot] commented on PR #9998: URL: https://github.com/apache/incubator-doris/pull/9998#issuecomment-1149890099 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8513: [Bug] fix don't rewrite nested union statement bug
github-actions[bot] commented on PR #8513: URL: https://github.com/apache/incubator-doris/pull/8513#issuecomment-1149906546 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #9056: [feature-wip](array-type) Add array aggregation functions
HappenLee commented on PR #9056: URL: https://github.com/apache/incubator-doris/pull/9056#issuecomment-1149935128 array agg([]) should return `NULL`, support it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on pull request #9056: [feature-wip](array-type) Add array aggregation functions
HappenLee commented on PR #9056: URL: https://github.com/apache/incubator-doris/pull/9056#issuecomment-1149936408 please add doc for this function -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a diff in pull request #9056: [feature-wip](array-type) Add array aggregation functions
HappenLee commented on code in PR #9056:
URL: https://github.com/apache/incubator-doris/pull/9056#discussion_r892413523
##
be/src/vec/functions/array/function_array_mapped.h:
##
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/array/FunctionArrayMapped.h
+// and modified by Doris
+
+#include "vec/columns/column.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/functions/function.h"
+
+namespace doris {
+namespace vectorized {
+
+/** Higher-order functions for arrays.
+ * These functions optionally apply a map (transform) to array (or multiple
arrays of identical size) by lambda function,
+ * and return some result based on that transformation.
+ *
+ * Examples:
+ * arrayMap(x1,...,xn -> expression, array1,...,arrayn) - apply the
expression to each element of the array (or set of parallel arrays).
+ * arrayFilter(x -> predicate, array) - leave in the array only the elements
for which the expression is true.
+ *
+ * For some functions arrayCount, arrayExists, arrayAll, an overload of the
form f(array) is available,
+ * which works in the same way as f(x -> x, array).
+ *
+ * See the example of Impl template parameter in arrayMap.cpp
+ */
+template
+class FunctionArrayMapped : public IFunction {
+public:
+static constexpr auto name = Name::name;
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+const auto& typed_column = block.get_by_position(arguments[0]);
+const typename Impl::column_type* column_array;
+if (typed_column.column->is_nullable()) {
Review Comment:
if pass a nullable column ？ do nothing with `NULL` value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
BiteThet commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892414205
##
be/src/olap/reader.h:
##
@@ -108,7 +108,7 @@ class TabletReader {
virtual ~TabletReader();
// Initialize TabletReader with tablet, data version and fetch range.
-virtual Status init(const ReaderParams& read_params);
+virtual Status init(const ReaderParams& read_params, bool is_alter_table =
false);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
BiteThet commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892417931
##
be/src/olap/schema_change.cpp:
##
@@ -92,19 +91,20 @@ class RowBlockMerger {
std::priority_queue _heap;
};
-RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema) {
+RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema,
DescriptorTbl desc_tbl)
+: _desc_tbl(desc_tbl) {
_schema_mapping.resize(tablet_schema.num_columns());
}
RowBlockChanger::RowBlockChanger(const TabletSchema& tablet_schema,
- const DeleteHandler* delete_handler) {
+ const DeleteHandler* delete_handler,
DescriptorTbl desc_tbl)
+: _desc_tbl(desc_tbl) {
_schema_mapping.resize(tablet_schema.num_columns());
_delete_handler = delete_handler;
}
RowBlockChanger::~RowBlockChanger() {
-SchemaMapping::iterator it = _schema_mapping.begin();
-for (; it != _schema_mapping.end(); ++it) {
+for (auto it = _schema_mapping.begin(); it != _schema_mapping.end(); ++it)
{
SAFE_DELETE(it->default_value);
Review Comment:
`_schema_mapping.default_value` seems be inited by `_parse_request(...
RowBlockChanger* rb_changer ...`, so maybe we better delete it by whole vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
BiteThet commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892419782
##
be/src/olap/schema_change.h:
##
@@ -70,6 +59,8 @@ class RowBlockChanger {
// delete handler for filtering data which use specified in DELETE_DATA
const DeleteHandler* _delete_handler = nullptr;
+DescriptorTbl _desc_tbl;
Review Comment:
It is used to vectorize schema_change, will be used in later pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on a diff in pull request #10003: [Feature] [Vectorized] Some pre-refactorings or interface additions for schema change part2
BiteThet commented on code in PR #10003:
URL: https://github.com/apache/incubator-doris/pull/10003#discussion_r892420206
##
be/src/olap/schema_change.cpp:
##
@@ -807,22 +807,32 @@ bool
RowBlockAllocator::is_memory_enough_for_sorting(size_t num_rows, size_t all
RowBlockMerger::RowBlockMerger(TabletSharedPtr tablet) : _tablet(tablet) {}
-RowBlockMerger::~RowBlockMerger() {}
+RowBlockMerger::~RowBlockMerger() = default;
bool RowBlockMerger::merge(const std::vector& row_block_arr,
RowsetWriter* rowset_writer,
uint64_t* merged_rows) {
uint64_t tmp_merged_rows = 0;
RowCursor row_cursor;
std::unique_ptr mem_pool(new MemPool("RowBlockMerger"));
std::unique_ptr agg_object_pool(new ObjectPool());
+
+auto merge_error = [&]() -> bool {
+while (_heap.size() > 0) {
Review Comment:
done
##
be/src/olap/schema_change.cpp:
##
@@ -917,37 +918,33 @@ void RowBlockMerger::_pop_heap() {
return;
Review Comment:
removed
##
be/src/olap/schema_change.h:
##
@@ -94,20 +85,62 @@ class SchemaChange {
SchemaChange() : _filtered_rows(0), _merged_rows(0) {}
virtual ~SchemaChange() = default;
-virtual Status process(RowsetReaderSharedPtr rowset_reader, RowsetWriter*
new_rowset_builder,
- TabletSharedPtr tablet, TabletSharedPtr
base_tablet) = 0;
-
-void add_filtered_rows(uint64_t filtered_rows) { _filtered_rows +=
filtered_rows; }
-
-void add_merged_rows(uint64_t merged_rows) { _merged_rows += merged_rows; }
+virtual Status process(RowsetReaderSharedPtr rowset_reader, RowsetWriter*
rowset_writer,
+ TabletSharedPtr new_tablet, TabletSharedPtr
base_tablet) {
+if (rowset_reader->rowset()->empty() ||
rowset_reader->rowset()->num_rows() == 0) {
+RETURN_WITH_WARN_IF_ERROR(
+rowset_writer->flush(),
+Status::OLAPInternalError(OLAP_ERR_INPUT_PARAMETER_ERROR),
+fmt::format("create empty version for schema change
failed. version= {}-{}",
+rowset_writer->version().first,
rowset_writer->version().second));
+
+return Status::OK();
+}
+
+_filtered_rows = 0;
+_merged_rows = 0;
+
+RETURN_IF_ERROR(_inner_process(rowset_reader, rowset_writer,
new_tablet, base_tablet));
+_add_filtered_rows(rowset_reader->filtered_rows());
+
+// Check row num changes
+if (config::row_nums_check) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9953: [feature-wip](multi-catalog)(step2) Introduce Internal Data Source
github-actions[bot] commented on PR #9953: URL: https://github.com/apache/incubator-doris/pull/9953#issuecomment-1149961215 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9953: [feature-wip](multi-catalog)(step2) Introduce Internal Data Source
yiguolei merged PR #9953: URL: https://github.com/apache/incubator-doris/pull/9953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9873: [bug][hudi] use lowerCase to get hudi fileFormatType
github-actions[bot] commented on PR #9873: URL: https://github.com/apache/incubator-doris/pull/9873#issuecomment-1150001674 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9873: [bug][hudi] use lowerCase to get hudi fileFormatType
github-actions[bot] commented on PR #9873: URL: https://github.com/apache/incubator-doris/pull/9873#issuecomment-1150001727 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on a diff in pull request #9993: [feature](nereids) Plan Translator
jackwener commented on code in PR #9993: URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892465090 ## fe/fe-core/src/main/java/org/apache/doris/nereids/properties/PhysicalProperties.java: ## @@ -17,9 +17,20 @@ package org.apache.doris.nereids.properties; +import org.apache.doris.analysis.DistributionDesc; Review Comment: config `style` according to [doc](https://doris.apache.org/zh-CN/developer-guide/java-format-code.html#import-order) ## fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java: ## @@ -140,6 +141,60 @@ public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, TableRef inne } } +public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, JoinOperator joinOp, List eqJoinConjuncts, Review Comment: Why this PR include `planner`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] smallx commented on pull request #9985: [fix] Fix type description in PrimitiveType
smallx commented on PR #9985: URL: https://github.com/apache/incubator-doris/pull/9985#issuecomment-1150073192 @BiteThet Unit test passed. Please review it again. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] alangui opened a new issue, #10020: [Bug] configure: error: in `/root/apache-doris-1.0.0-incubating-src/thirdparty/src/unixODBC-2.3.7'
alangui opened a new issue, #10020: URL: https://github.com/apache/incubator-doris/issues/10020 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version docker build env: apache/incubator-doris:build-env-for-1.0.0 code tag: 1.0.0-rc3 ### What's Wrong? = begin build unixODBC-2.3.7 checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for gawk... (cached) gawk checking for bison... bison -y checking for style of include used by make... GNU checking for gcc... /var/local/ldb-toolchain/bin/../bin/gcc checking whether the C compiler works... no configure: error: in `/root/incubator-doris-1.0.0-rc03/thirdparty/src/unixODBC-2.3.7': configure: error: C compiler cannot create executables See `config.log' for more details ### What You Expected? Build success ### How to Reproduce? unset DORIS_THIRDPARTY sh build.sh --clean --be --fe --ui ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Kikyou1997 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
Kikyou1997 commented on code in PR #9993: URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892593582 ## fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java: ## @@ -140,6 +141,60 @@ public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, TableRef inne } } +public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, JoinOperator joinOp, List eqJoinConjuncts, Review Comment: > Why this PR include `planner`? `tableRef` is useless in the new optimizer, and we can't get and no need to a such object from nereids, so add a new constructor which invoked by PlanTranslator -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Kikyou1997 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
Kikyou1997 commented on code in PR #9993: URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892593582 ## fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java: ## @@ -140,6 +141,60 @@ public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, TableRef inne } } +public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, JoinOperator joinOp, List eqJoinConjuncts, Review Comment: > Why this PR include `planner`? `tableRef` is useless in the new optimizer, and we can't get and no need to construct a such object from nereids, so add a new constructor which invoked by PlanTranslator -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [enhance] improve dict in-predicate evaluate (#10009)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9c52b4a508 [enhance] improve dict in-predicate evaluate (#10009)
9c52b4a508 is described below
commit 9c52b4a508f7c4865ae077e286ea0b7794efa03b
Author: minghong
AuthorDate: Thu Jun 9 00:25:30 2022 +0800
[enhance] improve dict in-predicate evaluate (#10009)
---
be/src/olap/in_list_predicate.cpp | 13 -
be/src/vec/columns/column_dictionary.h | 17 +
2 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/be/src/olap/in_list_predicate.cpp
b/be/src/olap/in_list_predicate.cpp
index b33ef09c49..82c8241368 100644
--- a/be/src/olap/in_list_predicate.cpp
+++ b/be/src/olap/in_list_predicate.cpp
@@ -132,13 +132,14 @@ IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
auto* nested_col_ptr = vectorized::check_and_get_column<
\
vectorized::ColumnDictionary>(nested_col);\
auto& data_array = nested_col_ptr->get_data();
\
-auto dict_codes = nested_col_ptr->find_codes(_values);
\
+std::vector selected;
\
+nested_col_ptr->find_codes(_values, selected);
\
for (uint16_t i = 0; i < *size; i++) {
\
uint16_t idx = sel[i];
\
sel[new_size] = idx;
\
const auto& cell_value = data_array[idx];
\
-bool ret = !null_bitmap[idx] &&
\
- (dict_codes.find(cell_value) OP
dict_codes.end());\
+DCHECK(cell_value < selected.size());
\
+bool ret = !null_bitmap[idx] && (selected[cell_value]
OP false); \
new_size += _opposite ? !ret : ret;
\
}
\
}
\
@@ -161,12 +162,14 @@ IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
reinterpret_cast&>( \
column);
\
auto& data_array = dict_col.get_data();
\
-auto dict_codes = dict_col.find_codes(_values);
\
+std::vector selected;
\
+dict_col.find_codes(_values, selected);
\
for (uint16_t i = 0; i < *size; i++) {
\
uint16_t idx = sel[i];
\
sel[new_size] = idx;
\
const auto& cell_value = data_array[idx];
\
-auto result = (dict_codes.find(cell_value) OP
dict_codes.end()); \
+DCHECK(cell_value < selected.size());
\
+auto result = (selected[cell_value] OP false);
\
new_size += _opposite ? !result : result;
\
}
\
}
\
diff --git a/be/src/vec/columns/column_dictionary.h
b/be/src/vec/columns/column_dictionary.h
index 532d7eb6a7..29db3a334c 100644
--- a/be/src/vec/columns/column_dictionary.h
+++ b/be/src/vec/columns/column_dictionary.h
@@ -258,9 +258,9 @@ public:
uint32_t get_hash_value(uint32_t idx) const { return
_dict.get_hash_value(_codes[idx]); }
-phmap::flat_hash_set find_codes(
-const phmap::flat_hash_set& values) const {
-return _dict.find_codes(values);
+void find_codes(const phmap::flat_hash_set& values,
+std::vector& selected) const {
+return _dict.find_codes(values, selected);
}
bool is_dict_sorted
[GitHub] [incubator-doris] yiguolei merged pull request #10009: [enhance] improve dict in-predicate evaluate
yiguolei merged PR #10009: URL: https://github.com/apache/incubator-doris/pull/10009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Kikyou1997 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
Kikyou1997 commented on code in PR #9993: URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892595594 ## fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java: ## @@ -140,6 +141,60 @@ public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, TableRef inne } } +public HashJoinNode(PlanNodeId id, PlanNode outer, PlanNode inner, JoinOperator joinOp, List eqJoinConjuncts, Review Comment: > Why this PR include `planner`? And any other modification to Planner is based on the similiar reason as commented above -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei closed issue #9570: [Feature] datetime column type support auto-initialized with default CURRENT_TIMESTAMP
yiguolei closed issue #9570: [Feature] datetime column type support auto-initialized with default CURRENT_TIMESTAMP URL: https://github.com/apache/incubator-doris/issues/9570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9972: [feature] datetime column type support auto-initialized with default …
yiguolei merged PR #9972: URL: https://github.com/apache/incubator-doris/pull/9972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature] datetime column type support auto-initialized with default … (#9972)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 99fb830023 [feature] datetime column type support auto-initialized
with default … (#9972)
99fb830023 is described below
commit 99fb830023aa38696890ef512a0535b4673a112e
Author: BePPPower <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Thu Jun 9 00:28:03 2022 +0800
[feature] datetime column type support auto-initialized with default …
(#9972)
---
fe/fe-core/src/main/cup/sql_parser.cup | 8 ++-
.../java/org/apache/doris/analysis/ColumnDef.java | 39 +--
.../org/apache/doris/analysis/DataDescription.java | 4 +-
.../apache/doris/analysis/DefaultValueExprDef.java | 58 ++
.../apache/doris/analysis/FunctionCallExpr.java| 17 +++
.../java/org/apache/doris/analysis/InsertStmt.java | 10 +++-
.../main/java/org/apache/doris/catalog/Column.java | 19 +--
.../src/main/java/org/apache/doris/load/Load.java | 12 -
.../org/apache/doris/planner/LoadScanNode.java | 6 ++-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../data/correctness/test_current_timestamp.out| 7 +++
.../test_current_timestamp_streamload.csv | 4 ++
.../correctness/test_current_timestamp.groovy | 55
13 files changed, 225 insertions(+), 15 deletions(-)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index 4affd1e466..2bdb818ed8 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -241,7 +241,7 @@ parser code {:
// Total keywords of doris
terminal String KW_ADD, KW_ADMIN, KW_AFTER, KW_AGGREGATE, KW_ALIAS, KW_ALL,
KW_ALTER, KW_AND, KW_ANTI, KW_APPEND, KW_AS, KW_ASC, KW_AUTHORS, KW_ARRAY,
KW_BACKEND, KW_BACKUP, KW_BETWEEN, KW_BEGIN, KW_BIGINT, KW_BINLOG,
KW_BITMAP, KW_BITMAP_UNION, KW_QUANTILE_STATE, KW_QUANTILE_UNION, KW_BLOB,
KW_BOOLEAN, KW_BROKER, KW_BACKENDS, KW_BY, KW_BUILTIN,
-KW_CANCEL, KW_CASE, KW_CAST, KW_CHAIN, KW_CHAR, KW_CHARSET, KW_CHECK,
KW_CLUSTER, KW_CLUSTERS, KW_CLEAN,
+KW_CANCEL, KW_CASE, KW_CAST, KW_CHAIN, KW_CHAR, KW_CHARSET, KW_CHECK,
KW_CLUSTER, KW_CLUSTERS, KW_CLEAN, KW_CURRENT_TIMESTAMP,
KW_COLLATE, KW_COLLATION, KW_COLUMN, KW_COLUMNS, KW_COMMENT, KW_COMMIT,
KW_COMMITTED, KW_COMPACT,
KW_CONFIG, KW_CONNECTION, KW_CONNECTION_ID, KW_CONSISTENT, KW_CONVERT,
KW_COUNT, KW_CREATE, KW_CREATION, KW_CROSS, KW_CUBE, KW_CURRENT,
KW_CURRENT_USER,
KW_DATA, KW_DATABASE, KW_DATABASES, KW_DATE, KW_DATETIME, KW_DAY,
KW_DECIMAL, KW_DECOMMISSION, KW_DEFAULT, KW_DESC, KW_DESCRIBE,
@@ -2443,6 +2443,10 @@ opt_default_value ::=
{:
RESULT = ColumnDef.DefaultValue.NULL_DEFAULT_VALUE;
:}
+| KW_DEFAULT KW_CURRENT_TIMESTAMP
+{:
+RESULT = ColumnDef.DefaultValue.CURRENT_TIMESTAMP_DEFAULT_VALUE;
+:}
;
opt_is_key ::=
@@ -5808,6 +5812,8 @@ keyword ::=
{: RESULT = id; :}
| KW_POLICY:id
{: RESULT = id; :}
+| KW_CURRENT_TIMESTAMP:id
+{: RESULT = id; :}
;
// Identifier that contain keyword
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
index d3831f2de6..cffa8df492 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
@@ -61,12 +61,31 @@ public class ColumnDef {
public static class DefaultValue {
public boolean isSet;
public String value;
+// used for column which defaultValue is an expression.
+public DefaultValueExprDef defaultValueExprDef;
public DefaultValue(boolean isSet, String value) {
this.isSet = isSet;
this.value = value;
+this.defaultValueExprDef = null;
}
+/**
+ * used for column which defaultValue is an expression.
+ * @param isSet is Set DefaultValue
+ * @param value default value
+ * @param exprName default value expression
+ */
+public DefaultValue(boolean isSet, String value, String exprName) {
+this.isSet = isSet;
+this.value = value;
+this.defaultValueExprDef = new DefaultValueExprDef(exprName);
+}
+
+// default "CURRENT_TIMESTAMP", only for DATETIME type
+public static String CURRENT_TIMESTAMP = "CURRENT_TIMESTAMP";
+public static String NOW = "now";
+public static DefaultValue CURRENT_TIMESTAMP_DEFAULT_VALUE = new
DefaultValue(true, CURRENT_TIMESTAMP, NOW);
// no default value
public static DefaultValue NOT_SET = new DefaultValue(false, null);
// default null
@@ -279,11 +298,13 @@ public class Col
[GitHub] [incubator-doris] yiguolei merged pull request #9985: [fix] Fix type description in PrimitiveType
yiguolei merged PR #9985: URL: https://github.com/apache/incubator-doris/pull/9985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix] Fix type description in PrimitiveType (#9985)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 342ab52270 [fix] Fix type description in PrimitiveType (#9985)
342ab52270 is described below
commit 342ab5227079194ac0c5f457bbcc666ee622bfb4
Author: smallx
AuthorDate: Thu Jun 9 00:30:32 2022 +0800
[fix] Fix type description in PrimitiveType (#9985)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
index d88369b77e..a953386630 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
@@ -57,7 +57,7 @@ public enum PrimitiveType {
ARRAY("ARRAY", 24, TPrimitiveType.ARRAY),
MAP("MAP", 24, TPrimitiveType.MAP),
-STRUCT("MAP", 24, TPrimitiveType.STRUCT),
+STRUCT("STRUCT", 24, TPrimitiveType.STRUCT),
STRING("STRING", 16, TPrimitiveType.STRING),
// Unsupported scalar types.
BINARY("BINARY", -1, TPrimitiveType.BINARY),
@@ -297,7 +297,7 @@ public enum PrimitiveType {
// QUANTILE_STATE
builder.put(QUANTILE_STATE, QUANTILE_STATE);
-//TIME
+// TIME
builder.put(TIME, TIME);
builder.put(TIME, DOUBLE);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei closed pull request #9969: [enhance] improve dict in-predicate evaluate
yiguolei closed pull request #9969: [enhance] improve dict in-predicate evaluate URL: https://github.com/apache/incubator-doris/pull/9969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9966: [test] Fix date function regression test case.
github-actions[bot] commented on PR #9966: URL: https://github.com/apache/incubator-doris/pull/9966#issuecomment-1150141127 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9966: [test] Fix date function regression test case.
github-actions[bot] commented on PR #9966: URL: https://github.com/apache/incubator-doris/pull/9966#issuecomment-1150141155 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9966: [test] Fix date function regression test case.
yiguolei merged PR #9966: URL: https://github.com/apache/incubator-doris/pull/9966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (342ab52270 -> 688f8f6e90)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git from 342ab52270 [fix] Fix type description in PrimitiveType (#9985) add 688f8f6e90 [test] Fix date function regression test case. (#9966) No new revisions were added by this update. Summary of changes: .../datetime_functions/test_date_function.out | 3 - .../datetime_functions/test_date_function.groovy | 120 + 2 files changed, 78 insertions(+), 45 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Kikyou1997 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
Kikyou1997 commented on code in PR #9993: URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r892602734 ## fe/fe-core/src/main/java/org/apache/doris/nereids/properties/PhysicalProperties.java: ## @@ -17,9 +17,20 @@ package org.apache.doris.nereids.properties; +import org.apache.doris.analysis.DistributionDesc; Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9955: [fix](load action) fix a thread safe problem in LoadAction.java
yiguolei merged PR #9955: URL: https://github.com/apache/incubator-doris/pull/9955 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: fix: fix a thread safe problem in LoadAction.java (#9955)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 449bfe10d1 fix: fix a thread safe problem in LoadAction.java (#9955)
449bfe10d1 is described below
commit 449bfe10d15c34c9009ae244247047375422f1bb
Author: tarepanda1024
AuthorDate: Thu Jun 9 00:34:07 2022 +0800
fix: fix a thread safe problem in LoadAction.java (#9955)
---
.../main/java/org/apache/doris/httpv2/rest/LoadAction.java| 11 +++
1 file changed, 3 insertions(+), 8 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java
index aab70c3742..72b088300e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java
@@ -56,8 +56,6 @@ public class LoadAction extends RestBaseController {
private ExecuteEnv execEnv = ExecuteEnv.getInstance();
-private boolean isStreamLoad = false;
-
@RequestMapping(path = "/api/{" + DB_KEY + "}/{" + TABLE_KEY + "}/_load",
method = RequestMethod.PUT)
public Object load(HttpServletRequest request, HttpServletResponse
response,
@PathVariable(value = DB_KEY) String db,
@PathVariable(value = TABLE_KEY) String table) {
@@ -65,9 +63,8 @@ public class LoadAction extends RestBaseController {
ResponseEntity entity = ResponseEntityBuilder.notFound("The mini
load operation has been disabled by default, if you need to add
disable_mini_load=false in fe.conf.");
return entity;
} else {
-this.isStreamLoad = false;
executeCheckPassword(request, response);
-return executeWithoutPassword(request, response, db, table);
+return executeWithoutPassword(request, response, db, table, false);
}
}
@@ -75,16 +72,14 @@ public class LoadAction extends RestBaseController {
public Object streamLoad(HttpServletRequest request,
HttpServletResponse response,
@PathVariable(value = DB_KEY) String db,
@PathVariable(value = TABLE_KEY) String table) {
-this.isStreamLoad = true;
executeCheckPassword(request, response);
-return executeWithoutPassword(request, response, db, table);
+return executeWithoutPassword(request, response, db, table, true);
}
@RequestMapping(path = "/api/{" + DB_KEY + "}/_stream_load_2pc", method =
RequestMethod.PUT)
public Object streamLoad2PC(HttpServletRequest request,
HttpServletResponse response,
@PathVariable(value = DB_KEY) String db) {
-this.isStreamLoad = true;
executeCheckPassword(request, response);
return executeStreamLoad2PC(request, db);
}
@@ -92,7 +87,7 @@ public class LoadAction extends RestBaseController {
// Same as Multi load, to be compatible with http v1's response body,
// we return error by using RestBaseResult.
private Object executeWithoutPassword(HttpServletRequest request,
- HttpServletResponse response, String
db, String table) {
+ HttpServletResponse response, String
db, String table, boolean isStreamLoad) {
try {
String dbName = db;
String tableName = table;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9931: [fix](regression-test) fix the comparison errors caused by inconsistent accuracy
yiguolei merged PR #9931: URL: https://github.com/apache/incubator-doris/pull/9931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](regression-test) fix the comparison errors caused by inconsistent accuracy (#9931)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new e5ed259da6 [fix](regression-test) fix the comparison errors caused by
inconsistent accuracy (#9931)
e5ed259da6 is described below
commit e5ed259da6b4f401db39f0a4206ba2ba2a048086
Author: Ashin Gau
AuthorDate: Thu Jun 9 00:35:22 2022 +0800
[fix](regression-test) fix the comparison errors caused by inconsistent
accuracy (#9931)
---
.../org/apache/doris/regression/util/OutputUtils.groovy | 16 +++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/OutputUtils.groovy
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/OutputUtils.groovy
index e8f193f376..b220f4deeb 100644
---
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/OutputUtils.groovy
+++
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/OutputUtils.groovy
@@ -24,6 +24,8 @@ import org.apache.commons.csv.CSVPrinter
import org.apache.commons.csv.CSVRecord
import org.apache.commons.io.LineIterator
+import java.math.BigDecimal
+import java.math.RoundingMode
import java.util.function.Function
import java.sql.ResultSetMetaData
@@ -54,7 +56,7 @@ class OutputUtils {
}
static String checkCell(String info, int line, String expectCell, String
realCell, String dataType) {
-if (dataType == "FLOAT" || dataType == "DOUBLE") {
+if (dataType == "FLOAT" || dataType == "DOUBLE" || dataType ==
"DECIMAL") {
Boolean expectNull = expectCell.equals("N")
Boolean actualNull = realCell.equals("N")
@@ -69,6 +71,18 @@ class OutputUtils {
double expectRelativeError = 1e-10
if(expectRelativeError < realRelativeError) {
+// Keep the scale of low precision data to solve TPCH
cases like:
+// "Expect cell is: 0.0395, But real is: 0.039535109"
+int expectDecimalPlaces = expectCell.contains(".") ?
expectCell.length() - expectCell.lastIndexOf(".") - 1 : 0
+int realDecimalPlaces = realCell.contains(".") ?
realCell.length() - realCell.lastIndexOf(".") - 1 : 0
+if (expectDecimalPlaces != realDecimalPlaces) {
+int lowDecimalPlaces = Math.min(expectDecimalPlaces,
realDecimalPlaces)
+double lowNumber = expectDecimalPlaces <
realDecimalPlaces ? expectDouble : realDouble
+double highNumber = expectDecimalPlaces <
realDecimalPlaces ? realDouble : expectDouble
+if (new
BigDecimal(highNumber).setScale(lowDecimalPlaces,
RoundingMode.HALF_UP).doubleValue() == lowNumber) {
+return null
+}
+}
return "${info}, line ${line}, ${dataType} result
mismatch.\nExpect cell is: ${expectCell}\nBut real is: ${realCell}\nrelative
error is: ${realRelativeError}, bigger than ${expectRelativeError}"
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a diff in pull request #9056: [feature-wip](array-type) Add array aggregation functions
adonis0147 commented on code in PR #9056:
URL: https://github.com/apache/incubator-doris/pull/9056#discussion_r892647180
##
be/src/vec/functions/array/function_array_mapped.h:
##
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/array/FunctionArrayMapped.h
+// and modified by Doris
+
+#include "vec/columns/column.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/functions/function.h"
+
+namespace doris {
+namespace vectorized {
+
+/** Higher-order functions for arrays.
+ * These functions optionally apply a map (transform) to array (or multiple
arrays of identical size) by lambda function,
+ * and return some result based on that transformation.
+ *
+ * Examples:
+ * arrayMap(x1,...,xn -> expression, array1,...,arrayn) - apply the
expression to each element of the array (or set of parallel arrays).
+ * arrayFilter(x -> predicate, array) - leave in the array only the elements
for which the expression is true.
+ *
+ * For some functions arrayCount, arrayExists, arrayAll, an overload of the
form f(array) is available,
+ * which works in the same way as f(x -> x, array).
+ *
+ * See the example of Impl template parameter in arrayMap.cpp
+ */
+template
+class FunctionArrayMapped : public IFunction {
+public:
+static constexpr auto name = Name::name;
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) override {
+const auto& typed_column = block.get_by_position(arguments[0]);
+const typename Impl::column_type* column_array;
+if (typed_column.column->is_nullable()) {
Review Comment:
`NULL` values will be skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9910: [Bugfix] be crash when executing sql contains bitmap_intersect function
yiguolei merged PR #9910: URL: https://github.com/apache/incubator-doris/pull/9910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei closed issue #9909: [Bug] be crash when executing sql contains bitmap_intersect function
yiguolei closed issue #9909: [Bug] be crash when executing sql contains bitmap_intersect function URL: https://github.com/apache/incubator-doris/issues/9909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Bugfix] be crash when executing sql contains bitmap_intersect function (#9910)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new bf8b4fb2d3 [Bugfix] be crash when executing sql contains
bitmap_intersect function (#9910)
bf8b4fb2d3 is described below
commit bf8b4fb2d3415b8f129f03a563884cea841e496d
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Thu Jun 9 08:45:46 2022 +0800
[Bugfix] be crash when executing sql contains bitmap_intersect function
(#9910)
* fix bitmap serialize bug
* add regression test for bitmap seralize bugfix
* add missing regression test out file
* fix reggresion test failed issue
---
be/src/exprs/bitmap_function.cpp | 3 +-
.../data/correctness/test_bitmap_serialize.out | 4 +++
.../correctness/test_bitmap_serialize.groovy | 39 ++
3 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index d49863ab73..5e38ab8f79 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -393,7 +393,8 @@ StringVal
BitmapFunctions::bitmap_hash(doris_udf::FunctionContext* ctx,
StringVal BitmapFunctions::bitmap_serialize(FunctionContext* ctx, const
StringVal& src) {
if (src.is_null) {
-return src;
+// bitmap functions should never return nullable value
+return serialize(ctx, nullptr);
}
auto src_bitmap = reinterpret_cast(src.ptr);
diff --git a/regression-test/data/correctness/test_bitmap_serialize.out
b/regression-test/data/correctness/test_bitmap_serialize.out
new file mode 100644
index 00..a36fb412ce
--- /dev/null
+++ b/regression-test/data/correctness/test_bitmap_serialize.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select_default --
+
+
diff --git a/regression-test/suites/correctness/test_bitmap_serialize.groovy
b/regression-test/suites/correctness/test_bitmap_serialize.groovy
new file mode 100644
index 00..b988316213
--- /dev/null
+++ b/regression-test/suites/correctness/test_bitmap_serialize.groovy
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+ // or more contributor license agreements. See the NOTICE file
+ // distributed with this work for additional information
+ // regarding copyright ownership. The ASF licenses this file
+ // to you under the Apache License, Version 2.0 (the
+ // "License"); you may not use this file except in compliance
+ // with the License. You may obtain a copy of the License at
+ //
+ // http://www.apache.org/licenses/LICENSE-2.0
+ //
+ // Unless required by applicable law or agreed to in writing,
+ // software distributed under the License is distributed on an
+ // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ // KIND, either express or implied. See the License for the
+ // specific language governing permissions and limitations
+ // under the License.
+
+ suite("test_bitmap_serialize") {
+ def tableName = "test_bitmap"
+
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+ sql """
+create table ${tableName} (tag varchar(20),user_ids bitmap
bitmap_union) aggregate key (tag)
+distributed by hash (tag) PROPERTIES("replication_num" = "1");
+ """
+
+ sql " insert into ${tableName} values('A', to_bitmap(1)); "
+ sql " insert into ${tableName} values('A', to_bitmap(2)); "
+ sql " insert into ${tableName} values('A', to_bitmap(3)); "
+ sql " insert into ${tableName} values('B', to_bitmap(1)); "
+ sql " insert into ${tableName} values('B', to_bitmap(2)); "
+
+ // test no vectorized
+ sql """ set enable_vectorized_engine = false; """
+
+ qt_select_default """
+ select bitmap_to_string(bitmap_intersect(user_ids)) from ( select tag,
bitmap_union(user_ids) user_ids
+ from ${tableName} group by tag having tag not in("A","B") ) t; """
+ }
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9804: [Feature] compaction quickly for small data import #9791
yangzhg commented on code in PR #9804: URL: https://github.com/apache/incubator-doris/pull/9804#discussion_r893005225 ## be/src/common/config.h: ## @@ -300,6 +300,9 @@ CONF_mInt32(convert_rowset_thread_num, "0"); // initial sleep interval in seconds of scan alpha rowset CONF_mInt32(scan_alpha_rowset_min_interval_sec, "3"); +// This config can be set to limit thread number in smallcompaction thread pool. +CONF_mInt32(small_compaction_max_threads, "10"); Review Comment: use quick compaction is better -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9804: [Feature] compaction quickly for small data import #9791
yangzhg commented on code in PR #9804:
URL: https://github.com/apache/incubator-doris/pull/9804#discussion_r893007504
##
be/src/olap/compaction.cpp:
##
@@ -55,6 +55,68 @@ Status Compaction::execute_compact() {
return st;
}
+Status Compaction::small_rowsets_compact() {
+std::unique_lock
lock(_tablet->get_cumulative_compaction_lock(), std::try_to_lock);
+if (!lock.owns_lock()) {
+LOG(WARNING) << "The tablet is under cumulative compaction. tablet="
+ << _tablet->full_name();
+return Status::OLAPInternalError(OLAP_ERR_CE_TRY_CE_LOCK_ERROR);
+}
+
+// Clone task may happen after compaction task is submitted to thread
pool, and rowsets picked
+// for compaction may change. In this case, current compaction task should
not be executed.
+if (_tablet->get_clone_occurred()) {
+_tablet->set_clone_occurred(false);
+return Status::OLAPInternalError(OLAP_ERR_CUMULATIVE_CLONE_OCCURRED);
+}
+
+_input_rowsets.clear();
+int version_count = _tablet->version_count();
+int64_t now = UnixMillis();
+int64_t permits = 0;
+_tablet->pick_small_verson_rowsets(&_input_rowsets, &permits);
+std::string input_ver = "";
Review Comment:
use sstream, and Version has implement operator <<
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9804: [Feature] compaction quickly for small data import #9791
yangzhg commented on code in PR #9804:
URL: https://github.com/apache/incubator-doris/pull/9804#discussion_r893008698
##
be/src/olap/compaction.cpp:
##
@@ -55,6 +55,68 @@ Status Compaction::execute_compact() {
return st;
}
+Status Compaction::small_rowsets_compact() {
+std::unique_lock
lock(_tablet->get_cumulative_compaction_lock(), std::try_to_lock);
+if (!lock.owns_lock()) {
+LOG(WARNING) << "The tablet is under cumulative compaction. tablet="
+ << _tablet->full_name();
+return Status::OLAPInternalError(OLAP_ERR_CE_TRY_CE_LOCK_ERROR);
+}
+
+// Clone task may happen after compaction task is submitted to thread
pool, and rowsets picked
+// for compaction may change. In this case, current compaction task should
not be executed.
+if (_tablet->get_clone_occurred()) {
+_tablet->set_clone_occurred(false);
+return Status::OLAPInternalError(OLAP_ERR_CUMULATIVE_CLONE_OCCURRED);
+}
+
+_input_rowsets.clear();
+int version_count = _tablet->version_count();
+int64_t now = UnixMillis();
Review Comment:
you can use MonotonicStopWatch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a diff in pull request #9804: [Feature] compaction quickly for small data import #9791
yangzhg commented on code in PR #9804:
URL: https://github.com/apache/incubator-doris/pull/9804#discussion_r893008941
##
be/src/olap/tablet.cpp:
##
@@ -870,10 +870,56 @@ void Tablet::calculate_cumulative_point() {
if (ret_cumulative_point == K_INVALID_CUMULATIVE_POINT) {
return;
}
-
set_cumulative_layer_point(ret_cumulative_point);
}
+//find rowsets that rows less then "config::small_compaction_max_rows"
+Status Tablet::pick_small_verson_rowsets(std::vector*
input_rowsets,
+ int64_t* permits) {
+if (!init_succeeded()) {
+return
Status::OLAPInternalError(OLAP_ERR_CUMULATIVE_INVALID_PARAMETERS);
+}
+int max_series_num = 1000;
+int max_rows = config::small_compaction_max_rows;
+if (max_rows <= 0) return Status::OK();
Review Comment:
```suggestion
if (max_rows <= 0) { return Status::OK(); }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on pull request #9804: [Feature] compaction quickly for small data import #9791
yangzhg commented on PR #9804: URL: https://github.com/apache/incubator-doris/pull/9804#issuecomment-1150596032 Can you provide some test data about how often this feature can be imported with and without it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #9804: [Feature] compaction quickly for small data import #9791
caiconghui commented on PR #9804: URL: https://github.com/apache/incubator-doris/pull/9804#issuecomment-1150597461 will there be any side effect for normal compaction performance, if there are many stream load and with not small data? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9887: [Enhancement] add some metrics for cpu and memory
github-actions[bot] commented on PR #9887: URL: https://github.com/apache/incubator-doris/pull/9887#issuecomment-1150614665 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9887: [Enhancement] add some metrics for cpu and memory
github-actions[bot] commented on PR #9887: URL: https://github.com/apache/incubator-doris/pull/9887#issuecomment-1150614681 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zhangstar333 commented on a diff in pull request #9930: [Vectorized][UDF] support java-udaf
zhangstar333 commented on code in PR #9930:
URL: https://github.com/apache/incubator-doris/pull/9930#discussion_r893032637
##
be/src/vec/aggregate_functions/aggregate_function_java_udaf.h:
##
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#ifdef LIBJVM
+
+#include
+#include
+
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/Exprs_types.h"
+#include "runtime/user_function_cache.h"
+#include "util/jni-util.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_string.h"
+#include "vec/common/exception.h"
+#include "vec/common/string_ref.h"
+#include "vec/core/block.h"
+#include "vec/core/column_numbers.h"
+#include "vec/core/field.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+const char* UDAF_EXECUTOR_CLASS = "org/apache/doris/udf/UdafExecutor";
+const char* UDAF_EXECUTOR_CTOR_SIGNATURE = "([B)V";
+const char* UDAF_EXECUTOR_CLOSE_SIGNATURE = "()V";
+const char* UDAF_EXECUTOR_ADD_SIGNATURE = "(JJ)V";
+const char* UDAF_EXECUTOR_SERIALIZE_SIGNATURE = "(Ljava/lang/Object;)V";
+const char* UDAF_EXECUTOR_MERGE_SIGNATURE = "(Ljava/lang/Object;)V";
+const char* UDAF_EXECUTOR_RESULT_SIGNATURE = "(J)Z";
+
+struct AggregateJavaUdafData {
+public:
+AggregateJavaUdafData() = default;
+AggregateJavaUdafData(int64_t num_args) {
+argument_size = num_args;
+first_init = true;
+input_values_buffer_ptr.reset(new int64_t[num_args]);
+input_nulls_buffer_ptr.reset(new int64_t[num_args]);
+input_offsets_ptrs.reset(new int64_t[num_args]);
+output_value_buffer.reset(new int64_t);
+output_null_value.reset(new int64_t);
+output_offsets_ptr.reset(new int64_t);
+output_intermediate_state_ptr.reset(new int64_t);
+}
+
+~AggregateJavaUdafData() {
+JNIEnv* env;
+Status status;
+RETURN_IF_STATUS_ERROR(status, JniUtil::GetJNIEnv(&env));
+env->CallNonvirtualVoidMethod(executor_obj, executor_cl,
executor_close_id);
+RETURN_IF_STATUS_ERROR(status, JniUtil::GetJniExceptionMsg(env));
+env->DeleteGlobalRef(executor_obj);
+}
+
+Status init_udaf(const TFunction& fn) {
+if (first_init) {
+JNIEnv* env = nullptr;
+RETURN_NOT_OK_STATUS_WITH_WARN(JniUtil::GetJNIEnv(&env),
+ "Java-Udaf init_udaf function");
+RETURN_IF_ERROR(JniUtil::GetGlobalClassRef(env,
UDAF_EXECUTOR_CLASS, &executor_cl));
+RETURN_NOT_OK_STATUS_WITH_WARN(register_func_id(env),
+ "Java-Udaf register_func_id
function");
+
+// Add a scoped cleanup jni reference object. This cleans up local
refs made below.
+JniLocalFrame jni_frame;
+{
+std::string local_location;
+auto function_cache = UserFunctionCache::instance();
+RETURN_IF_ERROR(function_cache->get_jarpath(fn.id,
fn.hdfs_location, fn.checksum,
+&local_location));
+TJavaUdfExecutorCtorParams ctor_params;
+ctor_params.__set_fn(fn);
+ctor_params.__set_location(local_location);
+
ctor_params.__set_input_offsets_ptrs((int64_t)input_offsets_ptrs.get());
+
ctor_params.__set_input_buffer_ptrs((int64_t)input_values_buffer_ptr.get());
+
ctor_params.__set_input_nulls_ptrs((int64_t)input_nulls_buffer_ptr.get());
+
ctor_params.__set_output_buffer_ptr((int64_t)output_value_buffer.get());
+
+
ctor_params.__set_output_null_ptr((int64_t)output_null_value.get());
+
ctor_params.__set_output_offsets_ptr((int64_t)output_offsets_ptr.get());
+ctor_params.__set_output_intermediate_state_ptr(
+(int64_t)output_intermediate_state_ptr.get());
+
+jbyteArray ctor_params_bytes;
+
+// Pushed frame will be popped when jni_frame goes
out-of-scope.
+RE
[GitHub] [incubator-doris] chenlinzhong commented on pull request #9804: [Feature] compaction quickly for small data import #9791
chenlinzhong commented on PR #9804: URL: https://github.com/apache/incubator-doris/pull/9804#issuecomment-1150624270 > will there be any side effect for normal compaction performance, if there are many stream load and with not small data? maybe， because small compaction use the same lock with cc compaction, so small compaction may stop cc compaction for one tablet at this round , but small compaction finished very quickly usually less then 1s , cc compaction maybe excute next round -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #9804: [Feature] compaction quickly for small data import #9791
caiconghui commented on PR #9804: URL: https://github.com/apache/incubator-doris/pull/9804#issuecomment-1150625402 > > will there be any side effect for normal compaction performance, if there are many stream load and with not small data? > > maybe， because small compaction use the same lock with cc compaction, so small compaction may stop cc compaction for one tablet at this round , but small compaction finished very quickly usually less then 1s , cc compaction maybe excute next round so，it should be optional choice for user to use this optimization, may be better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10013: [Feature] Support outfile on vectorized engine
yiguolei commented on code in PR #10013:
URL: https://github.com/apache/incubator-doris/pull/10013#discussion_r893038957
##
be/src/vec/runtime/vfile_result_writer.cpp:
##
@@ -0,0 +1,510 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/runtime/vfile_result_writer.h"
+
+#include "common/consts.h"
+#include "exec/broker_writer.h"
+#include "exec/hdfs_reader_writer.h"
+#include "exec/local_file_writer.h"
+#include "exec/s3_writer.h"
+#include "exprs/expr_context.h"
+#include "gutil/strings/numbers.h"
+#include "gutil/strings/substitute.h"
+#include "runtime/buffer_control_block.h"
+#include "runtime/descriptors.h"
+#include "runtime/large_int_value.h"
+#include "runtime/raw_value.h"
+#include "runtime/row_batch.h"
+#include "runtime/runtime_state.h"
+#include "runtime/string_value.h"
+#include "runtime/tuple_row.h"
+#include "service/backend_options.h"
+#include "util/file_utils.h"
+#include "util/mysql_global.h"
+#include "util/mysql_row_buffer.h"
+#include "vec/common/arena.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+const size_t VFileResultWriter::OUTSTREAM_BUFFER_SIZE_BYTES = 1024 * 1024;
+using doris::operator<<;
+
+VFileResultWriter::VFileResultWriter(const ResultFileOptions* file_opts,
+ const TStorageBackendType::type
storage_type,
+ const TUniqueId fragment_instance_id,
+ const std::vector&
output_expr_ctxs,
+ RuntimeProfile* parent_profile,
BufferControlBlock* sinker,
+ Block* output_block, bool
output_object_data,
+ const RowDescriptor&
output_row_descriptor)
+: _file_opts(file_opts),
+ _storage_type(storage_type),
+ _fragment_instance_id(fragment_instance_id),
+ _output_expr_ctxs(output_expr_ctxs),
+ _parent_profile(parent_profile),
+ _sinker(sinker),
+ _output_block(output_block),
+ _output_row_descriptor(output_row_descriptor) {
+_output_object_data = output_object_data;
+}
+
+VFileResultWriter::~VFileResultWriter() {
+_close_file_writer(true);
+}
+
+Status VFileResultWriter::init(RuntimeState* state) {
+_state = state;
+_init_profile();
+return _create_next_file_writer();
+}
+
+void VFileResultWriter::_init_profile() {
+RuntimeProfile* profile =
_parent_profile->create_child("VFileResultWriter", true, true);
+_append_row_batch_timer = ADD_TIMER(profile, "AppendBatchTime");
+_convert_tuple_timer = ADD_CHILD_TIMER(profile, "TupleConvertTime",
"AppendBatchTime");
+_file_write_timer = ADD_CHILD_TIMER(profile, "FileWriteTime",
"AppendBatchTime");
+_writer_close_timer = ADD_TIMER(profile, "FileWriterCloseTime");
+_written_rows_counter = ADD_COUNTER(profile, "NumWrittenRows",
TUnit::UNIT);
+_written_data_bytes = ADD_COUNTER(profile, "WrittenDataBytes",
TUnit::BYTES);
+}
+
+Status VFileResultWriter::_create_success_file() {
+std::string file_name;
+RETURN_IF_ERROR(_get_success_file_name(&file_name));
+RETURN_IF_ERROR(_create_file_writer(file_name));
+return _close_file_writer(true, true);
+}
+
+Status VFileResultWriter::_get_success_file_name(std::string* file_name) {
+std::stringstream ss;
+ss << _file_opts->file_path << _file_opts->success_file_name;
+*file_name = ss.str();
+if (_storage_type == TStorageBackendType::LOCAL) {
+// For local file writer, the file_path is a local dir.
+// Here we do a simple security verification by checking whether the
file exists.
+// Because the file path is currently arbitrarily specified by the
user,
+// Doris is not responsible for ensuring the correctness of the path.
+// This is just to prevent overwriting the existing file.
+if (FileUtils::check_exist(*file_name)) {
+return Status::InternalError("File already exists: " + *file_name +
+ ". Host: " +
BackendOptions::get_localhost());
+}
+}
+
+return Status::OK();
+}
+
+Status VFileResul
[GitHub] [incubator-doris] icedrugs89 opened a new issue, #10022: String Type should not be used in key column
icedrugs89 opened a new issue, #10022: URL: https://github.com/apache/incubator-doris/issues/10022 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0-preview2 ### What's Wrong? 1、1.0.0版本创建表时，使用更新模型UNIQUE KEY的key类型为String是可以创建成功的，升级到1.1.0-preview2后，提示如下错误： https://user-images.githubusercontent.com/100941547/172758441-7c9c68a6-5f0a-4e4b-bc9c-a2821711228c.png";> ### What You Expected? UNIQUE KEY更新模型可以使用字符串作为索引列 ### How to Reproduce? CREATE TABLE `test02`( `product_def_no` String COMMENT '产品定义号', `id` varchar(100) COMMENT '物理主键', `cust_source` varchar(100) COMMENT '客户渠道', `cust_seg` String COMMENT '客户风险等级', `pday` String ) UNIQUE KEY(product_def_no,id,cust_source) DISTRIBUTED BY HASH(product_def_no) BUCKETS 1 properties( "replication_allocation"="tag.location.group_import:1, tag.location.group_normal_read:2" ) ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] icedrugs89 commented on issue #10022: String Type should not be used in key column
icedrugs89 commented on issue #10022: URL: https://github.com/apache/incubator-doris/issues/10022#issuecomment-1150633806 `product_def_no` String类型 修改为varchar(100)可以解决 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
924060929 commented on code in PR #9993:
URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r893038269
##
fe/fe-core/src/main/java/org/apache/doris/nereids/operators/AbstractOperator.java:
##
@@ -17,13 +17,18 @@
package org.apache.doris.nereids.operators;
+import org.apache.doris.nereids.PlanOperatorVisitor;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+
import java.util.Objects;
/**
* Abstract class for all concrete operator.
*/
public abstract class AbstractOperator>
implements Operator {
protected final OperatorType type;
+protected long limited;
Review Comment:
```suggestion
protected final long limited;
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/operators/AbstractOperator.java:
##
@@ -33,4 +38,17 @@ public AbstractOperator(OperatorType type) {
public OperatorType getType() {
return type;
}
+
+public R accept(PlanOperatorVisitor visitor, PhysicalPlan physicalPlan, C context) {
+return null;
+}
+
+public long getLimited() {
+return limited;
+}
+
+public void setLimited(long limited) {
Review Comment:
Operator and Plan's properties should be immutable. So no setter.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/PlanOperatorVisitor.java:
##
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.nereids.operators.plans.physical.PhysicalOperator;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+
+@SuppressWarnings("rawtypes")
+public abstract class PlanOperatorVisitor {
+public abstract R visit(PhysicalPlan physicalPlan, C context);
+
+public R visitPhysicalAggregationPlan(PhysicalPlan physicalPlan,
Review Comment:
```suggestion
public R visitPhysicalAggregation(PhysicalPlan aggregationPlan, C context);
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/operators/plans/physical/PhysicalAggregation.java:
##
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.operators.plans.physical;
+
+import org.apache.doris.nereids.operators.OperatorType;
+import org.apache.doris.nereids.operators.plans.AggPhase;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+
+import java.util.List;
+
+public class PhysicalAggregation extends
PhysicalUnaryOperator {
+
+private List groupByExprList;
Review Comment:
operator's properties should be final
##
fe/fe-core/src/main/java/org/apache/doris/nereids/operators/plans/physical/PhysicalHashJoin.java:
##
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is dis
[GitHub] [incubator-doris] wangshuo128 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
wangshuo128 commented on code in PR #9993:
URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r893045887
##
fe/fe-core/src/main/java/org/apache/doris/nereids/PlanOperatorVisitor.java:
##
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.nereids.operators.plans.physical.PhysicalOperator;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+
+@SuppressWarnings("rawtypes")
+public abstract class PlanOperatorVisitor {
Review Comment:
Is this class a generic visitor for all the operators, i.e., both logical
and physical, or it's a dedicated visitor only for physical operators?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on a diff in pull request #9993: [feature](nereids) Plan Translator
924060929 commented on code in PR #9993:
URL: https://github.com/apache/incubator-doris/pull/9993#discussion_r893045453
##
fe/fe-core/src/main/java/org/apache/doris/nereids/PlanOperatorVisitor.java:
##
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids;
+
+import org.apache.doris.nereids.operators.plans.physical.PhysicalOperator;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+
+@SuppressWarnings("rawtypes")
+public abstract class PlanOperatorVisitor {
+public abstract R visit(PhysicalPlan physicalPlan, C context);
Review Comment:
PhysicalPlan should be Plan.
```
public abstract R visit(Plan plan, C
context);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a diff in pull request #9971: [Bugfix] Fix the bug data balance causes tablet loss
morningman commented on code in PR #9971:
URL: https://github.com/apache/incubator-doris/pull/9971#discussion_r893042637
##
fe/fe-core/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1655,4 +1655,11 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = true)
public static int backend_rpc_timeout_ms = 6; // 1 min
+/**
+ * If set to TRUE, FE will rebalance tablets on BE frequently.
+ * It's used to test the reliability in single replica case when tablet
scheduling are frequent.
+ * Default is false.
+ */
+@ConfField
Review Comment:
```suggestion
@ConfField(mutable = false, masterOnly = true)
```
##
be/src/olap/tablet_manager.cpp:
##
@@ -471,8 +472,36 @@ Status TabletManager::_drop_tablet_unlocked(TTabletId
tablet_id, bool keep_files
<< "tablet_id=" << tablet_id;
return Status::OK();
}
+if (to_drop_tablet->replica_id() != replica_id && replica_id != 0) {
+LOG(WARNING) << "fail to drop tablet because replica_id not match. "
+ << "tablet_id=" << tablet_id;
Review Comment:
print `replica_id` as well.
##
fe/fe-core/src/main/java/org/apache/doris/common/Config.java:
##
@@ -1655,4 +1655,11 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = true)
public static int backend_rpc_timeout_ms = 6; // 1 min
+/**
+ * If set to TRUE, FE will rebalance tablets on BE frequently.
+ * It's used to test the reliability in single replica case when tablet
scheduling are frequent.
+ * Default is false.
+ */
+@ConfField
Review Comment:
And please add comment to explain more about how this works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9856: [feature-wip](array-type) Add array type support for vectorized parquet-orc scanner
morningman merged PR #9856: URL: https://github.com/apache/incubator-doris/pull/9856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature-wip](array-type) Add array type support for vectorized parquet-orc scanner (#9856)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 19bc14cf8d [feature-wip](array-type) Add array type support for
vectorized parquet-orc scanner (#9856)
19bc14cf8d is described below
commit 19bc14cf8d96bd7990c0ad52d9356cf7e686a441
Author: yinzhijian <373141...@qq.com>
AuthorDate: Thu Jun 9 12:11:47 2022 +0800
[feature-wip](array-type) Add array type support for vectorized parquet-orc
scanner (#9856)
Only support one level array now.
for example:
- nullable(array(nullable(tinyint))) is **support**.
- nullable(array(nullable(array(xx))) is **not support**.
---
be/src/exec/arrow/orc_reader.h | 4 +-
be/src/exec/base_scanner.cpp | 1 +
be/src/vec/data_types/data_type.cpp| 4 +
be/src/vec/data_types/data_type.h | 2 +
be/src/vec/data_types/data_type_array.cpp | 91 ++-
be/src/vec/data_types/data_type_array.h| 4 +
be/src/vec/data_types/data_type_date.cpp | 12 ++
be/src/vec/data_types/data_type_date.h | 1 +
be/src/vec/data_types/data_type_date_time.cpp | 12 ++
be/src/vec/data_types/data_type_date_time.h| 2 +
be/src/vec/data_types/data_type_decimal.cpp| 12 ++
be/src/vec/data_types/data_type_decimal.h | 1 +
be/src/vec/data_types/data_type_factory.cpp| 5 +
be/src/vec/data_types/data_type_factory.hpp| 43 ---
be/src/vec/data_types/data_type_nullable.cpp | 30 +
be/src/vec/data_types/data_type_nullable.h | 2 +
be/src/vec/data_types/data_type_number_base.cpp| 28 +
be/src/vec/data_types/data_type_number_base.h | 1 +
be/src/vec/data_types/data_type_string.cpp | 6 +
be/src/vec/data_types/data_type_string.h | 1 +
be/src/vec/exec/varrow_scanner.cpp | 2 +-
be/src/vec/functions/function_cast.h | 110 +-
be/src/vec/utils/arrow_column_to_doris_column.cpp | 47 +++-
be/src/vec/utils/arrow_column_to_doris_column.h| 2 +-
.../utils/arrow_column_to_doris_column_test.cpp| 125 -
25 files changed, 511 insertions(+), 37 deletions(-)
diff --git a/be/src/exec/arrow/orc_reader.h b/be/src/exec/arrow/orc_reader.h
index 5213a18dcf..dd7853efe7 100644
--- a/be/src/exec/arrow/orc_reader.h
+++ b/be/src/exec/arrow/orc_reader.h
@@ -29,7 +29,7 @@
#include "exec/arrow/arrow_reader.h"
namespace doris {
-// Reader of orc file
+// Reader of ORC file
class ORCReaderWrap final : public ArrowReaderWrap {
public:
ORCReaderWrap(FileReader* file_reader, int64_t batch_size, int32_t
num_of_columns_from_file);
@@ -48,4 +48,4 @@ private:
bool _cur_file_eof; // is read over?
};
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git a/be/src/exec/base_scanner.cpp b/be/src/exec/base_scanner.cpp
index 005e64c703..11247a860c 100644
--- a/be/src/exec/base_scanner.cpp
+++ b/be/src/exec/base_scanner.cpp
@@ -329,6 +329,7 @@ Status
BaseScanner::_materialize_dest_block(vectorized::Block* dest_block) {
// PT1 => dest primitive type
RETURN_IF_ERROR(ctx->execute(&_src_block, &result_column_id));
auto column_ptr = _src_block.get_by_position(result_column_id).column;
+DCHECK(column_ptr != nullptr);
// because of src_slot_desc is always be nullable, so the column_ptr
after do dest_expr
// is likely to be nullable
diff --git a/be/src/vec/data_types/data_type.cpp
b/be/src/vec/data_types/data_type.cpp
index 0cffc102ff..414959a517 100644
--- a/be/src/vec/data_types/data_type.cpp
+++ b/be/src/vec/data_types/data_type.cpp
@@ -82,6 +82,10 @@ std::string IDataType::to_string(const IColumn& column,
size_t row_num) const {
LOG(FATAL) << fmt::format("Data type {} to_string not implement.",
get_name());
return "";
}
+Status IDataType::from_string(ReadBuffer& rb, IColumn* column) const {
+LOG(FATAL) << fmt::format("Data type {} from_string not implement.",
get_name());
+return Status::OK();
+}
void IDataType::insert_default_into(IColumn& column) const {
column.insert_default();
diff --git a/be/src/vec/data_types/data_type.h
b/be/src/vec/data_types/data_type.h
index 5e49fa90c6..6a4dc16899 100644
--- a/be/src/vec/data_types/data_type.h
+++ b/be/src/vec/data_types/data_type.h
@@ -28,6 +28,7 @@
#include "vec/common/cow.h"
#include "vec/common/string_buffer.hpp"
#include "vec/core/types.h"
+#include "vec/io/reader_buffer.h"
namespace doris {
class PBlock;
@@ -70,6 +71,7 @@ public:
virtual void to_string(const IColumn& column, size_t row_num,
BufferWritable& ostr) const;
virtual std::string to_string(const IColumn& column, size_t row_num) const;
+
[GitHub] [incubator-doris] morningman closed issue #9872: [Bug] query hudi table failed
morningman closed issue #9872: [Bug] query hudi table failed URL: https://github.com/apache/incubator-doris/issues/9872 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9873: [bug][hudi] use lowerCase to get hudi fileFormatType
morningman merged PR #9873: URL: https://github.com/apache/incubator-doris/pull/9873 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix][hudi] use lowerCase to get hudi fileFormatType (#9873)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 050cbba6e5 [fix][hudi] use lowerCase to get hudi fileFormatType (#9873)
050cbba6e5 is described below
commit 050cbba6e5a6569a86a48a2bcac455815241ff64
Author: dujl
AuthorDate: Thu Jun 9 12:13:02 2022 +0800
[fix][hudi] use lowerCase to get hudi fileFormatType (#9873)
use lowerCase of inputFormatName to get hudi fileFormatType
---
.../main/java/org/apache/doris/planner/HudiScanNode.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/HudiScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/HudiScanNode.java
index 756441f396..909f5cbe6d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/HudiScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/HudiScanNode.java
@@ -299,13 +299,13 @@ public class HudiScanNode extends BrokerScanNode {
hdfsParams.setFsName(fsName);
Log.debug("Hudi path's host is " + fsName);
-TFileFormatType formatType = null;
-if (this.inputFormatName.toUpperCase(Locale.ROOT).contains("parquet"))
{
-formatType = TFileFormatType.FORMAT_PARQUET;
-} else if
(this.inputFormatName.toUpperCase(Locale.ROOT).contains("orc")) {
-formatType = TFileFormatType.FORMAT_ORC;
+TFileFormatType fileFormatType = null;
+if (this.inputFormatName.toLowerCase().contains("parquet")) {
+fileFormatType = TFileFormatType.FORMAT_PARQUET;
+} else if
(this.inputFormatName.toLowerCase(Locale.ROOT).contains("orc")) {
+fileFormatType = TFileFormatType.FORMAT_ORC;
} else {
-throw new UserException("unsupported hudi table type [" +
this.inputFormatName + "].");
+throw new UserException("Unsupported hudi table format [" +
this.inputFormatName + "].");
}
ParamCreateContext context = getParamCreateContexts().get(0);
@@ -317,7 +317,7 @@ public class HudiScanNode extends BrokerScanNode {
getPartitionKeys());
int numberOfColumnsFromFile = context.slotDescByName.size() -
partitionValuesFromPath.size();
-TBrokerRangeDesc rangeDesc = createBrokerRangeDesc(fileSplit,
formatType,
+TBrokerRangeDesc rangeDesc = createBrokerRangeDesc(fileSplit,
fileFormatType,
partitionValuesFromPath, numberOfColumnsFromFile,
brokerDesc);
rangeDesc.setHdfsParams(hdfsParams);
rangeDesc.setReadByColumnDef(true);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [test] add bitmap index regression test (#10008)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 9bc82542fd [test] add bitmap index regression test (#10008) 9bc82542fd is described below commit 9bc82542fdb459677ea1aaeacaceec6fd5608985 Author: zy-kkk <815574...@qq.com> AuthorDate: Thu Jun 9 12:19:07 2022 +0800 [test] add bitmap index regression test (#10008) --- regression-test/data/index/test_bitmap_index.out | 91 .../suites/index/test_bitmap_index.groovy | 236 + .../suites/rollup/test_materialized_view.groovy| 58 +++-- .../suites/rollup/test_rollup_agg.groovy | 45 ++-- .../schema_change/test_alter_table_column.groovy | 129 +-- 5 files changed, 450 insertions(+), 109 deletions(-) diff --git a/regression-test/data/index/test_bitmap_index.out b/regression-test/data/index/test_bitmap_index.out new file mode 100644 index 00..f9962c5bfc --- /dev/null +++ b/regression-test/data/index/test_bitmap_index.out @@ -0,0 +1,91 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !sql -- +k1 TINYINT Yes true\N +k2 SMALLINTYes true\N +k3 INT Yes true\N +k4 BIGINT Yes false \N NONE +k5 CHAR(1) Yes false \N NONE +k6 VARCHAR(1) Yes false \N NONE +k7 DATEYes false \N NONE +k8 DATETIMEYes false \N NONE +k9 LARGEINTYes false \N NONE +k10DECIMAL(9,0)Yes false \N NONE +k11BOOLEAN Yes false \N NONE + +-- !sql -- +default_cluster:regression_test.test_bitmap_index_dup index1 k1 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index2 k2 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index3 k3 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index4 k4 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index5 k5 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index6 k6 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index7 k7 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index8 k8 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index9 k9 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index10 k10 BITMAP +default_cluster:regression_test.test_bitmap_index_dup index11 k11 BITMAP + +-- !sql -- +1 1 1 1 1 1 2022-05-31 2022-05-31T10:001 1 true + +-- !sql -- +k1 TINYINT Yes true\N +k2 SMALLINTYes true\N +k3 INT Yes true\N +k4 BIGINT Yes true\N +k5 CHAR(1) Yes true\N +k6 VARCHAR(1) Yes true\N +k7 DATEYes true\N +k8 DATETIMEYes true\N +k9 LARGEINTYes true\N +k10DECIMAL(9,0)Yes true\N +k11BOOLEAN Yes true\N +v1 INT Yes false \N SUM + +-- !sql -- +default_cluster:regression_test.test_bitmap_index_agg index1 k1 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index2 k2 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index3 k3 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index4 k4 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index5 k5 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index6 k6 BITMAP +default_cluster:regression_test.test_bitmap_index_agg index7 k7 BITMA
[GitHub] [incubator-doris] morningman merged pull request #10008: [test]Fix test case infinite loop
morningman merged PR #10008: URL: https://github.com/apache/incubator-doris/pull/10008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #10016: [enhancement]Cache parsed page data for bitshuffle encoding
yiguolei commented on code in PR #10016:
URL: https://github.com/apache/incubator-doris/pull/10016#discussion_r893055097
##
be/src/olap/rowset/segment_v2/parsed_page.h:
##
@@ -67,17 +70,38 @@ struct ParsedPage {
return Status::OK();
}
-~ParsedPage() {
-delete data_decoder;
-data_decoder = nullptr;
+ParsedPage() = default;
+
+ParsedPage(const ParsedPage& other) {
+page_handle = other.page_handle;
+has_null = other.has_null;
+null_decoder = other.null_decoder;
+null_bitmap = other.null_bitmap;
+other.data_decoder->clone_for_cache(data_decoder);
+first_ordinal = other.first_ordinal;
+num_rows = other.num_rows;
+first_array_item_ordinal = other.first_array_item_ordinal;
+page_pointer = other.page_pointer;
+page_index = other.page_index;
+offset_in_page = other.offset_in_page;
}
-PageHandle page_handle;
+ParsedPage& operator=(ParsedPage&& other) {
+this->~ParsedPage();
+new (this) ParsedPage(other);
+return *this;
+}
+
+~ParsedPage() { data_decoder = nullptr; }
Review Comment:
这么写的目的是啥？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
