[GitHub] [doris] Henry2SS opened a new issue, #11779: [Bug](docs of ALTER SYSTEM) make the docs more accurate
Henry2SS opened a new issue, #11779: URL: https://github.com/apache/doris/issues/11779 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? it should be `heartbeat_port` rather than `9050`. ### What You Expected? it should be `heartbeat_port` rather than `9050`. ### How to Reproduce? show alter system ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Henry2SS opened a new pull request, #11780: [Enhancement](docs) fix ALTER SYSTEM docs
Henry2SS opened a new pull request, #11780: URL: https://github.com/apache/doris/pull/11780 # Proposed changes Issue Number: close #11779 ## Problem summary fix ALTER SYSTEM docs ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [x] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Lchangliang opened a new pull request, #11781: [feature](light-schema-change) add a http interface for flink-doris-connector
Lchangliang opened a new pull request, #11781: URL: https://github.com/apache/doris/pull/11781 # Proposed changes Issue Number: close #xxx ## Problem summary The http interface is use for flink-doris-connector to sync ddl command from mysql to doris. flink-doris-connector need to check the downstream table whether enable light schema change. If true, connector can sync the add/drop column command to doris. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Lchangliang closed pull request #11781: [feature](light-schema-change) add a http interface for flink-doris-connector
Lchangliang closed pull request #11781: [feature](light-schema-change) add a http interface for flink-doris-connector URL: https://github.com/apache/doris/pull/11781 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #11782: [fix](agg)disallow group by bitmap or hll data type
starocean999 opened a new pull request, #11782: URL: https://github.com/apache/doris/pull/11782 # Proposed changes disallow "select distinct bitmap, hll from xxx", because in some case, the FE will introduce a agg node with group by bitmap or hll exprs to implement the "distinct" operator. But group by bitmap or hll data type are not supported in BE. ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11769: [feature-wip](parquet-reader) parquet physical type to doris logical type
morningman merged PR #11769: URL: https://github.com/apache/doris/pull/11769 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](parquet-reader) parquet physical type to doris logical type (#11769)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0b9bfd15b7 [feature-wip](parquet-reader) parquet physical type to
doris logical type (#11769)
0b9bfd15b7 is described below
commit 0b9bfd15b7ca45bcf5abe17accfcd11333a1229a
Author: Ashin Gau
AuthorDate: Mon Aug 15 16:08:11 2022 +0800
[feature-wip](parquet-reader) parquet physical type to doris logical type
(#11769)
Two improvements have been added:
1. Translate parquet physical type into doris logical type.
2. Decode parquet column chunk into doris ColumnPtr, and add unit tests to
show how to use related API.
---
be/src/vec/exec/format/parquet/parquet_common.cpp | 100 +++--
be/src/vec/exec/format/parquet/parquet_common.h| 81 ---
.../parquet/vparquet_column_chunk_reader.cpp | 49 +--
.../format/parquet/vparquet_column_chunk_reader.h | 17 ++-
be/test/vec/exec/parquet/parquet_thrift_test.cpp | 158 -
5 files changed, 255 insertions(+), 150 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/parquet_common.cpp
b/be/src/vec/exec/format/parquet/parquet_common.cpp
index 1d80676b9b..082d0fc57d 100644
--- a/be/src/vec/exec/format/parquet/parquet_common.cpp
+++ b/be/src/vec/exec/format/parquet/parquet_common.cpp
@@ -18,34 +18,36 @@
#include "parquet_common.h"
#include "util/coding.h"
+#include "vec/data_types/data_type_nullable.h"
namespace doris::vectorized {
-Status Decoder::getDecoder(tparquet::Type::type type, tparquet::Encoding::type
encoding,
- std::unique_ptr& decoder) {
+#define FOR_LOGICAL_NUMERIC_TYPES(M) \
+M(TypeIndex::Int32, Int32) \
+M(TypeIndex::UInt32, UInt32) \
+M(TypeIndex::Int64, Int64) \
+M(TypeIndex::UInt64, UInt64) \
+M(TypeIndex::Float32, Float32) \
+M(TypeIndex::Float64, Float64)
+
+Status Decoder::get_decoder(tparquet::Type::type type,
tparquet::Encoding::type encoding,
+std::unique_ptr& decoder) {
switch (encoding) {
case tparquet::Encoding::PLAIN:
switch (type) {
case tparquet::Type::BOOLEAN:
decoder.reset(new BoolPlainDecoder());
break;
-case tparquet::Type::INT32:
-decoder.reset(new PlainDecoder());
+case tparquet::Type::BYTE_ARRAY:
+decoder.reset(new ByteArrayPlainDecoder());
break;
+case tparquet::Type::INT32:
case tparquet::Type::INT64:
-decoder.reset(new PlainDecoder());
-break;
+case tparquet::Type::INT96:
case tparquet::Type::FLOAT:
-decoder.reset(new PlainDecoder());
-break;
case tparquet::Type::DOUBLE:
-decoder.reset(new PlainDecoder());
-break;
-case tparquet::Type::BYTE_ARRAY:
-decoder.reset(new BAPlainDecoder());
-break;
case tparquet::Type::FIXED_LEN_BYTE_ARRAY:
-decoder.reset(new FixedLengthBAPlainDecoder());
+decoder.reset(new PlainDecoder(type));
break;
default:
return Status::InternalError("Unsupported plain type {} in parquet
decoder",
@@ -60,34 +62,28 @@ Status Decoder::getDecoder(tparquet::Type::type type,
tparquet::Encoding::type e
return Status::OK();
}
-Status Decoder::decode_values(ColumnPtr& doris_column, size_t num_values) {
+Status Decoder::decode_values(ColumnPtr& doris_column, DataTypePtr& data_type,
size_t num_values) {
CHECK(doris_column->is_nullable());
auto* nullable_column = reinterpret_cast(
(*std::move(doris_column)).mutate().get());
MutableColumnPtr data_column = nullable_column->get_nested_column_ptr();
-return _decode_values(data_column, num_values);
+return decode_values(data_column, data_type, num_values);
}
-Status FixedLengthBAPlainDecoder::decode_values(Slice& slice, size_t
num_values) {
+Status PlainDecoder::decode_values(Slice& slice, size_t num_values) {
size_t to_read_bytes = _type_length * num_values;
if (UNLIKELY(_offset + to_read_bytes > _data->size)) {
return Status::IOError("Out-of-bounds access in parquet data decoder");
}
-// insert '\0' into the end of each binary
-if (UNLIKELY(to_read_bytes + num_values > slice.size)) {
+if (UNLIKELY(to_read_bytes > slice.size)) {
return Status::IOError("Slice does not have enough space to write out
the decoding data");
}
-uint32_t slice_offset = 0;
-for (int i = 0; i < num_values; ++i) {
-memcpy(slice.data + slice_offset, _data->data + _offset, _type_length);
-slice_offset += _type_length + 1;
-slice.data[slice_offset - 1] = '\0';
-_offset += _type_length;
-}
+memcpy(slice.data, _data-
[GitHub] [doris] starocean999 commented on pull request #11751: [enhancement](memory) Fix too much cache leads to less memory available for queries
starocean999 commented on PR #11751: URL: https://github.com/apache/doris/pull/11751#issuecomment-1214741165 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus opened a new pull request, #11783: [regression-test](bitmap) Add regression case for some bitmap funcations
smallhibiscus opened a new pull request, #11783: URL: https://github.com/apache/doris/pull/11783 # Proposed changes Issue Number: close #xxx ## Problem summary Add regression case for some bitmap funcations. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 opened a new pull request, #11784: [optimization](array-type) support the decimal/datetime as the nest t…
carlvinhust2012 opened a new pull request, #11784: URL: https://github.com/apache/doris/pull/11784 # Proposed changes 1. This pr is used to support the support the decimal/datetime as the nest type of array in print_value. Issue Number: close #7570 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #11785: [Enhancement](array-type) enable_array_type flag update
cambyzju opened a new pull request, #11785: URL: https://github.com/apache/doris/pull/11785 # Proposed changes Issue Number: close #xxx ## Problem summary At the first time, we add enable_array_type flag as a session variable inside FE(https://github.com/apache/doris/pull/9921). If users want use ARRAY, they need add a SQL (set enable_array_type=true) to open ARRAY. But some release versions later, we may open enable_array_type as default, even we remove these codes, then users have to remove the SQL(set enable_array_type=true). It is not friendly to users. So now we update the flag enable_array_type into fe.conf. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new issue, #11786: [Bug] BE crash with query `select avg(distinct k1), avg(k2) from test2;`
morningman opened a new issue, #11786: URL: https://github.com/apache/doris/issues/11786 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` F0815 16:39:54.389199 8227 vaggregation_node.cpp:613] Check failed: _aggregate_evaluators[i]->input_exprs_ctxs().size() == 1 && _aggregate_evaluators[i]->input_exprs_ctx s()[0]->root()->is_slot_ref() ``` ### What You Expected? No error ### How to Reproduce? ``` CREATE TABLE `test2` ( `k0` boolean NULL, `k1` tinyint(4) NULL, `k2` smallint(6) NULL, `k3` int(11) NULL, `k4` bigint(20) NULL, `k5` decimal(9, 3) NULL, `k6` char(5) NULL, `k10` date NULL, `k11` datetime NULL, `k7` varchar(20) NULL, `k8` double MAX NULL, `k9` float SUM NULL, `k12` text REPLACE_IF_NOT_NULL NULL, `k13` largeint(40) REPLACE NULL ) ENGINE=OLAP AGGREGATE KEY(`k0`, `k1`, `k2`, `k3`, `k4`, `k5`, `k6`, `k10`, `k11`, `k7`) COMMENT 'OLAP' DISTRIBUTED BY HASH(`k1`) BUCKETS 5 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ); insert into test2 values(1,2,3,4,5,6,"7",8,9,10,11,12,13,14); select avg(distinct k1), avg(k2) from test2; ``` ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on issue #11786: [Bug] BE crash with query `select avg(distinct k1), avg(k2) from test2;`
BiteThet commented on issue #11786: URL: https://github.com/apache/doris/issues/11786#issuecomment-1214766000 seems same with https://github.com/apache/doris/issues/10541 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] kpfly commented on pull request #11783: [regression-test](bitmap) Add regression case for some bitmap funcations
kpfly commented on PR #11783: URL: https://github.com/apache/doris/pull/11783#issuecomment-1214766613 bitmap funcations -》 bitmap functions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11783: [regression-test](bitmap) Add regression case for some bitmap funcations
github-actions[bot] commented on PR #11783: URL: https://github.com/apache/doris/pull/11783#issuecomment-1214771074 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11783: [regression-test](bitmap) Add regression case for some bitmap funcations
github-actions[bot] commented on PR #11783: URL: https://github.com/apache/doris/pull/11783#issuecomment-1214771123 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11602: [fix](array-type) disable cast function to array type on origin exec engine.
github-actions[bot] commented on PR #11602: URL: https://github.com/apache/doris/pull/11602#issuecomment-1214773022 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11602: [fix](array-type) disable cast function to array type on origin exec engine.
github-actions[bot] commented on PR #11602: URL: https://github.com/apache/doris/pull/11602#issuecomment-1214773062 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11585: [fix](array-type) Fix incorrect in function-set for array type
github-actions[bot] commented on PR #11585: URL: https://github.com/apache/doris/pull/11585#issuecomment-1214773481 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11585: [fix](array-type) Fix incorrect in function-set for array type
github-actions[bot] commented on PR #11585: URL: https://github.com/apache/doris/pull/11585#issuecomment-1214773460 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X commented on a diff in pull request #11785: [Enhancement](array-type) enable_array_type flag update
spaces-X commented on code in PR #11785:
URL: https://github.com/apache/doris/pull/11785#discussion_r945543674
##
fe/fe-core/src/test/java/org/apache/doris/analysis/ColumnDefTest.java:
##
@@ -133,7 +134,7 @@ public void testBooleanDefaultValue() throws
AnalysisException {
@Test
public void testArray() throws AnalysisException {
-ctx.getSessionVariable().setEnableArrayType(true);
+Config.enable_array_type = true;
Review Comment:
MayBe it is more reasonable to put this code in setUp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus commented on pull request #11783: [regression-test](bitmap) Add regression case for some bitmap functions
smallhibiscus commented on PR #11783: URL: https://github.com/apache/doris/pull/11783#issuecomment-1214772965 > bitmap funcations -》 bitmap functions thx，done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11680: [remote-udaf](optimize) Optimize RPC exception handling logic
github-actions[bot] commented on PR #11680: URL: https://github.com/apache/doris/pull/11680#issuecomment-1214774091 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11680: [remote-udaf](optimize) Optimize RPC exception handling logic
github-actions[bot] commented on PR #11680: URL: https://github.com/apache/doris/pull/11680#issuecomment-1214774120 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new issue, #11787: [Bug] BE crash with query `select * from (select 1 as a) b left join (select 2 as a) c using(a);`
morningman opened a new issue, #11787: URL: https://github.com/apache/doris/issues/11787 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/git/incubator-doris/be/src/common/signal_handler.h:420 1# 0x7F867FEFF400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# 0x56457E238009 in /mnt/disk1/git/incubator-doris/output/be/lib/doris_be 5# 0x56457E22D61D at src/logging.cc:1650 6# google::LogMessage::SendToLog() at src/logging.cc:1607 7# google::LogMessage::Flush() at src/logging.cc:1477 8# google::LogMessageFatal::~LogMessageFatal() at src/logging.cc:2227 9# doris::vectorized::VTupleIsNullPredicate::VTupleIsNullPredicate(doris::TExprNode const&) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vtuple_is_null_predicate.cpp:32 10# doris::vectorized::VExpr::create_expr(doris::ObjectPool*, doris::TExprNode const&, doris::vectorized::VExpr**) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vexpr.cpp:154 11# doris::vectorized::VExpr::create_tree_from_thrift(doris::ObjectPool*, std::vector > const&, doris::vectorized::VExpr*, int*, doris::vectorized::VExpr**, doris::vectorized::VExprContext**) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vexpr.cpp:172 12# doris::vectorized::VExpr::create_tree_from_thrift(doris::ObjectPool*, std::vector > const&, doris::vectorized::VExpr*, int*, doris::vectorized::VExpr**, doris::vectorized::VExprContext**) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vexpr.cpp:184 13# doris::vectorized::VExpr::create_tree_from_thrift(doris::ObjectPool*, std::vector > const&, doris::vectorized::VExpr*, int*, doris::vectorized::VExpr**, doris::vectorized::VExprContext**) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vexpr.cpp:184 14# doris::vectorized::VExpr::create_expr_tree(doris::ObjectPool*, doris::TExpr const&, doris::vectorized::VExprContext**) at /mnt/disk1/git/incubator-doris/be/src/vec/exprs/vexpr.cpp:202 15# doris::vectorized::HashJoinNode::init(doris::TPlanNode const&, doris::RuntimeState*) at /mnt/disk1/git/incubator-doris/be/src/vec/exec/join/vhash_join_node.cpp:807 16# doris::ExecNode::create_tree_helper(doris::RuntimeState*, doris::ObjectPool*, std::vector > const&, doris::DescriptorTbl const&, doris::ExecNode*, int*, doris::ExecNode**) at /mnt/disk1/git/incubator-doris/be/src/exec/exec_node.cpp:356 17# doris::ExecNode::create_tree(doris::RuntimeState*, doris::ObjectPool*, doris::TPlan const&, doris::DescriptorTbl const&, doris::ExecNode**) at /mnt/disk1/git/incubator-doris/be/src/exec/exec_node.cpp:311 18# doris::PlanFragmentExecutor::prepare(doris::TExecPlanFragmentParams const&, doris::QueryFragmentsCtx*) at /mnt/disk1/git/incubator-doris/be/src/runtime/plan_fragment_executor.cpp:139 19# doris::FragmentExecState::prepare(doris::TExecPlanFragmentParams const&) at /mnt/disk1/git/incubator-doris/be/src/runtime/fragment_mgr.cpp:233 20# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function) at /mnt/disk1/git/incubator-doris/be/src/runtime/fragment_mgr.cpp:680 21# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&) at /mnt/disk1/git/incubator-doris/be/src/runtime/fragment_mgr.cpp:551 22# doris::PInternalServiceImpl::_exec_plan_fragment(std::__cxx11::basic_string, std::allocator > const&, doris::PFragmentRequestVersion, bool) at /mnt/disk1/git/incubator-doris/be/src/service/internal_service.cpp:377 23# doris::PInternalServiceImpl::exec_plan_fragment(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /mnt/disk1/git/incubator-doris/be/src/service/internal_service.cpp:197 24# doris::PInternalServiceImpl::exec_plan_fragment_prepare(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /mnt/disk1/git/incubator-doris/be/src/service/internal_service.cpp:209 25# doris::PBackendService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*) at /mnt/disk1/git/incubator-doris/gensrc/build/gen_cpp/internal_service.pb.cc:23707 26# brpc::policy::ProcessHttpRequest(brpc::InputMessageBase*) at /mnt/disk1/git/incubator-doris/thirdparty/src/incubator-brpc-1.1.0/src/brpc/policy/http_rpc_protocol.cpp:1537 27# brpc::ProcessInputMessage(void*) at /mnt/disk1/git/incubator-doris/thirdparty/src/incubator-brpc-1.1.0/src/brpc/input_messenger.cpp:149 28# brpc::InputMessen
[GitHub] [doris] github-actions[bot] commented on pull request #11752: [remote-udaf](sample) add some java demo
github-actions[bot] commented on PR #11752: URL: https://github.com/apache/doris/pull/11752#issuecomment-1214774679 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11752: [remote-udaf](sample) add some java demo
github-actions[bot] commented on PR #11752: URL: https://github.com/apache/doris/pull/11752#issuecomment-1214774711 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11761: [remote-udaf](sample) add some c++ demo
github-actions[bot] commented on PR #11761: URL: https://github.com/apache/doris/pull/11761#issuecomment-1214776003 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11761: [remote-udaf](sample) add some c++ demo
github-actions[bot] commented on PR #11761: URL: https://github.com/apache/doris/pull/11761#issuecomment-1214776029 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new issue, #11788: [Bug] build fail on build_ui
BiteThet opened a new issue, #11788: URL: https://github.com/apache/doris/issues/11788 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ```npm npm ERR! Could not resolve dependency: npm ERR! peer antd@"3.x" from @ant-design/compatible@1.1.2 npm ERR! node_modules/@ant-design/compatible npm ERR! @ant-design/compatible@"^1.0.2" from the root project ``` ### What You Expected? fix ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11789: [Bug](build) fix build fail on build_ui
BiteThet opened a new pull request, #11789: URL: https://github.com/apache/doris/pull/11789 # Proposed changes Issue Number: close #11788 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web opened a new pull request, #11790: [regression](largeint)doris largeint type execute where query, the result is incorrect
caoliang-web opened a new pull request, #11790: URL: https://github.com/apache/doris/pull/11790 ## Problem summary doris largeint type execute where query, the result is incorrect ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11778: [docs] Change wrong words
github-actions[bot] commented on PR #11778: URL: https://github.com/apache/doris/pull/11778#issuecomment-1214803812 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11778: [docs] Change wrong words
github-actions[bot] commented on PR #11778: URL: https://github.com/apache/doris/pull/11778#issuecomment-1214803948 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X commented on a diff in pull request #11782: [fix](agg)disallow group by bitmap or hll data type
spaces-X commented on code in PR #11782:
URL: https://github.com/apache/doris/pull/11782#discussion_r945560043
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java:
##
@@ -160,6 +161,15 @@ public void setPartitionExprs(List exprs) {
partitionExprs = exprs;
}
+private static void validateGroupingExprs(List groupingExprs) throws
AnalysisException {
Review Comment:
I think it is more reasonable to put this checking logic into
`GroupByClause`'s analyze phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X commented on a diff in pull request #11784: [optimization](array-type) support the decimal/datetime as the nest t…
spaces-X commented on code in PR #11784:
URL: https://github.com/apache/doris/pull/11784#discussion_r945570945
##
regression-test/suites/export/test_array_export.groovy:
##
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.Paths
+
+suite("test_array_export", "export") {
+// check whether the FE config 'enable_outfile_to_local' is true
+StringBuilder strBuilder = new StringBuilder()
+strBuilder.append("curl --location-trusted -u " + context.config.jdbcUser
+ ":" + context.config.jdbcPassword)
+strBuilder.append(" http://"; + context.config.feHttpAddress +
"/rest/v1/config/fe")
+
+String command = strBuilder.toString()
+def process = command.toString().execute()
+def code = process.waitFor()
+def err = IOGroovyMethods.getText(new BufferedReader(new
InputStreamReader(process.getErrorStream(;
+def out = process.getText()
+logger.info("Request FE Config: code=" + code + ", out=" + out + ", err="
+ err)
+assertEquals(code, 0)
+def response = parseJson(out.trim())
+assertEquals(response.code, 0)
+assertEquals(response.msg, "success")
+def configJson = response.data.rows
+boolean enableOutfileToLocal = false
+for (Object conf: configJson) {
+assert conf instanceof Map
+if (((Map) conf).get("Name").toLowerCase() ==
"enable_outfile_to_local") {
+enableOutfileToLocal = ((Map)
conf).get("Value").toLowerCase() == "true"
+}
+}
+if (!enableOutfileToLocal) {
+logger.warn("Please set enable_outfile_to_local to true to run
test_outfile")
+return
+}
+
+// define the table and out file path
+def tableName = "array_outfile_test"
+def outFilePath = """${context.file.parent}/tmp"""
+logger.warn("test_array_export the outFilePath=" + outFilePath)
+
+try {
+sql """ DROP TABLE IF EXISTS ${tableName} """
+sql """ set enable_array_type = true """
+
+result1 = sql """
+CREATE TABLE IF NOT EXISTS ${tableName} (
+ `k1` INT(11) NULL COMMENT "",
+ `k2` ARRAY NOT NULL COMMENT "",
+ `k3` ARRAY NOT NULL COMMENT "",
+ `k4` ARRAY NOT NULL COMMENT "",
+ `k5` ARRAY NOT NULL COMMENT "",
+ `k6` ARRAY NULL COMMENT "",
+ `k7` ARRAY NOT NULL COMMENT "",
+ `k8` ARRAY NOT NULL COMMENT "",
+ `k9` ARRAY NOT NULL COMMENT "",
+ `k10` ARRAY NOT NULL COMMENT "",
+ `k11` ARRAY NULL COMMENT ""
+) ENGINE=OLAP
+DUPLICATE KEY(`k1`)
+DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"storage_format" = "V2"
+)
+"""
+
+// DDL/DML return 1 row and 3 column, the only value is update row
count
+assertTrue(result1.size() == 1)
+assertTrue(result1[0].size() == 1)
+assertTrue(result1[0][0] == 0, "Create table should update 0 rows")
+
+sql """ INSERT INTO ${tableName} VALUES
+(1, [1, 2, 3], [32767, 32768, 32769], [65534, 65535,
65536], ['a', 'b', 'c'], ["hello", "world"],
+['2022-07-13'], ['2022-08-15 12:30:00'], [0.33, 0.67],
[3.1415926, 0.878787878], [4, 5.5, 6.67])
+"""
+
+sql """ INSERT INTO ${tableName} VALUES
+(2, [4, 5, 6], [32767, 32768, 32769], [65534, 65535,
65536], ['d', 'e', 'f'], ["good", "luck"],
+['2022-07-13'], ['2022-08-15 15:59:59'], [0.36,
0.77], [3.1415926, 0.878787878], [4, 5.555, 6.7])
+"""
+
+qt_select_default """ SELECT * FROM ${tableName} t ORDER BY k1; """
+
+// check outfile
+File path = new File(outFilePath)
+if (!path.exists()) {
+assert path.mkdirs()
+} else {
+throw new IllegalStateException("""${outFilePath} already exists!
""")
+
[GitHub] [doris] github-actions[bot] commented on pull request #11780: [Enhancement](docs) fix ALTER SYSTEM docs
github-actions[bot] commented on PR #11780: URL: https://github.com/apache/doris/pull/11780#issuecomment-1214829231 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11780: [Enhancement](docs) fix ALTER SYSTEM docs
github-actions[bot] commented on PR #11780: URL: https://github.com/apache/doris/pull/11780#issuecomment-1214829254 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #11785: [Enhancement](array-type) enable_array_type flag update
cambyzju commented on code in PR #11785:
URL: https://github.com/apache/doris/pull/11785#discussion_r945576878
##
fe/fe-core/src/test/java/org/apache/doris/analysis/ColumnDefTest.java:
##
@@ -133,7 +134,7 @@ public void testBooleanDefaultValue() throws
AnalysisException {
@Test
public void testArray() throws AnalysisException {
-ctx.getSessionVariable().setEnableArrayType(true);
+Config.enable_array_type = true;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed issue #11425: [Bug] [daily] delete where char > xxx, result error
morningman closed issue #11425: [Bug] [daily] delete where char > xxx, result error URL: https://github.com/apache/doris/issues/11425 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11754: [fix](delete) fix query result error after delete
morningman merged PR #11754: URL: https://github.com/apache/doris/pull/11754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](delete) fix query result error after delete (#11754)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0f75bd0e38 [fix](delete) fix query result error after delete (#11754)
0f75bd0e38 is described below
commit 0f75bd0e385ad582747cd027b683d145ecaf9a38
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Mon Aug 15 17:52:03 2022 +0800
[fix](delete) fix query result error after delete (#11754)
convert dictionary code for delete predicates.
---
be/src/olap/block_column_predicate.h | 13
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 39 ++
be/src/olap/rowset/segment_v2/segment_iterator.h | 24 +++--
.../delete_p0/test_segment_iterator_delete.out | 4 +++
.../delete_p0/test_segment_iterator_delete.groovy | 18 ++
5 files changed, 78 insertions(+), 20 deletions(-)
diff --git a/be/src/olap/block_column_predicate.h
b/be/src/olap/block_column_predicate.h
index 219a92c3aa..c2830c223b 100644
--- a/be/src/olap/block_column_predicate.h
+++ b/be/src/olap/block_column_predicate.h
@@ -43,6 +43,9 @@ public:
virtual void get_all_column_ids(std::set& column_id_set) const =
0;
+virtual void get_all_column_predicate(
+std::set& predicate_set) const = 0;
+
virtual uint16_t evaluate(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size) const {
return selected_size;
@@ -68,6 +71,10 @@ public:
column_id_set.insert(_predicate->column_id());
};
+void get_all_column_predicate(std::set&
predicate_set) const override {
+predicate_set.insert(_predicate);
+}
+
uint16_t evaluate(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size) const override;
void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size,
@@ -103,6 +110,12 @@ public:
}
};
+void get_all_column_predicate(std::set&
predicate_set) const override {
+for (auto child_block_predicate : _block_column_predicate_vec) {
+child_block_predicate->get_all_column_predicate(predicate_set);
+}
+}
+
protected:
std::vector _block_column_predicate_vec;
};
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index cc728bdbe7..b246352b97 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -751,6 +751,16 @@ void SegmentIterator::_vec_init_lazy_materialization() {
std::set del_cond_id_set;
_opts.delete_condition_predicates->get_all_column_ids(del_cond_id_set);
+std::set delete_predicate_set {};
+
_opts.delete_condition_predicates->get_all_column_predicate(delete_predicate_set);
+for (const auto predicate : delete_predicate_set) {
+if (PredicateTypeTraits::is_range(predicate->type())) {
+_delete_range_column_ids.push_back(predicate->column_id());
+} else if (PredicateTypeTraits::is_bloom_filter(predicate->type())) {
+_delete_bloom_filter_column_ids.push_back(predicate->column_id());
+}
+}
+
if (!_col_predicates.empty() || !del_cond_id_set.empty()) {
std::set short_cir_pred_col_id_set; // using set for
distinct cid
std::set vec_pred_col_id_set;
@@ -1199,6 +1209,35 @@ Status SegmentIterator::next_batch(vectorized::Block*
block) {
return Status::OK();
}
+void SegmentIterator::_convert_dict_code_for_predicate_if_necessary() {
+for (auto predicate : _short_cir_eval_predicate) {
+_convert_dict_code_for_predicate_if_necessary_impl(predicate);
+}
+
+for (auto predicate : _pre_eval_block_predicate) {
+_convert_dict_code_for_predicate_if_necessary_impl(predicate);
+}
+
+for (auto column_id : _delete_range_column_ids) {
+
_current_return_columns[column_id].get()->convert_dict_codes_if_necessary();
+}
+
+for (auto column_id : _delete_bloom_filter_column_ids) {
+
_current_return_columns[column_id].get()->generate_hash_values_for_runtime_filter();
+}
+}
+
+void SegmentIterator::_convert_dict_code_for_predicate_if_necessary_impl(
+ColumnPredicate* predicate) {
+auto& column = _current_return_columns[predicate->column_id()];
+auto* col_ptr = column.get();
+if (PredicateTypeTraits::is_range(predicate->type())) {
+col_ptr->convert_dict_codes_if_necessary();
+} else if (PredicateTypeTraits::is_bloom_filter(predicate->type())) {
+col_ptr->generate_hash_values_for_runtime_filter();
+}
+}
+
void SegmentIterator::_update_max_row(const vectorized::Block* block) {
_estimate_row_size = false;
auto avg_row_size = block->bytes() / block->ro
[GitHub] [doris] morningman merged pull request #11767: [fix](storage) Convert dict columns in delete_condition_predicates
morningman merged PR #11767: URL: https://github.com/apache/doris/pull/11767 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [fix](storage) Convert dict columns in delete_condition_predicates (#11754) (#11767)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new 8a5d9f587a [fix](storage) Convert dict columns in
delete_condition_predicates (#11754) (#11767)
8a5d9f587a is described below
commit 8a5d9f587a91a61ae9b5217de44c208425f3d894
Author: Jerry Hu
AuthorDate: Mon Aug 15 17:52:52 2022 +0800
[fix](storage) Convert dict columns in delete_condition_predicates (#11754)
(#11767)
convert dictionary code for delete predicates.
---
be/src/olap/block_column_predicate.h | 62 +++---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 12 +
2 files changed, 54 insertions(+), 20 deletions(-)
diff --git a/be/src/olap/block_column_predicate.h
b/be/src/olap/block_column_predicate.h
index 6f679dbf25..466f67a7c7 100644
--- a/be/src/olap/block_column_predicate.h
+++ b/be/src/olap/block_column_predicate.h
@@ -44,16 +44,22 @@ public:
virtual void get_all_column_ids(std::set& column_id_set) const =
0;
-virtual void evaluate(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t* selected_size) const {};
-virtual void evaluate_and(vectorized::MutableColumns& block, uint16_t*
sel, uint16_t selected_size, bool* flags) const {};
-virtual void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size, bool* flags) const {};
-
-virtual void evaluate_vec(vectorized::MutableColumns& block, uint16_t
size, bool* flags) const {};
+virtual void get_all_column_predicates(std::set&)
const = 0;
+
+virtual void evaluate(vectorized::MutableColumns& block, uint16_t* sel,
+ uint16_t* selected_size) const {};
+virtual void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel,
+ uint16_t selected_size, bool* flags) const {};
+virtual void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel,
+ uint16_t selected_size, bool* flags) const {};
+
+virtual void evaluate_vec(vectorized::MutableColumns& block, uint16_t size,
+ bool* flags) const {};
};
class SingleColumnBlockPredicate : public BlockColumnPredicate {
public:
-explicit SingleColumnBlockPredicate(const ColumnPredicate*
pre):_predicate(pre) {};
+explicit SingleColumnBlockPredicate(const ColumnPredicate* pre) :
_predicate(pre) {};
void evaluate(RowBlockV2* block, uint16_t* selected_size) const override;
void evaluate_and(RowBlockV2* block, uint16_t selected_size, bool* flags)
const override;
@@ -63,10 +69,17 @@ public:
column_id_set.insert(_predicate->column_id());
};
-void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t*
selected_size) const override;
-void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size, bool* flags) const override;
-void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size, bool* flags) const override;
-
+void get_all_column_predicates(std::set&
predicates) const override {
+predicates.insert(_predicate);
+}
+
+void evaluate(vectorized::MutableColumns& block, uint16_t* sel,
+ uint16_t* selected_size) const override;
+void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size,
+ bool* flags) const override;
+void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel,
uint16_t selected_size,
+ bool* flags) const override;
+
void evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool*
flags) const override;
private:
@@ -87,9 +100,7 @@ public:
_block_column_predicate_vec.push_back(column_predicate);
}
-size_t num_of_column_predicate() const {
-return _block_column_predicate_vec.size();
-}
+size_t num_of_column_predicate() const { return
_block_column_predicate_vec.size(); }
void get_all_column_ids(std::set& column_id_set) const override {
for (auto child_block_predicate : _block_column_predicate_vec) {
@@ -97,6 +108,12 @@ public:
}
};
+void get_all_column_predicates(std::set&
predicates) const override {
+for (auto child_block_predicate : _block_column_predicate_vec) {
+child_block_predicate->get_all_column_predicates(predicates);
+}
+}
+
protected:
std::vector _block_column_predicate_vec;
};
@@ -111,9 +128,12 @@ public:
void evaluate_and(RowBlockV2* block, uint16_t selected_size, bool* flags)
const override;
void evaluate_or(RowBlockV2* block, uint16_t selected_size, bool* flags)
const override;
-void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t*
selected_size) const ove
[GitHub] [doris] geniusjoe opened a new pull request, #11791: [Enhancement](docs) fix en kafka-load document
geniusjoe opened a new pull request, #11791: URL: https://github.com/apache/doris/pull/11791 # Proposed changes Issue Number: close #xxx ## Problem summary When visit [kafka load](https://doris.apache.org/docs/data-operate/import/import-scenes/kafka-load) document, click [ROUTINE LOAD](https://doris.apache.org/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.html) will lead to NOT FOUND exception.  ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments It is because an markdown link error.Someone use .html link file instead of .md link file. If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11748: [Bugfix](schema change) fix memory exceeded when schema change
morningman merged PR #11748: URL: https://github.com/apache/doris/pull/11748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](schema change) fix memory exceeded when schema change (#11748)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 71df82696d [fix](schema change) fix memory exceeded when schema change
(#11748)
71df82696d is described below
commit 71df82696d73e4a44b6647992dbfc2d5fadbf4e1
Author: Lightman <31928846+lchangli...@users.noreply.github.com>
AuthorDate: Mon Aug 15 17:57:39 2022 +0800
[fix](schema change) fix memory exceeded when schema change (#11748)
In row mode schema change, it will fail sometime because memory exceeded.
When the left memory is enough for sorting but not enough for next block,
it will not flush row_block_arr which data in memory and continue to alloc
next block
so it can't alloc the memory and return directly.
And if it can't alloc the memory for block, it need to flush row_block_arr
and
try it again unless row_block_arr is empty.
---
be/src/olap/schema_change.cpp | 29 +++--
1 file changed, 15 insertions(+), 14 deletions(-)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 0bccf7ea2d..cbcb5c66c3 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -963,14 +963,8 @@ Status RowBlockAllocator::allocate(RowBlock** row_block,
size_t num_rows, bool n
size_t row_block_size = _row_len * num_rows;
if (_memory_limitation > 0 && _tracker->consumption() + row_block_size >
_memory_limitation) {
-LOG(WARNING)
-<< "RowBlockAllocator::alocate() memory exceeded. "
-<< "m_memory_allocated=" << _tracker->consumption() << " "
-<< "mem limit for schema change=" << _memory_limitation << " "
-<< "You can increase the memory "
-<< "by changing the
Config.memory_limitation_per_thread_for_schema_change_bytes";
*row_block = nullptr;
-return Status::OLAPInternalError(OLAP_ERR_INPUT_PARAMETER_ERROR);
+return Status::OLAPInternalError(OLAP_ERR_FETCH_MEMORY_EXCEEDED);
}
// TODO(lijiao) : Why abandon the original m_row_block_buffer
@@ -1356,11 +1350,15 @@ Status
SchemaChangeWithSorting::_inner_process(RowsetReaderSharedPtr rowset_read
RowBlock* ref_row_block = nullptr;
rowset_reader->next_block(&ref_row_block);
while (ref_row_block != nullptr && ref_row_block->has_remaining()) {
-if (!_row_block_allocator->allocate(&new_row_block,
ref_row_block->row_block_info().row_num,
-true)) {
-LOG(WARNING) << "failed to allocate RowBlock.";
-return Status::OLAPInternalError(OLAP_ERR_INPUT_PARAMETER_ERROR);
-} else {
+auto st = _row_block_allocator->allocate(&new_row_block,
+
ref_row_block->row_block_info().row_num, true);
+// if OLAP_ERR_FETCH_MEMORY_EXCEEDED == st.precise_code()
+// that mean RowBlockAllocator::alocate() memory exceeded.
+// But we can flush row_block_arr if row_block_arr is not empty.
+// Don't return directly.
+if (OLAP_ERR_MALLOC_ERROR == st.precise_code()) {
+return Status::OLAPInternalError(OLAP_ERR_MALLOC_ERROR);
+} else if (st) {
// do memory check for sorting, in case schema change task fail at
row block sorting because of
// not doing internal sorting first
if (!_row_block_allocator->is_memory_enough_for_sorting(
@@ -1375,8 +1373,11 @@ Status
SchemaChangeWithSorting::_inner_process(RowsetReaderSharedPtr rowset_read
if (new_row_block == nullptr) {
if (row_block_arr.empty()) {
LOG(WARNING) << "Memory limitation is too small for Schema
Change."
- << "memory_limitation=" << _memory_limitation;
-return
Status::OLAPInternalError(OLAP_ERR_INPUT_PARAMETER_ERROR);
+ << "memory_limitation=" << _memory_limitation
+ << "You can increase the memory "
+ << "by changing the "
+
"Config.memory_limitation_per_thread_for_schema_change_bytes";
+return
Status::OLAPInternalError(OLAP_ERR_FETCH_MEMORY_EXCEEDED);
}
// enter here while memory limitation is reached.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] whutpencil opened a new issue, #11792: [Bug] Broker Load kerberos authentication duration determination error
whutpencil opened a new issue, #11792: URL: https://github.com/apache/doris/issues/11792 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0 ### What's Wrong? Every time a new broker load comes in, Doris will update the start time of Kerberos authentication, and this logic is wrong. Because the authentication duration of Kerberos is calculated from the moment when the ticket is obtained. ### What You Expected? GSS error will not appear in broker load ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] whutpencil opened a new pull request, #11793: [fix](broker-load) Correction of kerberos authentication time determination rule
whutpencil opened a new pull request, #11793: URL: https://github.com/apache/doris/pull/11793 # Proposed changes Issue Number: https://github.com/apache/doris/issues/11792 ## Problem summary Every time a new broker load comes in, Doris will update the start time of Kerberos authentication, and this logic is wrong. Because the authentication duration of Kerberos is calculated from the moment when the ticket is obtained. Describe your changes. Delete the code that updates the lastAccessTime to get the correct judgment. ## Checklist(Required) 1. Does it affect the original behavior: - [ ✅ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ✅ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ✅ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ✅ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ✅ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #11794: [dev-1.1.2](cherry-pick) Optimize query memory accuracy
xinyiZzz opened a new pull request, #11794: URL: https://github.com/apache/doris/pull/11794 # Proposed changes Issue Number: close #11738 ## Problem summary cherry-pick 11740 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #45: [doc](sequence)Modify sequence error documentation
hf200012 merged PR #45: URL: https://github.com/apache/doris-website/pull/45 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: [doc](sequence)Modify sequence error documentation (#45)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new b298796f38a [doc](sequence)Modify sequence error documentation (#45) b298796f38a is described below commit b298796f38ac547ec594517d075062cf9df98eef Author: caoliang-web <71004656+caoliang-...@users.noreply.github.com> AuthorDate: Mon Aug 15 18:26:30 2022 +0800 [doc](sequence)Modify sequence error documentation (#45) * Modify sequence error documentation --- docs/data-operate/update-delete/sequence-column-manual.md | 8 +++- .../current/data-operate/update-delete/sequence-column-manual.md | 7 +++ 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/docs/data-operate/update-delete/sequence-column-manual.md b/docs/data-operate/update-delete/sequence-column-manual.md index a748c837bb5..25e6359c668 100644 --- a/docs/data-operate/update-delete/sequence-column-manual.md +++ b/docs/data-operate/update-delete/sequence-column-manual.md @@ -209,11 +209,10 @@ MySQL [test]> select * from test_table; +-++--+-+-+ | user_id | date | group_id | modify_date | keyword | +-++--+-+-+ -| 1 | 2020-02-22 |1 | 2020-02-23 | b | +| 1 | 2020-02-22 |1 | 2020-03-05 | c | +-++--+-+-+ ``` -In this import, the b is eventually retained in the keyword column because the value of the sequence column (the value in modify_date) is the maximum value: '2020-02-23'. -Try importing the following data again +In this import, the c is eventually retained in the keyword column because the value of the sequence column (the value in modify_date) in all imports is the maximum value: '2020-03-05'. Try importing the following data again ``` 1 2020-02-22 1 2020-02-22 a @@ -228,5 +227,4 @@ MySQL [test]> select * from test_table; | 1 | 2020-02-22 |1 | 2020-03-23 | w | +-++--+-+-+ ``` -At this point, you can replace the original data in the table. To sum up, if the key is the same in the imported data of the same batch, the sequence column value will be compared. Select the record with the largest value to import into the Doris table,and the sequence column between different batches will not work. - +At this point, you can replace the original data in the table. To sum up, the sequence column will be compared among all the batches, the largest value of the same key will be imported into Doris table. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/update-delete/sequence-column-manual.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/update-delete/sequence-column-manual.md index 36505cc6bcb..de7cad83d4c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/update-delete/sequence-column-manual.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/update-delete/sequence-column-manual.md @@ -226,12 +226,11 @@ MySQL [test]> select * from test_table; +-++--+-+-+ | user_id | date | group_id | modify_date | keyword | +-++--+-+-+ -| 1 | 2020-02-22 |1 | 2020-02-23 | b | +| 1 | 2020-02-22 |1 | 2020-03-05 | c | +-++--+-+-+ ``` -在这次导入的数据中，因sequence column的值（也就是modify_date中的值）中'22020-02-23'为最大值，所以keyword列中最终保留了b。 -再尝试导入如下数据 +在这次导入的数据中，会比较所有已导入数据的sequence column（也就是modify_date)，其中'2020-03-05'为最大值，所以keyword列中最终保留了c。再尝试导入如下数据 ```text 1 2020-02-22 1 2020-02-22 a @@ -249,7 +248,7 @@ MySQL [test]> select * from test_table; +-++--+-+-+ ``` -此时就可以替换表中原有的数据。综上，在同批次导入数据中key相同会比较sequence列值，选择值最大的记录导入Doris表中，不同批次之间sequence列不起作用。 +此时就可以替换表中原有的数据。综上，在导入过程中，会比较所有批次的sequence列值，选择值最大的记录导入Doris表中。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #44: Update xiaomi.md
hf200012 merged PR #44: URL: https://github.com/apache/doris-website/pull/44 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: Update xiaomi.md (#44)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 061daf46a4f Update xiaomi.md (#44)
061daf46a4f is described below
commit 061daf46a4f6187fd8f822d8a06426072ff089be
Author: ssusieee <91410365+ssusi...@users.noreply.github.com>
AuthorDate: Mon Aug 15 18:37:13 2022 +0800
Update xiaomi.md (#44)
---
.../zh-CN/docusaurus-plugin-content-blog/xiaomi.md | 222 +++--
1 file changed, 162 insertions(+), 60 deletions(-)
diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi.md
b/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi.md
index ee3f0e1e18b..8ba7796dd67 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-blog/xiaomi.md
@@ -1,8 +1,8 @@
---
{
-'title': 'Apache Doris 在小米集团的运维实践',
-'summary': "为了提高小米增长分析平台的查询性能以及降低平台的运维成本，2019 年 9 月小米集团引入了 Apache
Doris。在过去两年多的时间里，Apache Doris
在小米集团得到了广泛的应用，目前已经服务了增长分析、集团数据看板、天星金融、小米有品、用户画像、广告投放、A/B
实验平台、新零售等数十个业务。在小米集团，质量就是生命线，随着业务持续增长，如何保障线上 Apache Doris
集群的服务质量，对集群的运维人员来说是个不小的挑战。本文将从运维的角度对 Apache Doris 在小米集团的应用实践进行分享.",
-'date': '2022-07-20',
+'title': 'Apache Doris 在小米数据场景的应用实践与优化',
+'summary': "因增长分析业务需要，小米集团于 2019 年首次引入了 Apache Doris 。经过三年时间的发展，目前 Apache
Doris 已经在广告投放、新零售、增长分析、数据看板、天星数科、小米有品、用户画像等小米内部数十个业务中得到广泛应用 ，并且在小米内部已经形成一套以
Apache Doris 为核心的数据生态。",
+'date': '2022-08-15',
'author': 'Apache Doris',
'tags': ['最佳实践'],
}
@@ -29,110 +29,212 @@ under the License.
# 背景
-为了提高小米增长分析平台的查询性能以及降低平台的运维成本，2019 年 9 月小米集团引入了 Apache Doris。在过去两年多的时间里，Apache
Doris 在小米集团得到了广泛的应用，目前已经服务了增长分析、集团数据看板、天星金融、小米有品、用户画像、广告投放、A/B
实验平台、新零售等数十个业务。在小米集团，质量就是生命线，随着业务持续增长，如何保障线上 Apache Doris
集群的服务质量，对集群的运维人员来说是个不小的挑战。本文将从运维的角度对 Apache Doris 在小米集团的应用实践进行分享。
+因增长分析业务需要，小米集团于 2019 年首次引入了 Apache Doris 。经过三年时间的发展，目前 Apache Doris
已经在广告投放、新零售、增长分析、数据看板、天星数科、小米有品、用户画像等小米内部数十个业务中得到广泛应用 **，并且在小米内部已经形成一套以 Apache
Doris 为核心的数据生态。**
+
+当前 Apache Doris 在小米内部已经具有**数十个**集群、总体达到**数百台** BE
节点的规模，其中单集群最大规模达到**近百台节点**，拥有**数十个**流式数据导入产品线，每日单表最大增量 **120 亿**、支持 **PB
级别**存储，单集群每天可以支持 **2W 次以上**的多维分析查询。
+# 架构演进
-# 集群部署和升级
+小米引入 Apache Doris
的初衷是为了解决内部进行用户行为分析时所遇到的问题。随着小米互联网业务的发展，各个产品线利用用户行为数据对业务进行增长分析的需求越来越迫切。让每个业务产品线都自己搭建一套增长分析系统，不仅成本高昂，也会导致效率低下。因此能有一款产品能够帮助他们屏蔽底层复杂的技术细节，让相关业务人员能够专注于自己的技术领域，可以极大提高工作效率。基于此，小米大数据和云平台联合开发了增长分析系统
Growing Analytics（下文中简称 GA
)，旨在提供一个灵活的多维实时查询和分析平台，统一数据接入和查询方案，帮助业务线做精细化运营。（此处内容引用自：[基于Apache
Doris的小米增长分析平台实践](https://mp.weixin.qq.com/s?__biz=MzUxMDQxMDMyNg==&mid=2247486817&idx=1&sn=99fbef15b4d6f6059c3affbc77517e6e&scene=21#wechat_redirect)）
-基于 Apache Doris 社区发布的稳定版本，小米也维护了内部的 Doris 分支用于内部小版本的迭代。由于和社区编译 Docker
第三方库的硬件环境存在差异，基于社区 Docker 编译出的 Doris 二进制包运行在小米的线上环境会有问题，因此小米内部也维护了自己的 Docker
镜像，用于内部 Doris 分支的编译及发版。内部发版时，在 Docker 容器中会完成源码的编译和打包，并通过 Minos 将二进制包上传到 Tank
Server（小米内部的版本仓库）。
+
-Minos 是小米自研并开源的一款基于命令行的大数据部署和进程管理系统，支持 Doris、HDFS、HBase、Zookeeper
等服务的部署和管理。在小米内部，包上传、集群部署、集群下线、集群升级、进程重启、配置变更等操作都可以通过 Minos 完成， Minos
对于服务的管理依赖于配置文件 deployment-config ，其中配置了服务版本信息、集群的节点信息、集群的配置参数等信息。部署集群时，Minos
会根据 deployment-config 中配置的服务版本信息从 Tank Server 上拉取对应的二进制包，并根据 deployment-config
中配置的节点信息和集群参数部署集群。在集群部署之后，如果进程意外挂掉，Minos 会自动拉起进程，恢复服务。
+分析、决策、执行是一个循环迭代的过程，在对用户进行行为分析后，针对营销策略是否还有提升空间、是否需要在前端对用户进行个性化推送等问题进行决策，帮助小米实现业务的持续增长。这个过程是对用户行为进行**分析-决策-优化执行-再分析-再决策-再优化执行**的迭代过程。
-轻舟是小米自研的分布式服务生命周期管理平台，贯穿分布式大数据系统从需求评估开始到资源下线结束的生命周期互联互通管理，主要由发布中心、巡检中心、运营数仓、环境管理、故障管理、容量管理等组成，各模块之间逻辑互联、数据互通。轻舟发布中心提供了可编排、低代码、可视化的服务发布和进程管理能力。轻舟发布中心底层依赖
Minos ，因此，可以基于轻舟对 Doris 服务实现平台化管理，包括集群部署、集群下线、集群升级、进程重启、配置变更等操作，如果 Doris 的 FE
或 BE 进程意外挂掉，轻舟会自动拉起进程，恢复服务。
-# 业务实践
+### 历史架构
-Apache Doris 在小米的典型业务实践如下：
+增长分析平台立项于 2018 年年中，当时基于开发时间和成本，技术栈等因素的考虑，小米复用了现有各种大数据基础组件（HDFS, Kudu, SparkSQL
等），搭建了一套基于 Lamda 架构的增长分析查询系统。**GA 系统初代版本的架构如下图所示，包含了以下几个方面：**
-## 01 用户接入
+- 数据源：数据源是前端的埋点数据以及可能获取到的用户行为数据。
+- 数据接入层：对埋点数据进行统一的清洗后打到小米内部自研的消息队列 Talos 中，并通过 Spark Streaming 将数据导入存储层 Kudu
中。
+- 存储层：在存储层中进行冷热数据分离。热数据存放在 Kudu 中，冷数据则会存放在 HDFS
上。同时在存储层中进行分区，当分区单位为天时，每晚会将一部分数据转冷并存储到 HDFS 上。
+- 计算层/查询层：在查询层中，使用 SparkSQL 对 Kudu 与 HDFS 上数据进行联合视图查询，最终把查询结果在前端页面上进行显示。
-数据工场是小米自研的、面向数据开发和数据分析人员的一站式数据开发平台，底层支持
Doris、Hive、Kudu、Iceberg、ES、Talso、TiDB、MySQL 等数据源，同时支持 Flink、Spark、Presto 等计算能力 。
+
-在小米内部，用户需要通过数据工场接入 Doris 服务，用户需要在数据工场进行注册，并完成建库审批， Doris
运维同学会根据数据工场中用户提交的业务场景、数据使用预期等描述进行接入审批和指导，用户完成接入审批后即可使用 Doris
服务，在数据工场中进行可视化建表和数据导入等操作。
+**在当时的历史背景下，初代版本的增长分析平台帮助我们解决了一系列用户运营过程中的问题，但同时在历史架构中也存在了两个问题：**
-## 02 数据导入
+**第一个问题：** 由于历史架构是基于 SparkSQL + Kudu + HD
[doris-website] branch master updated: update
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new d6b5dde8bc2 update d6b5dde8bc2 is described below commit d6b5dde8bc20f012418ef0d34437ca7387e47c7b Author: jiafeng.zhang AuthorDate: Mon Aug 15 18:42:41 2022 +0800 update --- community/release-and-verify/release-verify.md| 15 +++ .../current/design/grouping_sets_design.md| 6 +- .../current/release-and-verify/release-verify.md | 15 +++ 3 files changed, 15 insertions(+), 21 deletions(-) diff --git a/community/release-and-verify/release-verify.md b/community/release-and-verify/release-verify.md index 93654bcf063..f95c73d85de 100644 --- a/community/release-and-verify/release-verify.md +++ b/community/release-and-verify/release-verify.md @@ -30,14 +30,13 @@ This validation step can be used for validation during release polling and also The following steps can be followed to verify. -1. [ ] Whether the download link is legitimate. -2. [ ] Whether the checksum and PGP signature are legitimate. 3. -3. [ ] Whether the DISCLAIMER or DISCLAIMER-WIP is included. -4. [ ] Whether the code matches the current release. -5. [ ] Whether the LICENSE and NOTICE files are correct. -6. [ ] All files carry the necessary protocol descriptions. -7. [ ] No compiled content is included in the source package. -8. [ ] Whether the compilation can be executed without problems. +1. [ ] The download link is legal. +2. [ ] The PGP signature are valid. +3. [ ] The source code matches the current release version. +4. [ ] The LICENSE and NOTICE files are correct. +5. [ ] All files carry the necessary protocol header. +6. [ ] The compiled content is not included in the source package. +7. [ ] The compilation can be executed smoothly. Here we use the verification of the Doris Core version as an example. Note that other components have their corresponding names changed. diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/design/grouping_sets_design.md b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/design/grouping_sets_design.md index 6ca788064c5..0c19094cffc 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/design/grouping_sets_design.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/design/grouping_sets_design.md @@ -502,11 +502,7 @@ select NULL, NULL, sum(k3) from t; 在 GroupByClause 类中为了将 GROUPING_ID 加到 groupingExprs 表达式列表中，需要创建 virtual SlotRef, 相应的，需要对这个 slot 创建一个 tuple, 叫 GROUPING_ID Tuple。 -对于 RepeatNode 这个执行计划，其输入是子节点的所有 tuple， 输出的 tuple 除了 repeat 子节点的数据外，还需要填写 GROUPING_ID 和其他grouping，grouping_id 对应的虚拟列。 - - 3.3.3 表达式和函数替换 -group子句表达式拓展：`expr -> if(bitand(pos, grouping_id)=0, expr, null)` -grouping_id函数：`grouping_id() -> grouping_id(grouping_id)` +对于 RepeatNode 这个执行计划，其输入是子节点的所有 tuple， 输出的 tuple 除了 repeat 子节点的数据外，还需要填写 GROUPING_ID 和其他grouping，grouping_id 对应的虚拟列，因此。 ### 3.4 BE 查询执行阶段 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/release-and-verify/release-verify.md b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/release-and-verify/release-verify.md index 4030152baa2..3dd1cd6243b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/release-and-verify/release-verify.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/release-and-verify/release-verify.md @@ -30,14 +30,13 @@ under the License. 可以按照以下步骤进行验证： -1. [ ] 下载链接是否合法。 -2. [ ] 校验值和 PGP 签名是否合法。 -3. [ ] 是否包含 DISCLAIMER 或 DISCLAIMER-WIP 文件。 -4. [ ] 代码是否和当前发布版本相匹配。 -5. [ ] LICENSE 和 NOTICE 文件是否正确。 -6. [ ] 所有文件都携带必要的协议说明。 -7. [ ] 在源码包中不包含已经编译好的内容。 -8. [ ] 编译是否能够顺利执行。 +1. [ ] 下载链接合法。 +2. [ ] 校验值和 PGP 签名合法。 +3. [ ] 代码和当前发布版本相匹配。 +4. [ ] LICENSE 和 NOTICE 文件正确。 +5. [ ] 所有文件都携带必要的协议说明。 +6. [ ] 在源码包中不包含已经编译好的内容。 +7. [ ] 编译能够顺利执行。 这里我们以 Doris Core 版本的验证为例。其他组件注意修改对应名称。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #46: Update flink connector version
hf200012 merged PR #46: URL: https://github.com/apache/doris-website/pull/46 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: Update flink connector version (#46)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new a8fe7206ea8 Update flink connector version (#46)
a8fe7206ea8 is described below
commit a8fe7206ea8859b6a41dd02d8f9d9cd7158f3893
Author: song7788q
AuthorDate: Mon Aug 15 18:43:15 2022 +0800
Update flink connector version (#46)
---
src/constant/download.data.ts | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/src/constant/download.data.ts b/src/constant/download.data.ts
index 53e667b967c..faa708215a8 100644
--- a/src/constant/download.data.ts
+++ b/src/constant/download.data.ts
@@ -157,6 +157,15 @@ export const ALL_RELEASE = [
];
export const FLINK_CONNECTOR = [
+{
+version: '1.2.0',
+date: '2022-08-15',
+flink: '1.15',
+scala: '2.12, 2.11',
+doris: '1.0+',
+download:
'https://dist.apache.org/repos/dist/release/doris/flink-connector/1.2.0/',
+github: 'https://github.com/apache/doris-flink-connector',
+},
{
version: '1.1.0',
date: '2022-07-11',
@@ -170,7 +179,7 @@ export const FLINK_CONNECTOR = [
version: '1.0.3',
date: '2021-03-18',
flink: '1.14, 1.13, 1.12, 1.11',
-scala: '2.12',
+scala: '2.12, 2.11',
doris: '0.15+',
download:
'https://dist.apache.org/repos/dist/release/doris/flink-connector/1.0.3/',
github: 'https://github.com/apache/doris-flink-connector',
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: update build script
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new b9cf02ea586 update build script b9cf02ea586 is described below commit b9cf02ea586e0952e79d568b684525887f6bface Author: jiafeng.zhang AuthorDate: Mon Aug 15 18:44:24 2022 +0800 update build script --- .github/workflows/cron-deploy-website.yml | 3 --- .github/workflows/manual-deploy-website.yml | 3 --- 2 files changed, 6 deletions(-) diff --git a/.github/workflows/cron-deploy-website.yml b/.github/workflows/cron-deploy-website.yml index 7ab2e572a78..04180749b68 100644 --- a/.github/workflows/cron-deploy-website.yml +++ b/.github/workflows/cron-deploy-website.yml @@ -19,15 +19,12 @@ jobs: - name: Build run: | git clone https://github.com/apache/doris.git -cp -R doris/docs/sidebarsCommunity.json . mkdir versioned_docs/version-dev/ cp -R doris/docs/en/docs/* versioned_docs/version-dev/ cp -R doris/docs/sidebars.json versioned_sidebars/version-dev-sidebars.json mkdir i18n/zh-CN/docusaurus-plugin-content-docs/version-dev cp -R doris/docs/zh-CN/docs/* i18n/zh-CN/docusaurus-plugin-content-docs/version-dev/ cp -R doris/docs/dev.json i18n/zh-CN/docusaurus-plugin-content-docs/version-dev.json -cp -R doris/docs/en/community . -cp -R doris/docs/zh-CN/community/* i18n/zh-CN/docusaurus-plugin-content-docs-community/current/ cp -R doris/docs/images static/ npm install -g yarn diff --git a/.github/workflows/manual-deploy-website.yml b/.github/workflows/manual-deploy-website.yml index e94b49dbe5c..4cd7708e44f 100644 --- a/.github/workflows/manual-deploy-website.yml +++ b/.github/workflows/manual-deploy-website.yml @@ -23,15 +23,12 @@ jobs: - name: Build run: | git clone https://github.com/apache/doris.git -cp -R doris/docs/sidebarsCommunity.json . mkdir versioned_docs/version-dev/ cp -R doris/docs/en/docs/* versioned_docs/version-dev/ cp -R doris/docs/sidebars.json versioned_sidebars/version-dev-sidebars.json mkdir i18n/zh-CN/docusaurus-plugin-content-docs/version-dev cp -R doris/docs/zh-CN/docs/* i18n/zh-CN/docusaurus-plugin-content-docs/version-dev/ cp -R doris/docs/dev.json i18n/zh-CN/docusaurus-plugin-content-docs/version-dev.json -cp -R doris/docs/en/community . -cp -R doris/docs/zh-CN/community/* i18n/zh-CN/docusaurus-plugin-content-docs-community/current/ cp -R doris/docs/images static/ npm install -g yarn - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: add spark load
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 517e280f584 add spark load
517e280f584 is described below
commit 517e280f5846298f8df22eec450dfd33464cc5c4
Author: jiafeng.zhang
AuthorDate: Mon Aug 15 18:48:06 2022 +0800
add spark load
---
community/design/spark_load.md | 29 +
1 file changed, 29 insertions(+)
diff --git a/community/design/spark_load.md b/community/design/spark_load.md
new file mode 100644
index 000..9e630f4393a
--- /dev/null
+++ b/community/design/spark_load.md
@@ -0,0 +1,29 @@
+---
+{
+"title": "Spark Load",
+"language": "en"
+}
+---
+
+
+
+# Spark load
+
+TODO
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on a diff in pull request #11782: [fix](agg)disallow group by bitmap or hll data type
starocean999 commented on code in PR #11782:
URL: https://github.com/apache/doris/pull/11782#discussion_r945617217
##
fe/fe-core/src/main/java/org/apache/doris/analysis/AggregateInfo.java:
##
@@ -160,6 +161,15 @@ public void setPartitionExprs(List exprs) {
partitionExprs = exprs;
}
+private static void validateGroupingExprs(List groupingExprs) throws
AnalysisException {
Review Comment:
Thanks for your time. But there is no group by clause in sql like "select
distinct bitmap, hll from xxx", the agg node is created by fe to implement the
"distinct" operator, so no analyze function is called in that case. Anyway, I
update the error message which is consistent with the error "select count(*)
from xxx group by bitmap" now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 7dfc360efb3 fix 7dfc360efb3 is described below commit 7dfc360efb3115d5cb8b3b4e36548e399a9d1270 Author: jiafeng.zhang AuthorDate: Mon Aug 15 19:23:24 2022 +0800 fix --- .github/workflows/manual-deploy-website.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/manual-deploy-website.yml b/.github/workflows/manual-deploy-website.yml index 4cd7708e44f..e3864afdea4 100644 --- a/.github/workflows/manual-deploy-website.yml +++ b/.github/workflows/manual-deploy-website.yml @@ -50,7 +50,7 @@ jobs: rm -rf node_modules rm -rf doris rm -rf versioned_docs/version-dev/ -rm -rf i18n/zh-CN/docusaurus-plugin-content-docs/dev/ +rm -rf i18n/zh-CN/docusaurus-plugin-content-docs/version-dev/ git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" git add . - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11757: [fix](query) fix orderby keys limit return less or no result
github-actions[bot] commented on PR #11757: URL: https://github.com/apache/doris/pull/11757#issuecomment-1214911725 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11757: [fix](query) fix orderby keys limit return less or no result
github-actions[bot] commented on PR #11757: URL: https://github.com/apache/doris/pull/11757#issuecomment-1214911757 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #11601: [feature-wip](parquet-reader) update column read model and add page index
morningman commented on code in PR #11601:
URL: https://github.com/apache/doris/pull/11601#discussion_r945588010
##
be/src/vec/exec/format/parquet/vparquet_reader.h:
##
@@ -58,18 +64,18 @@ class ParquetReadColumn {
class ParquetReader {
public:
-ParquetReader(FileReader* file_reader, int32_t num_of_columns_from_file,
+ParquetReader(FileReader* file_reader, int32_t num_of_columns_from_file,
size_t batch_size,
int64_t range_start_offset, int64_t range_size);
~ParquetReader();
Status init_reader(const TupleDescriptor* tuple_desc,
const std::vector& tuple_slot_descs,
- const std::vector& conjunct_ctxs, const
std::string& timezone);
+ std::vector& conjunct_ctxs, const
std::string& timezone);
-Status read_next_batch(Block* block);
+Status read_next_batch(Block* block, bool* eof);
-bool has_next() const { return !_batch_eof; };
+bool has_next() const { return !*_file_eof; };
Review Comment:
the `eof` of `read_next_batch` already told you the end-of-file info. No
need this `has_next()` method.
##
be/src/vec/exec/format/parquet/vparquet_reader.cpp:
##
@@ -45,26 +50,26 @@ void ParquetReader::close() {
Status ParquetReader::init_reader(const TupleDescriptor* tuple_desc,
const std::vector&
tuple_slot_descs,
- const std::vector&
conjunct_ctxs,
+ std::vector& conjunct_ctxs,
const std::string& timezone) {
_file_reader->open();
+_conjunct_ctxs.reset(&conjunct_ctxs);
RETURN_IF_ERROR(parse_thrift_footer(_file_reader, _file_metadata));
auto metadata = _file_metadata->to_thrift_metadata();
Review Comment:
This `metadata` is unused?
You can save it as a field of this class.
##
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp:
##
@@ -24,235 +24,57 @@
namespace doris::vectorized {
RowGroupReader::RowGroupReader(doris::FileReader* file_reader,
- const std::shared_ptr&
file_metadata,
const std::vector&
read_columns,
- const std::map& map_column,
- const std::vector& conjunct_ctxs)
+ const int32_t row_group_id, tparquet::RowGroup&
row_group)
: _file_reader(file_reader),
- _file_metadata(file_metadata),
_read_columns(read_columns),
- _map_column(map_column),
- _conjunct_ctxs(conjunct_ctxs),
- _current_row_group(-1) {}
+ _row_group_id(row_group_id),
+ _row_group_meta(row_group),
+ _total_rows(row_group.num_rows) {}
RowGroupReader::~RowGroupReader() {
-for (auto& column_reader : _column_readers) {
-auto reader = column_reader.second;
-reader->close();
-delete reader;
-reader = nullptr;
-}
_column_readers.clear();
}
-Status RowGroupReader::init(const TupleDescriptor* tuple_desc, int64_t
split_start_offset,
-int64_t split_size) {
-_tuple_desc = tuple_desc;
-_split_start_offset = split_start_offset;
-_split_size = split_size;
-_init_conjuncts(tuple_desc, _conjunct_ctxs);
-RETURN_IF_ERROR(_init_column_readers());
+Status RowGroupReader::init(const FieldDescriptor& schema,
std::vector& row_ranges) {
+RETURN_IF_ERROR(_init_column_readers(schema, row_ranges));
return Status::OK();
}
-void RowGroupReader::_init_conjuncts(const TupleDescriptor* tuple_desc,
- const std::vector&
conjunct_ctxs) {
-if (tuple_desc->slots().empty()) {
-return;
-}
-for (auto& read_col : _read_columns) {
-_parquet_column_ids.emplace(read_col.parquet_column_id);
-}
-
-for (int i = 0; i < tuple_desc->slots().size(); i++) {
-auto col_iter = _map_column.find(tuple_desc->slots()[i]->col_name());
-if (col_iter == _map_column.end()) {
-continue;
-}
-int parquet_col_id = col_iter->second;
-if (_parquet_column_ids.end() ==
_parquet_column_ids.find(parquet_col_id)) {
-continue;
-}
-for (int conj_idx = 0; conj_idx < conjunct_ctxs.size(); conj_idx++) {
-Expr* conjunct = conjunct_ctxs[conj_idx]->root();
-if (conjunct->get_num_children() == 0) {
-continue;
-}
-Expr* raw_slot = conjunct->get_child(0);
-if (TExprNodeType::SLOT_REF != raw_slot->node_type()) {
-continue;
-}
-SlotRef* slot_ref = (SlotRef*)raw_slot;
-SlotId conjunct_slot_id = slot_ref->slot_id();
-if (conjunct_slot_id == tuple_desc->slots()[i]->id()) {
-// Get conjuncts by conjunct_slot_id
-
[GitHub] [doris] wsjz commented on a diff in pull request #11601: [feature-wip](parquet-reader) update column read model and add page index
wsjz commented on code in PR #11601:
URL: https://github.com/apache/doris/pull/11601#discussion_r945651384
##
be/src/vec/exec/format/parquet/vparquet_reader.cpp:
##
@@ -90,63 +95,240 @@ Status ParquetReader::_init_read_columns(const
std::vector& tup
auto physical_type =
_file_metadata->schema().get_column(parquet_col_id)->physical_type;
column.parquet_type = physical_type;
_read_columns.emplace_back(column);
+VLOG_DEBUG << "slot_desc " << slot_desc->debug_string();
}
return Status::OK();
}
-Status ParquetReader::read_next_batch(Block* block) {
-int32_t group_id = 0;
-RETURN_IF_ERROR(_row_group_reader->get_next_row_group(&group_id));
+Status ParquetReader::read_next_batch(Block* block, bool* eof) {
+DCHECK(_total_groups == _row_group_readers.size());
+if (_total_groups == 0) {
+*eof = true;
+}
+bool _batch_eof = false;
+auto row_group_reader = _row_group_readers[_current_row_group_id];
+RETURN_IF_ERROR(row_group_reader->next_batch(block, _batch_size,
&_batch_eof));
+if (_batch_eof) {
+_current_row_group_id++;
+if (_current_row_group_id > _total_groups) {
+*eof = true;
+}
Review Comment:
The `_current_row_group_id++;` statement can do this，so it's implicit to
mark `*eof = true`
##
be/src/vec/exec/format/parquet/vparquet_reader.cpp:
##
@@ -90,63 +95,240 @@ Status ParquetReader::_init_read_columns(const
std::vector& tup
auto physical_type =
_file_metadata->schema().get_column(parquet_col_id)->physical_type;
column.parquet_type = physical_type;
_read_columns.emplace_back(column);
+VLOG_DEBUG << "slot_desc " << slot_desc->debug_string();
}
return Status::OK();
}
-Status ParquetReader::read_next_batch(Block* block) {
-int32_t group_id = 0;
-RETURN_IF_ERROR(_row_group_reader->get_next_row_group(&group_id));
+Status ParquetReader::read_next_batch(Block* block, bool* eof) {
+DCHECK(_total_groups == _row_group_readers.size());
+if (_total_groups == 0) {
+*eof = true;
+}
+bool _batch_eof = false;
+auto row_group_reader = _row_group_readers[_current_row_group_id];
+RETURN_IF_ERROR(row_group_reader->next_batch(block, _batch_size,
&_batch_eof));
+if (_batch_eof) {
+_current_row_group_id++;
+if (_current_row_group_id > _total_groups) {
+*eof = true;
+}
Review Comment:
The `_current_row_group_id++;` statement can do this，so it's implicit to
mark `*eof =false`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #11601: [feature-wip](parquet-reader) update column read model and add page index
wsjz commented on code in PR #11601:
URL: https://github.com/apache/doris/pull/11601#discussion_r945651384
##
be/src/vec/exec/format/parquet/vparquet_reader.cpp:
##
@@ -90,63 +95,240 @@ Status ParquetReader::_init_read_columns(const
std::vector& tup
auto physical_type =
_file_metadata->schema().get_column(parquet_col_id)->physical_type;
column.parquet_type = physical_type;
_read_columns.emplace_back(column);
+VLOG_DEBUG << "slot_desc " << slot_desc->debug_string();
}
return Status::OK();
}
-Status ParquetReader::read_next_batch(Block* block) {
-int32_t group_id = 0;
-RETURN_IF_ERROR(_row_group_reader->get_next_row_group(&group_id));
+Status ParquetReader::read_next_batch(Block* block, bool* eof) {
+DCHECK(_total_groups == _row_group_readers.size());
+if (_total_groups == 0) {
+*eof = true;
+}
+bool _batch_eof = false;
+auto row_group_reader = _row_group_readers[_current_row_group_id];
+RETURN_IF_ERROR(row_group_reader->next_batch(block, _batch_size,
&_batch_eof));
+if (_batch_eof) {
+_current_row_group_id++;
+if (_current_row_group_id > _total_groups) {
+*eof = true;
+}
Review Comment:
The `_current_row_group_id++;` statement can do this，so it's implicit to
mark `*eof =false`, the `next_batch() ` will be called in the upper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #11601: [feature-wip](parquet-reader) update column read model and add page index
wsjz commented on code in PR #11601:
URL: https://github.com/apache/doris/pull/11601#discussion_r945657012
##
be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp:
##
@@ -43,11 +44,12 @@ Status ColumnChunkReader::init() {
// get the block compression codec
RETURN_IF_ERROR(get_block_compression_codec(_metadata.codec,
_block_compress_codec));
+VLOG_DEBUG << "initColumnChunkReader finish";
return Status::OK();
}
Status ColumnChunkReader::next_page() {
-RETURN_IF_ERROR(_page_reader->next_page());
+RETURN_IF_ERROR(_page_reader->next_page_header());
_num_values = _page_reader->get_page_header()->data_page_header.num_values;
Review Comment:
remaining
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaojunjie opened a new pull request, #11795: [enhancement](Nereids)support max function
xiaojunjie opened a new pull request, #11795: URL: https://github.com/apache/doris/pull/11795 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [X] No - [ ] I don't know 2. Has unit tests been added: - [X] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [X] No Need 4. Does it need to update dependencies: - [ ] Yes - [X] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [X] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz commented on a diff in pull request #11601: [feature-wip](parquet-reader) update column read model and add page index
wsjz commented on code in PR #11601:
URL: https://github.com/apache/doris/pull/11601#discussion_r945666374
##
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:
##
@@ -19,50 +19,82 @@
#include
#include
+#include
#include "schema_desc.h"
#include "vparquet_column_chunk_reader.h"
namespace doris::vectorized {
-Status ScalarColumnReader::init(const FileReader* file, const FieldSchema*
field,
-const tparquet::ColumnChunk* chunk, const
TypeDescriptor& col_type,
-int64_t chunk_size) {
-// todo1: init column chunk reader
-// BufferedFileStreamReader stream_reader(reader, 0, chunk_size);
-// _chunk_reader(&stream_reader, chunk, field);
-// _chunk_reader.init();
-return Status();
-}
-
-Status ParquetColumnReader::create(const FileReader* file, int64_t chunk_size,
- const FieldSchema* field, const
ParquetReadColumn& column,
- const TypeDescriptor& col_type,
+Status ParquetColumnReader::create(FileReader* file, FieldSchema* field,
+ const ParquetReadColumn& column,
const tparquet::RowGroup& row_group,
- const ParquetColumnReader* reader) {
+ std::vector& row_ranges,
+ std::unique_ptr&
reader) {
if (field->type.type == TYPE_MAP || field->type.type == TYPE_STRUCT) {
return Status::Corruption("not supported type");
}
if (field->type.type == TYPE_ARRAY) {
return Status::Corruption("not supported array type yet");
} else {
+VLOG_DEBUG << "field->physical_column_index: " <<
field->physical_column_index;
+tparquet::ColumnChunk chunk =
row_group.columns[field->physical_column_index];
ScalarColumnReader* scalar_reader = new ScalarColumnReader(column);
-RETURN_IF_ERROR(scalar_reader->init(file, field,
-
&row_group.columns[field->physical_column_index],
-col_type, chunk_size));
-reader = scalar_reader;
+scalar_reader->init_column_metadata(chunk);
+RETURN_IF_ERROR(scalar_reader->init(file, field, &chunk, row_ranges));
+reader.reset(scalar_reader);
}
return Status::OK();
}
-Status ScalarColumnReader::read_column_data(const tparquet::RowGroup&
row_group_meta,
-ColumnPtr* data) {
-// todo2: read data with chunk reader to load page data
-// while (_chunk_reader.has_next) {
-// _chunk_reader.next_page();
-// _chunk_reader.load_page_data();
-// }
-return Status();
+void ParquetColumnReader::init_column_metadata(const tparquet::ColumnChunk&
chunk) {
+auto chunk_meta = chunk.meta_data;
+int64_t chunk_start = chunk_meta.__isset.dictionary_page_offset
+ ? chunk_meta.dictionary_page_offset
+ : chunk_meta.data_page_offset;
+size_t chunk_len = chunk_meta.total_compressed_size;
+_metadata.reset(new ParquetColumnMetadata(chunk_start, chunk_len,
chunk_meta));
+}
+
+void ParquetColumnReader::_skipped_pages() {}
+
+Status ScalarColumnReader::init(FileReader* file, FieldSchema* field,
tparquet::ColumnChunk* chunk,
+std::vector& row_ranges) {
+BufferedFileStreamReader stream_reader(file, _metadata->start_offset(),
_metadata->size());
+_row_ranges.reset(&row_ranges);
+_chunk_reader.reset(new ColumnChunkReader(&stream_reader, chunk, field));
+_chunk_reader->init();
+return Status::OK();
+}
+
+Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column,
DataTypePtr& type,
+size_t batch_size, int64_t*
read_rows, bool* eof) {
+if (_chunk_reader->num_values() <= 0) {
+// seek to next page header
+_chunk_reader->next_page();
+if (_row_ranges->size() != 0) {
+_skipped_pages();
+}
+// load data to decoder
+_chunk_reader->load_page_data();
+}
+size_t read_values =
+_chunk_reader->num_values() < batch_size ?
_chunk_reader->num_values() : batch_size;
Review Comment:
default size of a block is `batch_size`, I think we should limit it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11796: [Bug](cast) change binary predicate finally cast to varchar
BiteThet opened a new pull request, #11796: URL: https://github.com/apache/doris/pull/11796 # Proposed changes The phone is largeint here. ```sql mysql> select * from test_demo where phone='19545678934567895565443878'; +-++ | user_id | phone | +-++ | 10003 | 19545678934567895565443878 | | 10004 | 19545678934567895565443878 | | 10005 | 19545678934567895565443877 | | 10006 | 19545678934567895565443878 | | 10009 | 19545678934567895565443878 | +-++ ``` ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #11797: [fix](memtracker) Fix core in logout task mem tracker
xinyiZzz opened a new pull request, #11797: URL: https://github.com/apache/doris/pull/11797 # Proposed changes Issue Number: close #xxx ## Problem summary  ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11029: [fix] Fix potential unhandled exception cause data inconsistency
morningman commented on PR #11029: URL: https://github.com/apache/doris/pull/11029#issuecomment-1215015904 @TsukiokaKogane please rebase to pass the check -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #11585: [fix](array-type) Fix incorrect in function-set for array type
xy720 merged PR #11585: URL: https://github.com/apache/doris/pull/11585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](array-type) Fix incorrect in function-set for array type (#11585)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fef37990a3 [fix](array-type) Fix incorrect in function-set for array
type (#11585)
fef37990a3 is described below
commit fef37990a39a2404dd49ee835947665d51b66d83
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Mon Aug 15 21:29:57 2022 +0800
[fix](array-type) Fix incorrect in function-set for array type (#11585)
There is some wrong logic in FunctionSet.java and it may causes potential
risks for array functions invoke.
---
fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java | 2 +-
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
index a2a9092b10..4e1441fdbe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
@@ -118,7 +118,7 @@ public class ArrayType extends Type {
return false;
}
ArrayType otherArrayType = (ArrayType) other;
-return otherArrayType.itemType.equals(itemType);
+return otherArrayType.itemType.equals(itemType) &&
otherArrayType.containsNull == containsNull;
}
public static boolean canCastTo(ArrayType type, ArrayType targetType) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index 5e9b89c790..fd6fad49f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -1200,7 +1200,7 @@ public class FunctionSet {
if (!(descArgTypes[0] instanceof ScalarType)
|| !(candicateArgTypes[0] instanceof ScalarType)) {
if (candicateArgTypes[0] instanceof ArrayType) {
-return candicateArgTypes[0].matchesType(descArgTypes[0]);
+return descArgTypes[0].matchesType(candicateArgTypes[0]);
}
return false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 merged pull request #11602: [fix](array-type) disable cast function to array type on origin exec engine.
xy720 merged PR #11602: URL: https://github.com/apache/doris/pull/11602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xy720 closed issue #11592: [Bug] cast to array function causes be core dump on non vec engine
xy720 closed issue #11592: [Bug] cast to array function causes be core dump on non vec engine URL: https://github.com/apache/doris/issues/11592 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](array-type) disable cast function to array type on origin exec engine. (#11602)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8f98357c0b [fix](array-type) disable cast function to array type on
origin exec engine. (#11602)
8f98357c0b is described below
commit 8f98357c0b573423940775514db9f6831436a867
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Mon Aug 15 21:30:56 2022 +0800
[fix](array-type) disable cast function to array type on origin exec
engine. (#11602)
This commit disable cast to array type on origin exec engine, except cast
varchar to array type.
---
.../java/org/apache/doris/analysis/CastExpr.java | 10 +-
...=> test_cast_array_functions_by_literal.groovy} | 121 +
2 files changed, 130 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
index bb1f8f4ef6..8b63d461ff 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
@@ -29,6 +29,7 @@ import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.VectorizedUtil;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.thrift.TExpr;
import org.apache.doris.thrift.TExprNode;
@@ -300,10 +301,17 @@ public class CastExpr extends Expr {
searchDesc, Function.CompareMode.IS_IDENTICAL);
}
} else if (type.isArrayType()) {
-fn = ScalarFunction.createBuiltin(getFnName(Type.ARRAY),
+if (VectorizedUtil.isVectorized()) {
+// Vec engine don't need a scala cast function, but we still
create one to pass the check.
+fn = ScalarFunction.createBuiltin("CAST", type,
Lists.newArrayList(), false,
+"", null, null, true);
+} else if (childType.isVarchar()) {
+// only support varchar cast to array for origin exec engine.
+fn = ScalarFunction.createBuiltin(getFnName(Type.ARRAY),
type, Function.NullableMode.ALWAYS_NULLABLE,
Lists.newArrayList(Type.VARCHAR), false,
"doris::CastFunctions::cast_to_array_val", null, null,
true);
+}
}
if (fn == null) {
diff --git
a/regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions.groovy
b/regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions_by_literal.groovy
similarity index 51%
rename from
regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions.groovy
rename to
regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions_by_literal.groovy
index 83f891f9ee..e046987202 100644
---
a/regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions.groovy
+++
b/regression-test/suites/query/sql_functions/array_functions/test_cast_array_functions_by_literal.groovy
@@ -113,4 +113,125 @@ suite("test_cast_array_functions", "query") {
exception "errCode = 2,"
}
+// Not Vectorized Engine
+sql """ set enable_vectorized_engine = false """
+
+// == cast scalar to array ===
+
+test {
+sql "select cast(cast('x' as char) as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(cast('x' as string) as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(1 as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(999.999 as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(cast(999.999 as double) as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(cast(999.999 as decimal) as array)"
+// check exception message contains
+exception "errCode = 2,"
+}
+// == cast array to scalar ===
+
+test {
+sql "select cast(['x'] as char)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(['x'] as varchar)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+test {
+sql "select cast(['x'] as string)"
+// check exception message contains
+exception "errCode = 2,"
+}
+
+
[GitHub] [doris] morningman commented on pull request #11091: [doc] Fix two urls at README.md
morningman commented on PR #11091: URL: https://github.com/apache/doris/pull/11091#issuecomment-1215018369 The README has been changed. close this PR. Thank you for your contribution @shenxingwuying -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman closed pull request #11091: [doc] Fix two urls at README.md
morningman closed pull request #11091: [doc] Fix two urls at README.md URL: https://github.com/apache/doris/pull/11091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11189: [doc] update get-starting.md
morningman commented on PR #11189: URL: https://github.com/apache/doris/pull/11189#issuecomment-1215021951 @emerkfu please rebase to resolve conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11458: [enhancement](tracing) append the profile counter to trace.
morningman merged PR #11458: URL: https://github.com/apache/doris/pull/11458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](tracing) append the profile counter to trace. (#11458)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5104982614 [enhancement](tracing) append the profile counter to trace.
(#11458)
5104982614 is described below
commit 5104982614009fb954070c3c07802391ceccc918
Author: luozenglin <37725793+luozeng...@users.noreply.github.com>
AuthorDate: Mon Aug 15 21:36:38 2022 +0800
[enhancement](tracing) append the profile counter to trace. (#11458)
1. append the profile counter and infos to span attributes.
2. output traceid to audit log.
---
be/src/exec/data_sink.h| 1 +
be/src/exec/exec_node.cpp | 2 +
be/src/runtime/plan_fragment_executor.cpp | 1 +
be/src/util/runtime_profile.cpp| 85 +-
be/src/util/runtime_profile.h | 29 ++--
be/src/util/telemetry/telemetry.cpp| 6 +-
be/src/util/telemetry/telemetry.h | 13
be/src/vec/exec/volap_scan_node.cpp| 19 +
.../java/org/apache/doris/plugin/AuditEvent.java | 8 ++
.../java/org/apache/doris/qe/ConnectProcessor.java | 9 ++-
.../java/org/apache/doris/qe/StmtExecutor.java | 68 ++---
11 files changed, 171 insertions(+), 70 deletions(-)
diff --git a/be/src/exec/data_sink.h b/be/src/exec/data_sink.h
index 952e51e5cd..423e60a74a 100644
--- a/be/src/exec/data_sink.h
+++ b/be/src/exec/data_sink.h
@@ -70,6 +70,7 @@ public:
// It must be okay to call this multiple times. Subsequent calls should
// be ignored.
virtual Status close(RuntimeState* state, Status exec_status) {
+profile()->add_to_span();
_closed = true;
return Status::OK();
}
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index be48218fb6..f581a11c40 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -284,6 +284,8 @@ Status ExecNode::close(RuntimeState* state) {
state->exec_env()->buffer_pool()->DeregisterClient(&_buffer_pool_client);
}
+runtime_profile()->add_to_span();
+
return result;
}
diff --git a/be/src/runtime/plan_fragment_executor.cpp
b/be/src/runtime/plan_fragment_executor.cpp
index 381649c730..6599785a41 100644
--- a/be/src/runtime/plan_fragment_executor.cpp
+++ b/be/src/runtime/plan_fragment_executor.cpp
@@ -688,6 +688,7 @@ void PlanFragmentExecutor::close() {
<< print_id(_runtime_state->fragment_instance_id());
}
+profile()->add_to_span();
_closed = true;
}
diff --git a/be/src/util/runtime_profile.cpp b/be/src/util/runtime_profile.cpp
index 4babd77269..51e12397f1 100644
--- a/be/src/util/runtime_profile.cpp
+++ b/be/src/util/runtime_profile.cpp
@@ -29,7 +29,6 @@
#include "util/container_util.hpp"
#include "util/cpu_info.h"
#include "util/debug_util.h"
-#include "util/pretty_printer.h"
#include "util/thrift_util.h"
#include "util/url_coding.h"
@@ -398,7 +397,7 @@ RuntimeProfile::Counter* RuntimeProfile::add_counter(const
std::string& name, TU
DCHECK(parent_counter_name == ROOT_COUNTER ||
_counter_map.find(parent_counter_name) != _counter_map.end());
-Counter* counter = _pool->add(new Counter(type, 0, name));
+Counter* counter = _pool->add(new Counter(type, 0));
_counter_map[name] = counter;
std::set* child_counters =
find_or_insert(&_child_counter_map, parent_counter_name,
std::set());
@@ -528,6 +527,88 @@ void RuntimeProfile::pretty_print(std::ostream* s, const
std::string& prefix) co
}
}
+void RuntimeProfile::add_to_span() {
+auto span = opentelemetry::trace::Tracer::GetCurrentSpan();
+if (!span->IsRecording() || _added_to_span) {
+return;
+}
+_added_to_span = true;
+
+CounterMap counter_map;
+ChildCounterMap child_counter_map;
+{
+std::lock_guard l(_counter_map_lock);
+counter_map = _counter_map;
+child_counter_map = _child_counter_map;
+}
+
+auto total_time = counter_map.find("TotalTime");
+DCHECK(total_time != counter_map.end());
+
+// profile name like "VDataBufferSender
(dst_fragment_instance_id=-2608c96868f3b77d--713968f450bfbe0d):"
+// to "VDataBufferSender"
+auto i = _name.find_first_of("(: ");
+auto short_name = _name.substr(0, i);
+span->SetAttribute("TotalTime", print_json_counter(short_name,
total_time->second));
+
+{
+std::lock_guard l(_info_strings_lock);
+for (const std::string& key : _info_strings_display_order) {
+// nlohmann json will core dump when serializing 'KeyRanges', here
temporarily skip it.
+if (key.compare("KeyRanges") == 0) {
+continue;
+}
+span->SetAttribute(key, print_json_info(short_name,
_info_strings.f
[GitHub] [doris] morningman commented on pull request #11473: [Bug][Vectorized] Fix nullptr deref in data sink
morningman commented on PR #11473: URL: https://github.com/apache/doris/pull/11473#issuecomment-1215024076 @amosbird please rebase to trigger the checks again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11494: [feature](schema change) Light schema change support rollup/mv
morningman merged PR #11494: URL: https://github.com/apache/doris/pull/11494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature](schema change) Light schema change support rollup (#11494)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d2d4423c88 [feature](schema change) Light schema change support rollup
(#11494)
d2d4423c88 is described below
commit d2d4423c8825b058649b5e91dd0d55aad8fbc2ae
Author: Lei Zhang <1091517...@qq.com>
AuthorDate: Mon Aug 15 21:39:27 2022 +0800
[feature](schema change) Light schema change support rollup (#11494)
1. Move max colUniqueId from OlapTable to IndexMeta.
2. Add updateSlotUniqueId.
---
.../doris/alter/MaterializedViewHandler.java | 41 +++-
.../java/org/apache/doris/alter/RollupJobV2.java | 15 +-
.../apache/doris/alter/SchemaChangeHandler.java| 209 -
.../org/apache/doris/alter/SchemaChangeJobV2.java | 23 ++-
.../main/java/org/apache/doris/catalog/Column.java | 1 -
.../main/java/org/apache/doris/catalog/Env.java| 6 +-
.../doris/catalog/MaterializedIndexMeta.java | 33
.../java/org/apache/doris/catalog/OlapTable.java | 43 ++---
.../org/apache/doris/catalog/TableProperty.java| 14 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 8 +-
.../doris/datasource/InternalDataSource.java | 18 +-
.../org/apache/doris/planner/OlapScanNode.java | 21 +++
.../doris/alter/SchemaChangeHandlerTest.java | 6 +-
13 files changed, 270 insertions(+), 168 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
index 504db3aa71..ce0e30f3bf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
@@ -504,10 +504,18 @@ public class MaterializedViewHandler extends AlterHandler
{
newMVColumns.add(new Column(olapTable.getSequenceCol()));
}
-// set MV column unique id to Column.COLUMN_UNIQUE_ID_INIT_VALUE
support old unique id rule.
-newMVColumns.stream().forEach(column -> {
-column.setUniqueId(Column.COLUMN_UNIQUE_ID_INIT_VALUE);
-});
+if (olapTable.getEnableLightSchemaChange()) {
+int nextColUniqueId = Column.COLUMN_UNIQUE_ID_INIT_VALUE + 1;
+for (Column column : newMVColumns) {
+column.setUniqueId(nextColUniqueId);
+nextColUniqueId++;
+}
+} else {
+newMVColumns.stream().forEach(column -> {
+column.setUniqueId(Column.COLUMN_UNIQUE_ID_INIT_VALUE);
+});
+}
+LOG.debug("lightSchemaChange:{}, newMVColumns:{}",
olapTable.getEnableLightSchemaChange(), newMVColumns);
return newMVColumns;
}
@@ -549,25 +557,27 @@ public class MaterializedViewHandler extends AlterHandler
{
for (Column column : olapTable.getSchemaByIndexId(baseIndexId, true)) {
baseColumnNameToColumn.put(column.getName(), column);
}
+LOG.debug("baseSchema:{}", olapTable.getSchemaByIndexId(baseIndexId,
true));
if (keysType.isAggregationFamily()) {
int keysNumOfRollup = 0;
for (String columnName : rollupColumnNames) {
-Column oneColumn = baseColumnNameToColumn.get(columnName);
-if (oneColumn == null) {
+Column baseColumn = baseColumnNameToColumn.get(columnName);
+if (baseColumn == null) {
throw new DdlException("Column[" + columnName + "] does
not exist");
}
-if (oneColumn.isKey() && meetValue) {
+if (baseColumn.isKey() && meetValue) {
throw new DdlException("Invalid column order. value should
be after key");
}
-if (oneColumn.isKey()) {
+if (baseColumn.isKey()) {
keysNumOfRollup += 1;
hasKey = true;
} else {
meetValue = true;
-if (oneColumn.getAggregationType().isReplaceFamily()) {
+if (baseColumn.getAggregationType().isReplaceFamily()) {
meetReplaceValue = true;
}
}
+Column oneColumn = new Column(baseColumn);
rollupSchema.add(oneColumn);
}
@@ -696,6 +706,19 @@ public class MaterializedViewHandler extends AlterHandler {
}
}
}
+if (olapTable.getEnableLightSchemaChange()) {
+int nextColUniqueId = Column.COLUMN_UNIQUE_ID_INIT_VALUE + 1;
+for (Column column : rollupSchema) {
+column.setUniqueId(nextColUniqueId);
+nextColUniqueId++;
+}
+
[GitHub] [doris] morningman commented on pull request #11565: [fix] (docs) Fix Data type: string length and parameter description
morningman commented on PR #11565: URL: https://github.com/apache/doris/pull/11565#issuecomment-1215027391 > 实际上目前的string不适合存储大字符串，所以这个可以理解为是当前Doris的功能限制，而不是一个可供用户选择的方案 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11741: [docs](fix)[fix some error path]
morningman merged PR #11741: URL: https://github.com/apache/doris/pull/11741 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [docs](fix)fix some error path (#11741)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ce10151273 [docs](fix)fix some error path (#11741)
ce10151273 is described below
commit ce101512736b0d49811ee1bb045c9299c9c54d77
Author: daikon <1059907...@qq.com>
AuthorDate: Mon Aug 15 21:43:56 2022 +0800
[docs](fix)fix some error path (#11741)
---
docs/en/docs/ecosystem/udf/native-user-defined-function.md| 8
docs/zh-CN/docs/ecosystem/udf/native-user-defined-function.md | 8
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/docs/en/docs/ecosystem/udf/native-user-defined-function.md
b/docs/en/docs/ecosystem/udf/native-user-defined-function.md
index 1ccacd3047..289b3a46b9 100644
--- a/docs/en/docs/ecosystem/udf/native-user-defined-function.md
+++ b/docs/en/docs/ecosystem/udf/native-user-defined-function.md
@@ -160,17 +160,17 @@ Running `sh build.sh` in the root directory of Doris will
generate a static libr
```
# Include udf
-include_directories(thirdparty/include)
+include_directories(../thirdparty/include)
# Set all libraries
add_library(udf STATIC IMPORTED)
-set_target_properties(udf PROPERTIES IMPORTED_LOCATION
thirdparty/lib/libDorisUdf.a)
+set_target_properties(udf PROPERTIES IMPORTED_LOCATION
../thirdparty/lib/libDorisUdf.a)
# where to put generated libraries
-set(LIBRARY_OUTPUT_PATH "${BUILD_DIR}/src/udf_samples")
+set(LIBRARY_OUTPUT_PATH "src/udf_samples")
# where to put generated binaries
-set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/src/udf_samples")
+set(EXECUTABLE_OUTPUT_PATH "src/udf_samples")
add_library(udfsample SHARED udf_sample.cpp)
target_link_libraries(udfsample
diff --git a/docs/zh-CN/docs/ecosystem/udf/native-user-defined-function.md
b/docs/zh-CN/docs/ecosystem/udf/native-user-defined-function.md
index 4e6650dd0e..8b04988f94 100644
--- a/docs/zh-CN/docs/ecosystem/udf/native-user-defined-function.md
+++ b/docs/zh-CN/docs/ecosystem/udf/native-user-defined-function.md
@@ -161,17 +161,17 @@ UDF 能满足的分析需求分为两种：UDF 和 UDAF。本文中的 UDF 指
```
# Include udf
-include_directories(thirdparty/include)
+include_directories(../thirdparty/include)
# Set all libraries
add_library(udf STATIC IMPORTED)
-set_target_properties(udf PROPERTIES IMPORTED_LOCATION
thirdparty/lib/libDorisUdf.a)
+set_target_properties(udf PROPERTIES IMPORTED_LOCATION
../thirdparty/lib/libDorisUdf.a)
# where to put generated libraries
-set(LIBRARY_OUTPUT_PATH "${BUILD_DIR}/src/udf_samples")
+set(LIBRARY_OUTPUT_PATH "src/udf_samples")
# where to put generated binaries
-set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/src/udf_samples")
+set(EXECUTABLE_OUTPUT_PATH "src/udf_samples")
add_library(udfsample SHARED udf_sample.cpp)
target_link_libraries(udfsample
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] amosbird commented on pull request #11473: [Bug][Vectorized] Fix nullptr deref in data sink
amosbird commented on PR #11473: URL: https://github.com/apache/doris/pull/11473#issuecomment-1215041396 Is there a way to rebase right here? (I don't have access to the repo now). In ClickHouse we can:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju opened a new pull request, #11798: [fix](group_concat) group_concat be crash in vectorized engine
cambyzju opened a new pull request, #11798: URL: https://github.com/apache/doris/pull/11798 # Proposed changes Issue Number: close #7288 ## Problem summary The key points to reproduce the problem: 1. create table with string NOT NULL column; 2. call left join with group_concat; The reason is because: 1. aggregate function framework pick combined null function by arguments type; 2. as the input argument is NOT NULL, so the origin group_concat is selected; 3. as a result, we need to consider insert_result_into with input column = ColumnNullabe(ColumnString); ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11785: [Enhancement](array-type) enable_array_type flag update
github-actions[bot] commented on PR #11785: URL: https://github.com/apache/doris/pull/11785#issuecomment-1215133956 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11785: [Enhancement](array-type) enable_array_type flag update
github-actions[bot] commented on PR #11785: URL: https://github.com/apache/doris/pull/11785#issuecomment-1215134000 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11624: [regression-test](p0) add p0 test cases
morningman merged PR #11624: URL: https://github.com/apache/doris/pull/11624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11624: [regression-test](p0) add p0 test cases
morningman commented on PR #11624: URL: https://github.com/apache/doris/pull/11624#issuecomment-1215138308 #11787 #11786 There are 2 issues need to be fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TsukiokaKogane commented on pull request #11029: [fix] Fix potential unhandled exception cause data inconsistency
TsukiokaKogane commented on PR #11029: URL: https://github.com/apache/doris/pull/11029#issuecomment-1215170316 > @TsukiokaKogane please rebase to pass the check done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new issue, #11799: [Bug] Analyze failed Unexpected exception: Index: 8, Size: 7
morningman opened a new issue, #11799: URL: https://github.com/apache/doris/issues/11799 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? Analyze fail with error: ``` ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Index: 8, Size: 7 ``` ### What You Expected? No error ### How to Reproduce? ``` CREATE TABLE `baseall` ( `k0` boolean NULL, `k1` tinyint(4) NULL, `k2` smallint(6) NULL, `k3` int(11) NULL, `k4` bigint(20) NULL, `k5` decimal(9, 3) NULL, `k6` char(5) NULL, `k10` date NULL, `k11` datetime NULL, `k7` varchar(20) NULL, `k8` double MAX NULL, `k9` float SUM NULL, `k12` text REPLACE NULL, `k13` largeint(40) REPLACE NULL ) ENGINE=OLAP AGGREGATE KEY(`k0`, `k1`, `k2`, `k3`, `k4`, `k5`, `k6`, `k10`, `k11`, `k7`) COMMENT 'OLAP' DISTRIBUTED BY HASH(`k1`) BUCKETS 5 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ); CREATE TABLE `test` ( `k0` boolean NULL, `k1` tinyint(4) NULL, `k2` smallint(6) NULL, `k3` int(11) NULL, `k4` bigint(20) NULL, `k5` decimal(9, 3) NULL, `k6` char(5) NULL, `k10` date NULL, `k11` datetime NULL, `k7` varchar(20) NULL, `k8` double MAX NULL, `k9` float SUM NULL, `k12` text REPLACE_IF_NOT_NULL NULL, `k13` largeint(40) REPLACE NULL ) ENGINE=OLAP AGGREGATE KEY(`k0`, `k1`, `k2`, `k3`, `k4`, `k5`, `k6`, `k10`, `k11`, `k7`) COMMENT 'OLAP' DISTRIBUTED BY HASH(`k1`) BUCKETS 5 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ); select a.k1, a.k2, b.k1 from baseall a join (select * from test where k6 = "false" order by k1 limit 3) b on a.k1 = b.k1 where a.k2 > 0 order by 1; ``` ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju closed pull request #11798: [fix](group_concat) group_concat be crash in vectorized engine
cambyzju closed pull request #11798: [fix](group_concat) group_concat be crash in vectorized engine URL: https://github.com/apache/doris/pull/11798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on pull request #11798: [fix](group_concat) group_concat be crash in vectorized engine
cambyzju commented on PR #11798: URL: https://github.com/apache/doris/pull/11798#issuecomment-1215197109 https://github.com/apache/doris/pull/11700 already fix this case from FE side. Make sure input column type is always nullabe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #11800: (fix)[regression-test] add qt_having1
morningman opened a new pull request, #11800: URL: https://github.com/apache/doris/pull/11800 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #11801: [improvement](agg) Serialize the fixed-length aggregation results with corresponding columns instead of ColumnString
mrhhsg opened a new pull request, #11801: URL: https://github.com/apache/doris/pull/11801 # Proposed changes Issue Number: close #xxx ## Problem Summary Some aggregate results are fixed-length types, serializing them into `ColumnString` is not unnecessary. ## Checklist(Required) 1. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [x] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
