[GitHub] [doris] github-actions[bot] commented on pull request #13005: [chore](docs)Add local documentation compilation checks to reduce subsequent operations caused by pipeline build failures
github-actions[bot] commented on PR #13005:
URL: https://github.com/apache/doris/pull/13005#issuecomment-1259070292
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3133148046";) output.
shellcheck errors
```
'shellcheck ' found no issues.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- docs/build_check.sh.orig
+++ docs/build_check.sh
@@ -33,28 +33,28 @@
rm -rf website/community
cp -R docs/en/community website/
rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs/*
-mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
+mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
cp -R docs/zh-CN/docs/*
website/i18n/zh-CN/docusaurus-plugin-content-docs/current/
cp docs/dev.json
website/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
-rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
+rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current
cp -R docs/zh-CN/community/*
website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/
cp -R docs/sidebarsCommunity.json website/
cp -R docs/sidebars.json website/
cp -R docs/images website/static/
-sed '2,3d' website/versions.json > website/versions.json1
+sed '2,3d' website/versions.json >website/versions.json1
rm -rf website/versions.json
mv website/versions.json1 website/versions.json
-sed '123,128d' website/docusaurus.config.js > website/docusaurus.config.js1
+sed '123,128d' website/docusaurus.config.js >website/docusaurus.config.js1
rm -rf website/docusaurus.config.js
mv website/docusaurus.config.js1 website/docusaurus.config.js
cd website
npm install -g npm@8.19.1
npm install -g yarn
yarn cache clean
-yarn && yarn build
+yarn && yarn build
cd ../
-rm -rf website
+rm -rf website
echo "***"
echo "Docs build check pass"
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu commented on issue #13010: [Bug] show proc "/cluster_health/tablet_health" returns wrong result
dutyu commented on issue #13010: URL: https://github.com/apache/doris/issues/13010#issuecomment-1259086821 I found that this problem has been fixed in master branch. But in 1.1-lts, this problem is still exists. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dutyu closed issue #13010: [Bug] show proc "/cluster_health/tablet_health" returns wrong result
dutyu closed issue #13010: [Bug] show proc "/cluster_health/tablet_health" returns wrong result URL: https://github.com/apache/doris/issues/13010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new issue, #13011: [Bug] core dump on load with lateral view
BiteThet opened a new issue, #13011:
URL: https://github.com/apache/doris/issues/13011
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
master
### What's Wrong?
```sql
CREATE TABLE `test` (
`k1` smallint NULL,
`k2` int NULL,
`k3` bigint NULL,
`k4` largeint NULL
) ENGINE=OLAP
DUPLICATE KEY(`k1`,`k2`,`k3`,`k4`)
DISTRIBUTED BY HASH(`k1`) BUCKETS 1
PROPERTIES("replication_num" = "1");
insert into test select e1,e1,e1,e1 from (select 1 k1) as t lateral view
explode_numbers(1) tmp1 as e1;
```
```cpp
=
==225980==ERROR: AddressSanitizer: global-buffer-overflow on address
0x563c7fb77140 at pc 0x563c86f50d6b bp 0x7f3fad3370a0 sp 0x7f3fad337098
READ of size 2 at 0x563c7fb77140 thread T631
#0 0x563c86f50d6a in
doris::vectorized::ColumnVector::insert_indices_from(doris::vectorized::IColumn
const&, int const*, int const*)
/mnt/disk1/pxl/doris/be/src/vec/columns/column_vector.cpp:365:70
#1 0x563c86e722a8 in
doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn
const&, int const*, int const*)
/mnt/disk1/pxl/doris/be/src/vec/columns/column_nullable.cpp:241:25
#2 0x563c872cab97 in
doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const*, int
const*, int const*) /mnt/disk1/pxl/doris/be/src/vec/core/block.cpp:966:14
#3 0x563c8317b1a6 in doris::MemTable::insert(doris::vectorized::Block
const*, std::vector> const&)
/mnt/disk1/pxl/doris/be/src/olap/memtable.cpp:192:26
#4 0x563c835bcd5f in doris::DeltaWriter::write(doris::vectorized::Block
const*, std::vector> const&)
/mnt/disk1/pxl/doris/be/src/olap/delta_writer.cpp:219:17
#5 0x563c84835026 in doris::Status
doris::TabletsChannel::add_batch(doris::PTabletWriterAddBlockRequest const&,
doris::PTabletWriterAddBlockResult*)
/mnt/disk1/pxl/doris/be/src/runtime/tablets_channel.cpp:420:47
#6 0x563c84acb5ca in doris::Status
doris::LoadChannel::add_batch(doris::PTabletWriterAddBlockRequest const&,
doris::PTabletWriterAddBlockResult*)
/mnt/disk1/pxl/doris/be/src/runtime/load_channel.h:151:13
#7 0x563c84ac89f3 in doris::Status
doris::LoadChannelMgr::add_batch(doris::PTabletWriterAddBlockRequest const&,
doris::PTabletWriterAddBlockResult*)
/mnt/disk1/pxl/doris/be/src/runtime/load_channel_mgr.h:144:5
#8 0x563c84a9ee48 in
doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*,
doris::PTabletWriterAddBlockRequest const*,
doris::PTabletWriterAddBlockResult*,
google::protobuf::Closure*)::$_1::operator()() const
/mnt/disk1/pxl/doris/be/src/service/internal_service.cpp:268:54
#9 0x563c84a9ea64 in void std::__invoke_impl(std::__invoke_other,
doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*,
doris::PTabletWriterAddBlockRequest const*,
doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_1&)
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#10 0x563c84a9ea04 in std::enable_if,
void>::type std::__invoke_r(doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*,
doris::PTabletWriterAddBlockRequest const*,
doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::$_1&)
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#11 0x563c84a9e86c in std::_Function_handler::_M_invoke(std::_Any_data const&)
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#12 0x563c8439ad12 in std::function::operator()() const
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
#13 0x563c843976e0 in doris::PriorityThreadPool::work_thread(int)
/mnt/disk1/pxl/doris/be/src/util/priority_thread_pool.hpp:136:17
#14 0x563c84398fe2 in void std::__invoke_impl(std::__invoke_memfun_deref, void (doris::PriorityThreadPool::*
const&)(int), doris::PriorityThreadPool*&, int&)
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
#15 0x563c84398eb4 in std::__invoke_result::type std::__invoke(void (doris::PriorityThreadPool::*
const&)(int), doris::PriorityThreadPool*&, int&)
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#16 0x563c84398e84 in decltype(std::__invoke((*this)._M_pmf,
std::forward(fp), std::forward(fp)))
std::_Mem_fn_base::operator()(doris::PriorityThreadPool*&, int&) const
/mnt/disk1/pxl/ldb/ldb_toolchain/bin/../lib/gcc
[GitHub] [doris] yiguolei merged pull request #12854: [test](join)add join case5
yiguolei merged PR #12854: URL: https://github.com/apache/doris/pull/12854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression-test](join)add join case5 #12854
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3dfcfc69ee [regression-test](join)add join case5 #12854
3dfcfc69ee is described below
commit 3dfcfc69ee9244c1834bfe83c7d3f1b32a7e140a
Author: zy-kkk
AuthorDate: Tue Sep 27 15:47:36 2022 +0800
[regression-test](join)add join case5 #12854
---
regression-test/data/query/join/test_join5.out | 26
.../suites/query/join/test_join5.groovy| 169 +
2 files changed, 195 insertions(+)
diff --git a/regression-test/data/query/join/test_join5.out
b/regression-test/data/query/join/test_join5.out
new file mode 100644
index 00..6ab749f3a7
--- /dev/null
+++ b/regression-test/data/query/join/test_join5.out
@@ -0,0 +1,26 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !join1 --
+0
+1
+
+
+-- !join2 --
+0 1
+
+-- !join3 --
+53 \N \N
+
+-- !join4 --
+53 \N \N
+
+-- !join5 --
+A p 2 -1
+B q 0 -1
+C \N 0 -1
+
+-- !join5 --
+1 1 1 1
+
+-- !join6 --
+1 1 1 1
+
diff --git a/regression-test/suites/query/join/test_join5.groovy
b/regression-test/suites/query/join/test_join5.groovy
new file mode 100644
index 00..3f7a12bfae
--- /dev/null
+++ b/regression-test/suites/query/join/test_join5.groovy
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_join5", "query,p0") {
+def DBname = "regression_test_join5"
+sql "DROP DATABASE IF EXISTS ${DBname}"
+sql "CREATE DATABASE IF NOT EXISTS ${DBname}"
+sql "use ${DBname}"
+
+def tbName1 = "tt3"
+def tbName2 = "tt4"
+def tbName3 = "tt4x"
+
+sql "DROP TABLE IF EXISTS ${tbName1};"
+sql "DROP TABLE IF EXISTS ${tbName2};"
+sql "DROP TABLE IF EXISTS ${tbName3};"
+
+sql """create table ${tbName1} (f1 int, f2 text) DISTRIBUTED BY HASH(f1)
properties("replication_num" = "1");"""
+sql """create table ${tbName2} (f1 int) DISTRIBUTED BY HASH(f1)
properties("replication_num" = "1");"""
+sql """create table ${tbName3} (c1 int, c2 int, c3 int) DISTRIBUTED BY
HASH(c1) properties("replication_num" = "1");"""
+
+sql "insert into ${tbName1} values (1,null);"
+sql "insert into ${tbName1} values (null,null);"
+sql "insert into ${tbName2} values (0),(1),();"
+sql "insert into ${tbName3} values (0,1,);"
+
+qt_join1 """
+SELECT a.f1
+FROM ${tbName2} a
+LEFT JOIN (
+SELECT b.f1
+FROM ${tbName1} b LEFT JOIN ${tbName1} c ON (b.f1 = c.f1)
+WHERE c.f1 IS NULL
+) AS d ON (a.f1 = d.f1)
+WHERE d.f1 IS NULL
+ORDER BY 1;
+"""
+
+qt_join2 """
+select * from ${tbName3} t1
+where not exists (
+ select 1 from ${tbName3} t2
+left join ${tbName3} t3 on t2.c3 = t3.c1
+left join ( select t5.c1 as c1
+from ${tbName3} t4 left join ${tbName3} t5 on
t4.c2 = t5.c1
+ ) a1 on t3.c2 = a1.c1
+ where t1.c1 = t2.c2
+)
+ORDER BY 1;
+"""
+
+sql "DROP TABLE IF EXISTS ${tbName1};"
+sql "DROP TABLE IF EXISTS ${tbName1};"
+sql "DROP TABLE IF EXISTS ${tbName3};"
+
+def tbName4 = "uq1"
+def tbName5 = "uq2"
+def tbName6 = "uq3"
+def tbName7 = "uqv1"
+
+sql "DROP TABLE IF EXISTS ${tbName4};"
+sql "DROP TABLE IF EXISTS ${tbName5};"
+sql "DROP TABLE IF EXISTS ${tbName6};"
+
+
+sql """create table ${tbName4} (f1 int) UNIQUE KEY (f1) DISTRIBUTED BY
HASH(f1) properties("replication_num" = "1");"""
+sql """create table ${tbName5} (f2 int) UNIQUE KEY (f2) DISTRIBUTED BY
HASH(f2) properties("replication_num" = "1");"""
+sql """create table ${tbName6} (f3 int) UNIQUE KEY (f3) DISTRIBUTED BY
HASH(f3) properties("replication_num" = "1");"""
+
+sq
[GitHub] [doris] dutyu opened a new pull request, #13012: [branch-1.1-lts](cherry-pick) fix wrong result of tablet health stmt …
dutyu opened a new pull request, #13012: URL: https://github.com/apache/doris/pull/13012 …(#13010) # Proposed changes Issue Number: close #13010 ## Problem summary This problem has been fixed in master branch, but not in branch-1.1-lts. We will use branch-1.1-lts in production env so we need to fix this problem. See #13010 ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] **No** - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] **No** - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] **No** - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] **No** 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] **No** ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12844: [test](join)add join case3
yiguolei merged PR #12844: URL: https://github.com/apache/doris/pull/12844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [test](join)add join case5 #12854
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cbdef66757 [test](join)add join case5 #12854
cbdef66757 is described below
commit cbdef6675709cc887b0fbee9bf07abae351c3404
Author: zy-kkk
AuthorDate: Tue Sep 27 15:48:36 2022 +0800
[test](join)add join case5 #12854
---
regression-test/data/query/join/test_join3.out | 28 +++
.../suites/query/join/test_join2.groovy| 4 +-
.../suites/query/join/test_join3.groovy| 96 ++
.../suites/query/join/test_join4.groovy| 4 +-
4 files changed, 128 insertions(+), 4 deletions(-)
diff --git a/regression-test/data/query/join/test_join3.out
b/regression-test/data/query/join/test_join3.out
new file mode 100644
index 00..e7a6efa415
--- /dev/null
+++ b/regression-test/data/query/join/test_join3.out
@@ -0,0 +1,28 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !join1 --
+\N \N \N \N dd 33
+\N \N cc 22 cc 23
+\N \N ee 42 \N \N
+bb 11 bb 12 bb 13
+
+-- !join2 --
+bb 12 bb 13
+cc 22 cc 23
+
+-- !join3 --
+bb 12 bb 13
+cc 22 cc 23
+ee 42 \N \N
+
+-- !join4 --
+\N \N dd 33
+bb 12 bb 13
+cc 22 cc 23
+ee 42 \N \N
+
+-- !join7 --
+\N \N bb 2
+\N \N cc 2
+\N \N ee 2
+bb 11 \N 2
+
diff --git a/regression-test/suites/query/join/test_join2.groovy
b/regression-test/suites/query/join/test_join2.groovy
index 60feb3207f..9070eeb82f 100644
--- a/regression-test/suites/query/join/test_join2.groovy
+++ b/regression-test/suites/query/join/test_join2.groovy
@@ -16,7 +16,7 @@
// under the License.
suite("test_join2", "query,p0") {
-def DBname = "test_join2"
+def DBname = "regression_test_join2"
def TBname1 = "J1_TBL"
def TBname2 = "J2_TBL"
@@ -158,4 +158,4 @@ suite("test_join2", "query,p0") {
sql "DROP TABLE IF EXISTS ${TBname1};"
sql "DROP TABLE IF EXISTS ${TBname2};"
sql "DROP DATABASE IF EXISTS ${DBname};"
-}
\ No newline at end of file
+}
diff --git a/regression-test/suites/query/join/test_join3.groovy
b/regression-test/suites/query/join/test_join3.groovy
new file mode 100644
index 00..20a567b3a7
--- /dev/null
+++ b/regression-test/suites/query/join/test_join3.groovy
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_join3", "query,p0") {
+def DBname = "regression_test_join3"
+sql "DROP DATABASE IF EXISTS ${DBname}"
+sql "CREATE DATABASE IF NOT EXISTS ${DBname}"
+sql "use ${DBname}"
+
+def tbName1 = "t1"
+def tbName2 = "t2"
+def tbName3 = "t3"
+
+sql """CREATE TABLE ${tbName1} (name varchar(255), n INTEGER) DISTRIBUTED
BY HASH(name) properties("replication_num" = "1");"""
+sql """CREATE TABLE ${tbName2} (name varchar(255), n INTEGER) DISTRIBUTED
BY HASH(name) properties("replication_num" = "1");"""
+sql """CREATE TABLE ${tbName3} (name varchar(255), n INTEGER) DISTRIBUTED
BY HASH(name) properties("replication_num" = "1");"""
+
+sql "INSERT INTO ${tbName1} VALUES ( 'bb', 11 );"
+sql "INSERT INTO ${tbName2} VALUES ( 'bb', 12 );"
+sql "INSERT INTO ${tbName2} VALUES ( 'cc', 22 );"
+sql "INSERT INTO ${tbName2} VALUES ( 'ee', 42 );"
+sql "INSERT INTO ${tbName3} VALUES ( 'bb', 13 );"
+sql "INSERT INTO ${tbName3} VALUES ( 'cc', 23 );"
+sql "INSERT INTO ${tbName3} VALUES ( 'dd', 33 );"
+
+qt_join1 """
+SELECT * FROM ${tbName1} FULL JOIN ${tbName2} USING (name) FULL
JOIN ${tbName3} USING (name) ORDER BY 1,2,3,4,5,6;
+"""
+qt_join2 """
+SELECT * FROM
+(SELECT * FROM ${tbName2}) as s2
+INNER JOIN
+(SELECT * FROM ${tbName3}) s3
+USING (name)
+ORDER BY 1,2,3,4;
+"""
+qt_join3 """
+
[GitHub] [doris] yiguolei merged pull request #12807: [Enhancement](optimize) optimize for insert_indices_from
yiguolei merged PR #12807: URL: https://github.com/apache/doris/pull/12807 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](optimize) optimize for insert_indices_from (#12807)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 64988cb3d4 [Enhancement](optimize) optimize for insert_indices_from
(#12807)
64988cb3d4 is described below
commit 64988cb3d4ea177bc3174b8fc71c28db21d075ea
Author: Pxl
AuthorDate: Tue Sep 27 15:49:15 2022 +0800
[Enhancement](optimize) optimize for insert_indices_from (#12807)
---
be/src/vec/columns/column_decimal.h | 5 ++---
be/src/vec/columns/column_vector.cpp | 22 --
2 files changed, 14 insertions(+), 13 deletions(-)
diff --git a/be/src/vec/columns/column_decimal.h
b/be/src/vec/columns/column_decimal.h
index bd1d90e63c..124a7e7ba2 100644
--- a/be/src/vec/columns/column_decimal.h
+++ b/be/src/vec/columns/column_decimal.h
@@ -102,14 +102,13 @@ public:
void insert_indices_from(const IColumn& src, const int* indices_begin,
const int* indices_end) override {
-const Self& src_vec = assert_cast(src);
auto origin_size = size();
auto new_size = indices_end - indices_begin;
data.resize(origin_size + new_size);
+const T* src_data = reinterpret_cast(src.get_raw_data().data);
for (int i = 0; i < new_size; ++i) {
-auto offset = *(indices_begin + i);
-data[origin_size + i] = offset == -1 ? T {} :
src_vec.get_element(offset);
+data[origin_size + i] = src_data[indices_begin[i]];
}
}
diff --git a/be/src/vec/columns/column_vector.cpp
b/be/src/vec/columns/column_vector.cpp
index 2ca35f6948..5c5ec8835c 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -349,20 +349,22 @@ void ColumnVector::insert_range_from(const IColumn&
src, size_t start, size_t
template
void ColumnVector::insert_indices_from(const IColumn& src, const int*
indices_begin,
const int* indices_end) {
-const Self& src_vec = assert_cast(src);
auto origin_size = size();
auto new_size = indices_end - indices_begin;
data.resize(origin_size + new_size);
-for (int i = 0; i < new_size; ++i) {
-int offset = indices_begin[i];
-if constexpr (std::is_same_v) {
-// Now Uint8 use to identify null and non null
-// 1. nullable column : offset == -1 means is null at the here,
set true here
-// 2. real data column : offset == -1 what at is meaningless
-data[origin_size + i] = (offset == -1) ? T {1} :
src_vec.get_element(offset);
-} else {
-data[origin_size + i] = (offset == -1) ? T {0} :
src_vec.get_element(offset);
+const T* src_data = reinterpret_cast(src.get_raw_data().data);
+
+if constexpr (std::is_same_v) {
+// nullmap : indices_begin[i] == -1 means is null at the here, set
true here
+for (int i = 0; i < new_size; ++i) {
+data[origin_size + i] = (indices_begin[i] == -1) +
+(indices_begin[i] != -1) *
src_data[indices_begin[i]];
+}
+} else {
+// real data : indices_begin[i] == -1 what at is meaningless
+for (int i = 0; i < new_size; ++i) {
+data[origin_size + i] = src_data[indices_begin[i]];
}
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13012: [branch-1.1-lts](cherry-pick) fix wrong result of tablet health stmt …
yiguolei merged PR #13012: URL: https://github.com/apache/doris/pull/13012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [branch-1.1-lts](cherry-pick) fix wrong result of tablet health stmt (#13010) (#13012)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new 006bd07171 [branch-1.1-lts](cherry-pick) fix wrong result of tablet
health stmt (#13010) (#13012)
006bd07171 is described below
commit 006bd07171f8830f58e60004854ed060076dd70b
Author: wxy
AuthorDate: Tue Sep 27 15:50:27 2022 +0800
[branch-1.1-lts](cherry-pick) fix wrong result of tablet health stmt
(#13010) (#13012)
Co-authored-by: wangxian...@360shuke.com
---
.../java/org/apache/doris/common/proc/TabletHealthProcDir.java | 10 +-
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletHealthProcDir.java
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletHealthProcDir.java
index 2e4122ca53..54f5d50196 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletHealthProcDir.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletHealthProcDir.java
@@ -285,11 +285,11 @@ public class TabletHealthProcDir implements
ProcDirInterface {
this.colocateMismatchNum += other.colocateMismatchNum;
this.colocateRedundantNum += other.colocateRedundantNum;
this.needFurtherRepairNum += other.needFurtherRepairNum;
-this.unrecoverableNum += unrecoverableNum;
-this.replicaCompactionTooSlowNum +=
replicaCompactionTooSlowNum;
-this.inconsistentNum += inconsistentNum;
-this.oversizeNum += oversizeNum;
-this.cloningNum += cloningNum;
+this.unrecoverableNum += other.unrecoverableNum;
+this.replicaCompactionTooSlowNum +=
other.replicaCompactionTooSlowNum;
+this.inconsistentNum += other.inconsistentNum;
+this.oversizeNum += other.oversizeNum;
+this.cloningNum += other.cloningNum;
return this;
} else if (other.summary) {
return other.reduce(this);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree commented on issue #12760: [Bug] 使用insert-values语句导致be节点挂了
LemonLiTree commented on issue #12760: URL: https://github.com/apache/doris/issues/12760#issuecomment-1259121571 I try it in version 1.1-lts, it's OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on pull request #12987: [feature](nereids) Enhancement for statistics
Kikyou1997 commented on PR #12987: URL: https://github.com/apache/doris/pull/12987#issuecomment-1259124074 @morrySnow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13005: [chore](docs)Add local documentation compilation checks to reduce subsequent operations caused by pipeline build failures
github-actions[bot] commented on PR #13005:
URL: https://github.com/apache/doris/pull/13005#issuecomment-1259124101
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3133253438";) output.
shellcheck errors
```
'shellcheck ' found no issues.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- docs/build_check.sh.orig
+++ docs/build_check.sh
@@ -33,27 +33,27 @@
rm -rf website/community
cp -R docs/en/community website/
rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs/*
-mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
+mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
cp -R docs/zh-CN/docs/*
website/i18n/zh-CN/docusaurus-plugin-content-docs/current/
cp docs/dev.json
website/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
-rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
+rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current
cp -R docs/zh-CN/community/*
website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/
cp -R docs/sidebarsCommunity.json website/
cp -R docs/sidebars.json website/
cp -R docs/images website/static/
-sed '2,3d' website/versions.json > website/versions.json1
+sed '2,3d' website/versions.json >website/versions.json1
rm -rf website/versions.json
mv website/versions.json1 website/versions.json
-sed '123,128d' website/docusaurus.config.js > website/docusaurus.config.js1
+sed '123,128d' website/docusaurus.config.js >website/docusaurus.config.js1
rm -rf website/docusaurus.config.js
mv website/docusaurus.config.js1 website/docusaurus.config.js
cd website
npm install -g npm@8.19.1
npm install -g yarn
yarn cache clean
-yarn && yarn build
-rm -rf website
+yarn && yarn build
+rm -rf website
echo "***"
echo "Docs build check pass"
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #12985: [feature](Nereids): Eliminate outer join
jackwener commented on PR #12985: URL: https://github.com/apache/doris/pull/12985#issuecomment-1259168238 Future: todo use constant viariable to handle it (we can handle more case like nullsafeEqual ..) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #12890: [feature](Nereids): support Alias join reorder and fix bug.
jackwener commented on PR #12890: URL: https://github.com/apache/doris/pull/12890#issuecomment-1259171597 TODO feature: we can separete `Alias project` and `project` to different Rule to reduce time cost -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13001: [optimization](array-type) optimize the help docs of array type
github-actions[bot] commented on PR #13001: URL: https://github.com/apache/doris/pull/13001#issuecomment-1259205627 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13001: [optimization](array-type) optimize the help docs of array type
github-actions[bot] commented on PR #13001: URL: https://github.com/apache/doris/pull/13001#issuecomment-1259205576 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12999: [optimization](array-type) optimize error prompts when sql parser rep…
github-actions[bot] commented on PR #12999: URL: https://github.com/apache/doris/pull/12999#issuecomment-1259209083 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12999: [optimization](array-type) optimize error prompts when sql parser rep…
github-actions[bot] commented on PR #12999: URL: https://github.com/apache/doris/pull/12999#issuecomment-1259209138 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #12866: [enhancement](compaction) introduce segment compaction (#12609)
freemandealer commented on code in PR #12866: URL: https://github.com/apache/doris/pull/12866#discussion_r980977310 ## docs/zh-CN/docs/admin-manual/maint-monitor/be-olap-error-code.md: ## @@ -6,7 +6,7 @@ --- -
[GitHub] [doris] freemandealer commented on a diff in pull request #12866: [enhancement](compaction) introduce segment compaction (#12609)
freemandealer commented on code in PR #12866: URL: https://github.com/apache/doris/pull/12866#discussion_r980977717 ## docs/en/docs/admin-manual/maint-monitor/be-olap-error-code.md: ## @@ -6,7 +6,7 @@ --- -
[GitHub] [doris] HappenLee commented on a diff in pull request #12921: [Improvement](sort) Reuse memory in sort node
HappenLee commented on code in PR #12921:
URL: https://github.com/apache/doris/pull/12921#discussion_r980957363
##
be/src/vec/exec/vsort_node.cpp:
##
@@ -48,13 +48,16 @@ Status VSortNode::init(const TPlanNode& tnode,
RuntimeState* state) {
!row_desc.has_varlen_slots()) {
_sorter.reset(new HeapSorter(_vsort_exec_exprs, _limit, _offset,
_pool, _is_asc_order,
_nulls_first, row_desc));
+reuse_mem = false;
Review Comment:
maybe set default value == `true`, only need init false only here？
##
be/src/vec/core/sort_block.cpp:
##
@@ -114,6 +114,60 @@ void sort_block(Block& block, const SortDescription&
description, UInt64 limit)
}
}
+void sort_block(Block& src_block, Block& dest_block, const SortDescription&
description,
Review Comment:
here many code is same as upper， why only one method， you can call same
`src_block` and `dest_block` to replace upper method
##
be/src/vec/core/sort_block.cpp:
##
@@ -114,6 +114,60 @@ void sort_block(Block& block, const SortDescription&
description, UInt64 limit)
}
}
+void sort_block(Block& src_block, Block& dest_block, const SortDescription&
description,
+UInt64 limit) {
+if (!src_block) {
+return;
+}
+
+/// If only one column to sort by
+if (description.size() == 1) {
+bool reverse = description[0].direction == -1;
+
+const IColumn* column =
+!description[0].column_name.empty()
+?
src_block.get_by_name(description[0].column_name).column.get()
+:
src_block.safe_get_by_position(description[0].column_number).column.get();
+
+IColumn::Permutation perm;
+column->get_permutation(reverse, limit,
description[0].nulls_direction, perm);
+
+size_t columns = src_block.columns();
+for (size_t i = 0; i < columns; ++i) {
+dest_block.replace_by_position(
+i, src_block.get_by_position(i).column->permute(perm,
limit));
+}
+} else {
+size_t size = src_block.rows();
+IColumn::Permutation perm(size);
+for (size_t i = 0; i < size; ++i) {
+perm[i] = i;
+}
+
+if (limit >= size) {
+limit = 0;
+}
+
+ColumnsWithSortDescriptions columns_with_sort_desc =
+get_columns_with_sort_description(src_block, description);
+{
+EqualFlags flags(size, 1);
+EqualRange range {0, size};
+
+for (size_t i = 0; i < columns_with_sort_desc.size(); i++) {
+ColumnSorter sorter(columns_with_sort_desc[i], limit);
Review Comment:
TODO：seems we can reuse the ColumnSorter， do not need construct every time？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #12698: [enhancement](test) add tpch_sf10 cases to p2
dataroaring merged PR #12698: URL: https://github.com/apache/doris/pull/12698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #13013: [enhancement](explain) compress descriptor table explain string
morrySnow opened a new pull request, #13013:
URL: https://github.com/apache/doris/pull/13013
# Proposed changes
1. compress slot descriptor explain string to one row
2. remove unmaterialized tuple descriptor and slot descriptor
current descriptor table explain string is like this:
```
TupleDescriptor{id=2, tbl=lineitem}
SlotDescriptor{id=1, col=l_extendedprice, type=DECIMAL(15,2),
nullable=false, slotIdx=3}
SlotDescriptor{id=2, col=l_discount, type=DECIMAL(15,2), nullable=false,
slotIdx=4}
SlotDescriptor{id=5, col=l_partkey, type=INT, nullable=false, slotIdx=0}
SlotDescriptor{id=7, col=l_suppkey, type=INT, nullable=false, slotIdx=1}
SlotDescriptor{id=8, col=l_orderkey, type=BIGINT, nullable=false,
slotIdx=2}
```
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
4. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
5. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
6. Does it need to update dependencies:
- [ ] Yes
- [ ] No
7. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (64988cb3d4 -> eba71cf5da)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 64988cb3d4 [Enhancement](optimize) optimize for insert_indices_from
(#12807)
add eba71cf5da [enhancement](test) add tpch_sf10 cases to p2 (#12698)
No new revisions were added by this update.
Summary of changes:
.../ddl/customer.sql | 0
.../ddl/customer_delete.sql| 0
.../tpch_sf10_unique_p2/ddl/customer_load.sql | 6 ++
.../ddl/customer_load_sequence.sql | 7 ++
.../ddl/customer_part_delete.sql | 1 +
.../ddl/customer_sequence.sql | 0
.../ddl/lineitem.sql | 0
.../ddl/lineitem_delete.sql| 0
.../tpch_sf10_unique_p2/ddl/lineitem_load.sql | 6 ++
.../ddl/lineitem_load_sequence.sql | 7 ++
.../ddl/lineitem_part_delete.sql | 1 +
.../ddl/lineitem_sequence.sql | 0
.../ddl/nation.sql | 0
.../ddl/nation_delete.sql | 0
.../suites/tpch_sf10_unique_p2/ddl/nation_load.sql | 6 ++
.../ddl/nation_load_sequence.sql | 7 ++
.../ddl/nation_part_delete.sql | 0
.../ddl/nation_sequence.sql| 0
.../ddl/orders.sql | 0
.../ddl/orders_delete.sql | 0
.../suites/tpch_sf10_unique_p2/ddl/orders_load.sql | 6 ++
.../ddl/orders_load_sequence.sql | 7 ++
.../tpch_sf10_unique_p2/ddl/orders_part_delete.sql | 1 +
.../ddl/orders_sequence.sql| 0
.../ddl/part.sql | 0
.../ddl/part_delete.sql| 0
.../suites/tpch_sf10_unique_p2/ddl/part_load.sql | 6 ++
.../tpch_sf10_unique_p2/ddl/part_load_sequence.sql | 7 ++
.../tpch_sf10_unique_p2/ddl/part_part_delete.sql | 1 +
.../ddl/part_sequence.sql | 0
.../ddl/partsupp.sql | 0
.../ddl/partsupp_delete.sql| 0
.../tpch_sf10_unique_p2/ddl/partsupp_load.sql | 6 ++
.../ddl/partsupp_load_sequence.sql | 7 ++
.../ddl/partsupp_part_delete.sql | 1 +
.../ddl/partsupp_sequence.sql | 0
.../ddl/region.sql | 0
.../ddl/region_delete.sql | 0
.../suites/tpch_sf10_unique_p2/ddl/region_load.sql | 6 ++
.../ddl/region_load_sequence.sql | 7 ++
.../ddl/region_part_delete.sql | 0
.../ddl/region_sequence.sql| 0
.../ddl/supplier.sql | 0
.../ddl/supplier_delete.sql| 0
.../tpch_sf10_unique_p2/ddl/supplier_load.sql | 6 ++
.../ddl/supplier_load_sequence.sql | 7 ++
.../ddl/supplier_part_delete.sql | 1 +
.../ddl/supplier_sequence.sql | 0
.../tpch_sf10_unique_p2/load_four_step.groovy | 120 +
.../tpch_sf10_unique_p2/load_one_step.groovy | 74 +
.../tpch_sf10_unique_p2/load_three_step.groovy | 99 +
.../tpch_sf10_unique_p2/load_two_step.groovy | 76 +
52 files changed, 479 insertions(+)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/customer.sql (100%)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/customer_delete.sql (100%)
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/customer_load.sql
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/customer_load_sequence.sql
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/customer_part_delete.sql
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/customer_sequence.sql (100%)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/lineitem.sql (100%)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/lineitem_delete.sql (100%)
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/lineitem_load.sql
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/lineitem_load_sequence.sql
create mode 100644
regression-test/suites/tpch_sf10_unique_p2/ddl/lineitem_part_delete.sql
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/lineitem_sequence.sql (100%)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/nation.sql (100%)
copy regression-test/suites/{tpch_sf100_unique_p2 =>
tpch_sf10_unique_p2}/ddl/nation_delete.sql (100%)
create mode 100644
regression
[GitHub] [doris] dataroaring merged pull request #12695: [enhancement](test) add tpcds_sf1000 to p2
dataroaring merged PR #12695: URL: https://github.com/apache/doris/pull/12695 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba commented on pull request #66: [Fix] fix deserialization error for QueryPlan exception
JNSimba commented on PR #66: URL: https://github.com/apache/doris-flink-connector/pull/66#issuecomment-1259223100 > However, the response of queryPlan has been judged in `RestService.getQueryPlan`, and an `exception` is also thrown when an error occurs, so is there no need to record the exception in queryPlan? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #12866: [enhancement](compaction) introduce segment compaction (#12609)
freemandealer commented on code in PR #12866:
URL: https://github.com/apache/doris/pull/12866#discussion_r980990083
##
be/src/olap/rowset/beta_rowset_writer.cpp:
##
@@ -304,60 +397,76 @@ void
BetaRowsetWriter::find_longest_consecutive_small_segment(
is_terminated_by_big = true;
break;
} else {
-rename_compacted_segment_plain(_segcompacted_point);
-++_segcompacted_point;
+
RETURN_NOT_OK(_rename_compacted_segment_plain(_segcompacted_point++));
}
} else {
let_big_terminate = true; // break if find a big after small
segments->push_back(seg);
-++_segcompacted_point;
}
}
size_t s = segments->size();
if (!is_terminated_by_big && s <=
(config::segcompaction_threshold_segment_num / 2)) {
// start with big segments and end with small, better to do it in next
// round to compact more at once
-_segcompacted_point -= s;
segments->clear();
-LOG(INFO) << "candidate segments num too small:" << s;
-return;
+return Status::OK();
}
if (s == 1) { // poor bachelor, let it go
LOG(INFO) << "only one candidate segment";
-rename_compacted_segment_plain(_segcompacted_point - 1);
+RETURN_NOT_OK(_rename_compacted_segment_plain(_segcompacted_point++));
segments->clear();
-return;
+return Status::OK();
}
std::stringstream ss;
for (auto& segment : (*segments.get())) {
ss << "[id:" << segment->id() << " num_rows:" << segment->num_rows()
<< "]";
}
LOG(INFO) << "candidate segments num:" << s << " list of candidates:" <<
ss.str();
+return Status::OK();
}
-SegCompactionCandidatesSharedPtr
BetaRowsetWriter::get_segcompaction_candidates(bool is_last) {
-SegCompactionCandidatesSharedPtr segments =
std::make_shared();
+Status
BetaRowsetWriter::_get_segcompaction_candidates(SegCompactionCandidatesSharedPtr&
segments,
+ bool is_last) {
Review Comment:
indent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12728: [fix](array-type) fix the be core dump when use string to insert array
github-actions[bot] commented on PR #12728: URL: https://github.com/apache/doris/pull/12728#issuecomment-1259226546 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12728: [fix](array-type) fix the be core dump when use string to insert array
github-actions[bot] commented on PR #12728: URL: https://github.com/apache/doris/pull/12728#issuecomment-1259226618 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13014: [Bug](load) fix core dump on big block load
BiteThet opened a new pull request, #13014: URL: https://github.com/apache/doris/pull/13014 # Proposed changes Issue Number: close #13011 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #8864: [feature-wip](statistics) step6: statistics is available
morrySnow merged PR #8864: URL: https://github.com/apache/doris/pull/8864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (c21ecdd867 -> ba5705a589)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c21ecdd867 [enhancement](test) add tpcds_sf1000 to p2 (#12695) add ba5705a589 [feature-wip](statistics) step6: statistics is available (#8864) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/catalog/Env.java| 2 + .../doris/nereids/stats/StatsCalculator.java | 20 +- .../java/org/apache/doris/qe/SessionVariable.java | 13 ++ .../org/apache/doris/statistics/ColumnStats.java | 201 +--- .../apache/doris/statistics/PartitionStats.java| 76 +--- .../org/apache/doris/statistics/Statistics.java| 161 +++- .../doris/statistics/StatisticsJobManager.java | 47 ++--- .../org/apache/doris/statistics/TableStats.java| 206 + .../org/apache/doris/statistics/TabletStats.java | 1 - .../nereids/jobs/cascades/DeriveStatsJobTest.java | 22 ++- .../doris/nereids/stats/StatsCalculatorTest.java | 42 +++-- .../apache/doris/statistics/ColumnStatsTest.java | 159 .../doris/statistics/PartitionStatsTest.java | 136 ++ .../statistics/StatisticsJobSchedulerTest.java | 173 + .../apache/doris/statistics/StatisticsJobTest.java | 120 .../doris/statistics/StatisticsManagerTest.java| 120 .../apache/doris/statistics/StatisticsTest.java| 46 ++--- .../apache/doris/statistics/TableStatsTest.java| 182 ++ .../statistics/util/InternalQueryBufferTest.java | 4 +- .../statistics/util/InternalQueryResultTest.java | 12 +- 20 files changed, 1331 insertions(+), 412 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/ColumnStatsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/PartitionStatsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsJobSchedulerTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsJobTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsManagerTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/statistics/TableStatsTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13005: [chore](docs)Add local documentation compilation checks to reduce subsequent operations caused by pipeline build failures
github-actions[bot] commented on PR #13005:
URL: https://github.com/apache/doris/pull/13005#issuecomment-1259253916
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3133657438";) output.
shellcheck errors
```
'shellcheck ' found no issues.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- docs/build_check.sh.orig
+++ docs/build_check.sh
@@ -33,27 +33,27 @@
rm -rf website/community
cp -R docs/en/community website/
rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs/*
-mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
+mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs/current
cp -R docs/zh-CN/docs/*
website/i18n/zh-CN/docusaurus-plugin-content-docs/current/
cp docs/dev.json
website/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
-rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
+rm -rf website/i18n/zh-CN/docusaurus-plugin-content-docs-community/*
mkdir website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current
cp -R docs/zh-CN/community/*
website/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/
cp -R docs/sidebarsCommunity.json website/
cp -R docs/sidebars.json website/
cp -R docs/images website/static/
-sed '2,3d' website/versions.json > website/versions.json1
+sed '2,3d' website/versions.json >website/versions.json1
rm -rf website/versions.json
mv website/versions.json1 website/versions.json
-sed '123,128d' website/docusaurus.config.js > website/docusaurus.config.js1
+sed '123,128d' website/docusaurus.config.js >website/docusaurus.config.js1
rm -rf website/docusaurus.config.js
mv website/docusaurus.config.js1 website/docusaurus.config.js
cd website
npm install -g npm@8.19.1
npm install -g yarn
yarn cache clean
-yarn && yarn build
-rm -rf website
+yarn && yarn build
+rm -rf website
echo "***"
echo "Docs build check pass"
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree commented on issue #12729: Doris使用limit导致查询时间延长
LemonLiTree commented on issue #12729: URL: https://github.com/apache/doris/issues/12729#issuecomment-1259257732 表数据量多大呢？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] weizhengte commented on a diff in pull request #12987: [feature](nereids) Enhancement for statistics
weizhengte commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r981021547
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStat.java:
##
@@ -54,7 +51,7 @@
* For example:
* "@ndv = 10" means that the number distinct values is 10 in the whole table.
*/
-public class ColumnStats {
+public class ColumnStat {
Review Comment:
Would it be better to use `stats`? the meaning of `stat` is not clear, and
the abbreviation of statistics is usually `stats`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #13015: [enhancement](Nereids) let BinaryArithmetic's dataType and nullable match with BE
morrySnow opened a new pull request, #13015: URL: https://github.com/apache/doris/pull/13015 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #12970: [Improvement](outfile) Support output null in parquet writer
HappenLee commented on code in PR #12970:
URL: https://github.com/apache/doris/pull/12970#discussion_r981024664
##
be/src/vec/runtime/vparquet_writer.cpp:
##
@@ -173,22 +178,27 @@ Status VParquetWriterWrapper::write(const Block& block) {
try {
for (size_t i = 0; i < block.columns(); i++) {
auto& raw_column = block.get_by_position(i).column;
-const auto col = raw_column->is_nullable()
- ? reinterpret_cast(
-
block.get_by_position(i).column.get())
- ->get_nested_column_ptr()
- .get()
- : block.get_by_position(i).column.get();
-auto null_map =
-raw_column->is_nullable() && reinterpret_cast(
-
block.get_by_position(i).column.get())
-
->get_null_map_column_ptr()
- ->has_null()
-? reinterpret_cast(
- block.get_by_position(i).column.get())
- ->get_null_map_column_ptr()
-: nullptr;
+auto nullable = raw_column->is_nullable();
+const auto col = nullable ? reinterpret_cast(
+
block.get_by_position(i).column.get())
+->get_nested_column_ptr()
+.get()
+ : block.get_by_position(i).column.get();
+auto null_map = nullable && reinterpret_cast(
+
block.get_by_position(i).column.get())
+->has_null()
+? reinterpret_cast(
+
block.get_by_position(i).column.get())
+ ->get_null_map_column_ptr()
+: nullptr;
auto& type = block.get_by_position(i).type;
+
+int16_t def_level[sz];
+for (size_t def_index = 0; def_index < sz; def_index++) {
Review Comment:
better use `std::fill` ?
##
be/src/vec/runtime/vparquet_writer.cpp:
##
@@ -215,58 +225,64 @@ Status VParquetWriterWrapper::write(const Block& block) {
parquet::RowGroupWriter* rgWriter = get_rg_writer();
parquet::Int32Writer* col_writer =
static_cast(rgWriter->column(i));
-int32_t default_int32 = 0;
if (null_map != nullptr) {
+auto& null_data = assert_cast(*null_map).get_data();
if (const auto* nested_column =
check_and_get_column>(col)) {
for (size_t row_id = 0; row_id < sz; row_id++) {
-col_writer->WriteBatch(
-1, nullptr, nullptr,
-(*null_map)[row_id] != 0
-? &default_int32
-: reinterpret_cast(
-
nested_column->get_data_at(row_id).data));
+if (null_data[row_id] != 0) {
+def_level[row_id] = 0;
+}
}
+col_writer->WriteBatch(sz, def_level, nullptr,
+
nested_column->get_data().data());
} else if (const auto* int16_column =
check_and_get_column>(col)) {
for (size_t row_id = 0; row_id < sz; row_id++) {
Review Comment:
seems not need to do the work of set `def_level`
##
be/src/vec/runtime/vparquet_writer.cpp:
##
@@ -215,58 +225,64 @@ Status VParquetWriterWrapper::write(const Block& block) {
parquet::RowGroupWriter* rgWriter = get_rg_writer();
parquet::Int32Writer* col_writer =
static_cast(rgWriter->column(i));
-int32_t default_int32 = 0;
if (null_map != nullptr) {
+auto& null_data = assert_cast(*null_map).get_data();
if (const auto* nested_column =
check_and_get_column>(col)) {
for (size_t row_id = 0; row_id < sz; row_id++) {
-col_writer->WriteBatch(
-1, nullptr, nullptr,
-
[GitHub] [doris] adonis0147 opened a new pull request, #13016: [chore](third-party) Fix compilation errors reported by clang-15
adonis0147 opened a new pull request, #13016: URL: https://github.com/apache/doris/pull/13016 # Proposed changes Add some compile flags to eliminate compilation errors reported by clang-15. ## Problem summary Some packages can't be compiled by clang-15. 1. libunixodbc 2. thrift 3. flatbuffers ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #12970: [Improvement](outfile) Support output null in parquet writer
HappenLee commented on code in PR #12970:
URL: https://github.com/apache/doris/pull/12970#discussion_r981044071
##
be/src/vec/runtime/vparquet_writer.cpp:
##
@@ -215,58 +225,64 @@ Status VParquetWriterWrapper::write(const Block& block) {
parquet::RowGroupWriter* rgWriter = get_rg_writer();
parquet::Int32Writer* col_writer =
static_cast(rgWriter->column(i));
-int32_t default_int32 = 0;
if (null_map != nullptr) {
+auto& null_data = assert_cast(*null_map).get_data();
if (const auto* nested_column =
check_and_get_column>(col)) {
for (size_t row_id = 0; row_id < sz; row_id++) {
Review Comment:
`def_level[row_id] = (bool) null_data[row_id] == 0` may SIMD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] spaces-X opened a new pull request, #13017: [Fix](meta): fix colocate group meta error
spaces-X opened a new pull request, #13017: URL: https://github.com/apache/doris/pull/13017 # Proposed changes Issue Number: close #12788 ## Problem summary From version 1.1.1, the types of distributed columns in colocate table are with length and precision info, which are defaut value may be -1 or 0 in the elder version. There are two solutions to solve this problem 1. temporary incomplete solution: skip the length and precision meta-check of varchar or decimal in colocate tables. 2. long term solution: incorrect the meta data to fix the problem. Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] JNSimba opened a new pull request, #67: add miss pr4
JNSimba opened a new pull request, #67: URL: https://github.com/apache/doris-flink-connector/pull/67 ## Problem Summary: #4 This pr seems to be lost when it is merged https://github.com/apache/doris-flink-connector/pull/4/files ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #13014: [Bug](load) fix core dump on big block load
HappenLee commented on code in PR #13014:
URL: https://github.com/apache/doris/pull/13014#discussion_r981057768
##
be/src/util/block_compression.cpp:
##
@@ -73,6 +73,13 @@ class Lz4BlockCompression : public BlockCompressionCodec {
}
Status compress(const Slice& input, faststring* output) override {
+if (input.size > INT_MAX) {
Review Comment:
unlikely ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12667: [Feature](serialize) move block_data_version to fe heart beat
morningman merged PR #12667: URL: https://github.com/apache/doris/pull/12667 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (ba5705a589 -> 9607f60845)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ba5705a589 [feature-wip](statistics) step6: statistics is available (#8864) add 9607f60845 [Feature](serialize) move block_data_version to fe heart beat (#12667) No new revisions were added by this update. Summary of changes: be/src/agent/heartbeat_server.cpp | 15 +++ be/src/agent/heartbeat_server.h | 21 +++-- be/src/common/config.h | 3 --- be/src/vec/core/block.cpp | 17 ++--- be/src/vec/core/block.h | 5 - be/src/vec/data_types/data_type.h | 7 --- be/src/vec/data_types/data_type_array.cpp | 13 +++-- be/src/vec/data_types/data_type_array.h | 6 +++--- be/src/vec/data_types/data_type_bitmap.cpp | 7 --- be/src/vec/data_types/data_type_bitmap.h| 6 +++--- be/src/vec/data_types/data_type_decimal.cpp | 6 +++--- be/src/vec/data_types/data_type_decimal.h | 6 +++--- .../data_types/data_type_fixed_length_object.cpp| 4 ++-- .../vec/data_types/data_type_fixed_length_object.h | 6 +++--- be/src/vec/data_types/data_type_hll.cpp | 6 +++--- be/src/vec/data_types/data_type_hll.h | 8 be/src/vec/data_types/data_type_nothing.cpp | 5 +++-- be/src/vec/data_types/data_type_nothing.h | 6 +++--- be/src/vec/data_types/data_type_nullable.cpp| 12 ++-- be/src/vec/data_types/data_type_nullable.h | 6 +++--- be/src/vec/data_types/data_type_number_base.cpp | 7 --- be/src/vec/data_types/data_type_number_base.h | 6 +++--- be/src/vec/data_types/data_type_string.cpp | 13 +++-- be/src/vec/data_types/data_type_string.h| 6 +++--- be/test/vec/core/column_complex_test.cpp| 10 ++ docs/en/docs/admin-manual/config/fe-config.md | 21 + docs/zh-CN/docs/admin-manual/config/fe-config.md| 21 + .../src/main/java/org/apache/doris/catalog/Env.java | 11 +++ .../main/java/org/apache/doris/common/Config.java | 18 ++ .../java/org/apache/doris/system/HeartbeatMgr.java | 1 + gensrc/proto/data.proto | 2 +- gensrc/thrift/HeartbeatService.thrift | 1 + 32 files changed, 191 insertions(+), 91 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ccoffline commented on pull request #11693: [Enhancement] add more metrics #11214
ccoffline commented on PR #11693: URL: https://github.com/apache/doris/pull/11693#issuecomment-1259305379 > Please update the document: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on pull request #13009: [fix](Broker Load)fix bug when load data using broker with md5sum()/sm3sum()
chenlinzhong commented on PR #13009: URL: https://github.com/apache/doris/pull/13009#issuecomment-1259344060 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #12785: [Feature](Nereids) Set pre-aggregation status for OLAP table scan.
morrySnow merged PR #12785: URL: https://github.com/apache/doris/pull/12785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (9607f60845 -> 57570f2090)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 9607f60845 [Feature](serialize) move block_data_version to fe heart
beat (#12667)
add 57570f2090 [feature](Nereids) Set pre-aggregation status for OLAP
table scan (#12785)
No new revisions were added by this update.
Summary of changes:
.../glue/translator/PhysicalPlanTranslator.java| 9 +-
.../jobs/batch/NereidsRewriteJobExecutor.java | 6 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../LogicalOlapScanToPhysicalOlapScan.java | 1 +
...tRollup.java => SelectRollupWithAggregate.java} | 347 ++
.../rules/mv/SelectRollupWithoutAggregate.java | 60
.../logical/PushdownFilterThroughProject.java | 2 +-
.../trees/expressions/functions/agg/Avg.java | 6 +
.../trees/expressions/functions/agg/Count.java | 6 +
.../trees/expressions/functions/agg/Max.java | 6 +
.../trees/expressions/functions/agg/Min.java | 6 +
.../trees/expressions/functions/agg/Sum.java | 6 +
.../expressions/visitor/ExpressionVisitor.java | 37 +-
.../doris/nereids/trees/plans/PreAggStatus.java| 75
.../doris/nereids/trees/plans/algebra/Project.java | 29 +-
.../trees/plans/logical/LogicalOlapScan.java | 30 +-
.../trees/plans/physical/PhysicalOlapScan.java | 22 +-
.../apache/doris/nereids/util/ExpressionUtils.java | 7 +
.../org/apache/doris/planner/OlapScanNode.java | 8 +-
.../translator/PhysicalPlanTranslatorTest.java | 3 +-
.../doris/nereids/rules/mv/SelectRollupTest.java | 396 ++---
.../doris/nereids/trees/plans/PlanEqualsTest.java | 6 +-
.../nereids/trees/plans/PlanToStringTest.java | 3 +-
.../org/apache/doris/nereids/util/PlanChecker.java | 8 +
.../suites/nereids_syntax_p0/rollup.groovy | 9 +
25 files changed, 760 insertions(+), 329 deletions(-)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/mv/{SelectRollup.java
=> SelectRollupWithAggregate.java} (51%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/mv/SelectRollupWithoutAggregate.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PreAggStatus.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #12886: [feature-wip](unique-key-merge-on-write) unique key with merge on write table support schema change
zhannngchen commented on code in PR #12886:
URL: https://github.com/apache/doris/pull/12886#discussion_r981101947
##
be/src/olap/schema_change.cpp:
##
@@ -1938,20 +1958,86 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
_tablet_ids_in_converting.insert(new_tablet->tablet_id());
}
res = _convert_historical_rowsets(sc_params);
-{
-std::lock_guard wrlock(_mutex);
-_tablet_ids_in_converting.erase(new_tablet->tablet_id());
+if (new_tablet->keys_type() != UNIQUE_KEYS ||
+!new_tablet->enable_unique_key_merge_on_write() || !res) {
+{
+std::lock_guard wrlock(_mutex);
+_tablet_ids_in_converting.erase(new_tablet->tablet_id());
+}
}
if (!res) {
break;
}
-// set state to ready
-std::lock_guard
new_wlock(new_tablet->get_header_lock());
-res = new_tablet->set_tablet_state(TabletState::TABLET_RUNNING);
-if (!res) {
-break;
+
+// For unique with merge-on-write table, should process delete bitmap
here.
+// 1. During double write, the newly imported rowsets does not
calculate
+// delete bitmap and publish successfully.
+// 2. After conversion, calculate delete bitmap for the rowsets
imported
+// during double write. During this period, new data can still be
imported
+// witout calculating delete bitmap and publish successfully.
+// 3. Block the new publish, calculate the delete bitmap of the
+// incremental rowsets.
+// 4. Switch the tablet status to TABLET_RUNNING. The newly imported
+// data will calculate delete bitmap.
+if (new_tablet->keys_type() == UNIQUE_KEYS &&
+new_tablet->enable_unique_key_merge_on_write()) {
+std::lock_guard cumulative_compaction_lock(
+new_tablet->get_cumulative_compaction_lock());
+int64_t max_version = new_tablet->max_version().second;
+std::vector rowsets;
+if (end_version < max_version) {
+LOG(INFO)
+<< "alter table for unique with merge-on-write,
calculate delete bitmap of "
+<< "double write rowsets for version: " << end_version
+ 1 << "-"
+<< max_version;
+RETURN_IF_ERROR(new_tablet->capture_consistent_rowsets(
+{end_version + 1, max_version}, &rowsets));
+}
Review Comment:
Add a comment here, it's step2
##
be/src/olap/schema_change.cpp:
##
@@ -1938,20 +1958,86 @@ Status
SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2&
_tablet_ids_in_converting.insert(new_tablet->tablet_id());
}
res = _convert_historical_rowsets(sc_params);
-{
-std::lock_guard wrlock(_mutex);
-_tablet_ids_in_converting.erase(new_tablet->tablet_id());
+if (new_tablet->keys_type() != UNIQUE_KEYS ||
+!new_tablet->enable_unique_key_merge_on_write() || !res) {
+{
+std::lock_guard wrlock(_mutex);
+_tablet_ids_in_converting.erase(new_tablet->tablet_id());
+}
}
if (!res) {
break;
}
-// set state to ready
-std::lock_guard
new_wlock(new_tablet->get_header_lock());
-res = new_tablet->set_tablet_state(TabletState::TABLET_RUNNING);
-if (!res) {
-break;
+
+// For unique with merge-on-write table, should process delete bitmap
here.
+// 1. During double write, the newly imported rowsets does not
calculate
+// delete bitmap and publish successfully.
+// 2. After conversion, calculate delete bitmap for the rowsets
imported
+// during double write. During this period, new data can still be
imported
+// witout calculating delete bitmap and publish successfully.
+// 3. Block the new publish, calculate the delete bitmap of the
+// incremental rowsets.
+// 4. Switch the tablet status to TABLET_RUNNING. The newly imported
+// data will calculate delete bitmap.
+if (new_tablet->keys_type() == UNIQUE_KEYS &&
+new_tablet->enable_unique_key_merge_on_write()) {
+std::lock_guard cumulative_compaction_lock(
+new_tablet->get_cumulative_compaction_lock());
+int64_t max_version = new_tablet->max_version().second;
+std::vector rowsets;
+if (end_version < max_version) {
+LOG(INFO)
+<< "alter table for unique with merge-on-write,
calculate delete bitmap of "
+<< "double write rowsets for version: " << end_version
+ 1 << "-"
+<< max_version;
+
[GitHub] [doris] github-actions[bot] commented on pull request #12981: [improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to control memory co…
github-actions[bot] commented on PR #12981:
URL: https://github.com/apache/doris/pull/12981#issuecomment-1259388558
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3134322503";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In bin/start_be.sh line 210:
mem_limit_str=$(grep ^mem_limit ${DORIS_HOME}/conf/be.conf)
^---^ SC2086 (info): Double
quote to prevent globbing and word splitting.
Did you mean:
mem_limit_str=$(grep ^mem_limit "${DORIS_HOME}"/conf/be.conf)
In bin/start_be.sh line 218:
case $unit in
^---^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
case ${unit} in
In bin/start_be.sh line 231:
if [[ "$mem_limit_mb" -gt "$total_mem_mb" ]]; then
^---^ SC2250 (style): Prefer putting braces around
variable references even when not strictly required.
^---^ SC2250 (style): Prefer putting
braces around variable references even when not strictly required.
Did you mean:
if [[ "${mem_limit_mb}" -gt "${total_mem_mb}" ]]; then
In bin/start_be.sh line 232:
echo "mem_limit is larger than whole memory of the server.
$mem_limit_mb > $total_mem_mb."
^---^ SC2250 (style): Prefer putting braces around variable references
even when not strictly required.
^---^ SC2250 (style): Prefer putting braces around variable
references even when not strictly required.
Did you mean:
echo "mem_limit is larger than whole memory of the server.
${mem_limit_mb} > ${total_mem_mb}."
In bin/start_be.sh line 235:
export TCMALLOC_HEAP_LIMIT_MB=$mem_limit_mb
^---^ SC2250 (style): Prefer
putting braces around variable references even when not strictly required.
Did you mean:
export TCMALLOC_HEAP_LIMIT_MB=${mem_limit_mb}
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing
...
https://www.shellcheck.net/wiki/SC2250 -- Prefer putting braces around
vari...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #13018: [improvement](test) cache data from s3 to cacheDataPath
dataroaring opened a new pull request, #13018: URL: https://github.com/apache/doris/pull/13018 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12987: [feature](nereids) Enhancement for statistics
Kikyou1997 commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r981150921
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStat.java:
##
@@ -54,7 +51,7 @@
* For example:
* "@ndv = 10" means that the number distinct values is 10 in the whole table.
*/
-public class ColumnStats {
+public class ColumnStat {
Review Comment:
Yes, I studied some other systems and `stats` (postgres, noisipage, tidb)is
do more common used than `stat`(Spark), change it to `stat` is due to the name
style of nereids that for the name of variable with collections type should
have a `s` as suffix. Andi t looks kind of wired to name a varibale as
`columnStatss`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12987: [feature](nereids) Enhancement for statistics
Kikyou1997 commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r981150921
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStat.java:
##
@@ -54,7 +51,7 @@
* For example:
* "@ndv = 10" means that the number distinct values is 10 in the whole table.
*/
-public class ColumnStats {
+public class ColumnStat {
Review Comment:
Yes, I studied some other systems and `stats` (postgres, noisipage, tidb) is
do more common used than `stat`(Spark), change it to `stat` is due to the
naming style of nereids is that for the name of variable with collections type
should have a `s` as suffix. Andi t looks kind of wired to name a varibale as
`columnStatss`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Kikyou1997 commented on a diff in pull request #12987: [feature](nereids) Enhancement for statistics
Kikyou1997 commented on code in PR #12987:
URL: https://github.com/apache/doris/pull/12987#discussion_r981150921
##
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStat.java:
##
@@ -54,7 +51,7 @@
* For example:
* "@ndv = 10" means that the number distinct values is 10 in the whole table.
*/
-public class ColumnStats {
+public class ColumnStat {
Review Comment:
Yes, I studied some other systems and `stats` (postgres, noisipage, tidb) is
do more common used than `stat`(Spark), change it to `stat` is due to the
naming style of nereids is that for the name of variable with collections type
should have a `s` as suffix. And it looks kind of wired to name a varibale as
`columnStatss`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ruleeeer commented on issue #12836: [Bug] slack link is no longer active
rulr commented on issue #12836: URL: https://github.com/apache/doris/issues/12836#issuecomment-1259424074 > you can try it：[join.slack.com/t/apachedoriscommunity/shared_invite/zt-1co4h0c90-Kfu5MUgaTi4n~lAwNXfbuQ](https://join.slack.com/t/apachedoriscommunity/shared_invite/zt-1co4h0c90-Kfu5MUgaTi4n~lAwNXfbuQ) And if you want（reply me or follow the wechat official account on the official website）, you can join the WeChat group of the open source user community。 Give me a wechat link is fine,thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] cambyzju commented on a diff in pull request #12897: [draft](cooldown) support local cache gc by disk usage
cambyzju commented on code in PR #12897:
URL: https://github.com/apache/doris/pull/12897#discussion_r978243282
##
be/src/common/config.h:
##
@@ -844,6 +844,7 @@ CONF_mString(file_cache_type, "");
CONF_Validator(file_cache_type, [](const std::string config) -> bool {
return config == "sub_file_cache" || config == "whole_file_cache" ||
config == "";
});
+CONF_mInt64(file_cache_max_storage_size_gb, "0"); // no limit
Review Comment:
TODO: multi disk
##
be/src/io/cache/file_cache_manager.h:
##
@@ -47,10 +50,20 @@ class FileCacheManager {
bool exist(const std::string& cache_path);
+void add_used_file_size(int64_t s) {
+_total_used_file_size.fetch_add(s);
+}
+// TODO: need call sub while clean
+void sub_used_file_size(int64_t s) {
+_total_used_file_size.fetch_sub(s);
+}
+
private:
std::shared_mutex _cache_map_lock;
// cache_path -> FileCache
std::map _file_cache_map;
+
+std::atomic _total_used_file_size;
Review Comment:
TODO: _cache_file_cleaner_tasks_producer_callback 计算
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #12985: [feature](Nereids): Eliminate outer join
morrySnow merged PR #12985: URL: https://github.com/apache/doris/pull/12985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (57570f2090 -> 9a38a9677a)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 57570f2090 [feature](Nereids) Set pre-aggregation status for OLAP table scan (#12785) add 9a38a9677a [feature](Nereids) Eliminate outer join (#12985) No new revisions were added by this update. Summary of changes: .../org/apache/doris/nereids/rules/RuleSet.java| 2 + .../org/apache/doris/nereids/rules/RuleType.java | 1 + .../rules/rewrite/logical/EliminateOuter.java | 91 ++ .../apache/doris/nereids/trees/plans/JoinType.java | 4 + .../rules/rewrite/logical/EliminateOuterTest.java | 88 + 5 files changed, 186 insertions(+) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOuter.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateOuterTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus closed pull request #12774: [regression-test](join)Add test case of left join sql
smallhibiscus closed pull request #12774: [regression-test](join)Add test case of left join sql URL: https://github.com/apache/doris/pull/12774 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg merged pull request #13009: [fix](Broker Load)fix bug when load data using broker with md5sum()/sm3sum()
yangzhg merged PR #13009: URL: https://github.com/apache/doris/pull/13009 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: fix_md5sum_and_sm3sum (#13009)
This is an automated email from the ASF dual-hosted git repository.
yangzhg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 16f5204cab fix_md5sum_and_sm3sum (#13009)
16f5204cab is described below
commit 16f5204cab20abc6b6a166b6ce0de6aff65f06a8
Author: yongjinhou <109586248+yongjin...@users.noreply.github.com>
AuthorDate: Tue Sep 27 21:41:14 2022 +0800
fix_md5sum_and_sm3sum (#13009)
---
be/src/exprs/encryption_functions.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/exprs/encryption_functions.cpp
b/be/src/exprs/encryption_functions.cpp
index 52eb572698..ba49c0c782 100644
--- a/be/src/exprs/encryption_functions.cpp
+++ b/be/src/exprs/encryption_functions.cpp
@@ -181,7 +181,7 @@ StringVal EncryptionFunctions::md5sum(FunctionContext* ctx,
int num_args, const
for (int i = 0; i < num_args; ++i) {
const StringVal& arg = args[i];
if (arg.is_null) {
-continue;
+return StringVal::null();
}
digest.update(arg.ptr, arg.len);
}
@@ -204,7 +204,7 @@ StringVal EncryptionFunctions::sm3sum(FunctionContext* ctx,
int num_args, const
for (int i = 0; i < num_args; ++i) {
const StringVal& arg = args[i];
if (arg.is_null) {
-continue;
+return StringVal::null();
}
digest.update(arg.ptr, arg.len);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg closed issue #12989: [Bug] The value of md5sum(NULL)/sm3sum(NULL) is not NULL when using broker to import data(Non-vectorized import)
yangzhg closed issue #12989: [Bug] The value of md5sum(NULL)/sm3sum(NULL) is not NULL when using broker to import data(Non-vectorized import) URL: https://github.com/apache/doris/issues/12989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #13019: [Improvement](outfile) Support ORC format in outfile
Gabriel39 opened a new pull request, #13019: URL: https://github.com/apache/doris/pull/13019 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](new-scan) support more load situation (#12953)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d80b7b9689 [feature-wip](new-scan) support more load situation (#12953)
d80b7b9689 is described below
commit d80b7b9689da6da4042ed25333a02386c9238f9b
Author: Mingyu Chen
AuthorDate: Tue Sep 27 21:48:32 2022 +0800
[feature-wip](new-scan) support more load situation (#12953)
---
be/src/exec/arrow/arrow_reader.cpp | 15 +-
be/src/exec/arrow/arrow_reader.h | 2 +
be/src/exec/arrow/orc_reader.cpp | 25 +-
be/src/exec/arrow/orc_reader.h | 5 +
be/src/vec/CMakeLists.txt | 1 -
be/src/vec/columns/column_const.h | 2 +-
be/src/vec/exec/file_hdfs_scanner.cpp | 98 ---
be/src/vec/exec/file_hdfs_scanner.h| 57
be/src/vec/exec/file_scan_node.cpp | 10 +-
be/src/vec/exec/format/generic_reader.h| 4 +
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 57 ++--
be/src/vec/exec/format/parquet/vparquet_reader.h | 16 +-
be/src/vec/exec/scan/vfile_scanner.cpp | 300 +
be/src/vec/exec/scan/vfile_scanner.h | 61 +++--
be/src/vec/exec/scan/vscan_node.h | 1 +
be/src/vec/exec/scan/vscanner.h| 4 -
be/src/vec/exprs/vexpr_context.cpp | 2 +-
be/src/vec/exprs/vliteral.cpp | 3 +-
be/src/vec/utils/arrow_column_to_doris_column.cpp | 56
be/src/vec/utils/arrow_column_to_doris_column.h| 4 +-
be/test/vec/exec/parquet/parquet_reader_test.cpp | 120 +
.../planner/external/ExternalFileScanNode.java | 74 -
.../doris/planner/external/FileScanProviderIf.java | 3 +
.../doris/planner/external/HiveScanProvider.java | 6 +
.../doris/planner/external/LoadScanProvider.java | 18 +-
gensrc/thrift/PlanNodes.thrift | 11 +-
26 files changed, 535 insertions(+), 420 deletions(-)
diff --git a/be/src/exec/arrow/arrow_reader.cpp
b/be/src/exec/arrow/arrow_reader.cpp
index d26efd32aa..72d4960a43 100644
--- a/be/src/exec/arrow/arrow_reader.cpp
+++ b/be/src/exec/arrow/arrow_reader.cpp
@@ -79,10 +79,7 @@ Status ArrowReaderWrap::column_indices() {
if (iter != _map_column.end()) {
_include_column_ids.emplace_back(iter->second);
} else {
-std::stringstream str_error;
-str_error << "Invalid Column Name:" << slot_desc->col_name();
-LOG(WARNING) << str_error.str();
-return Status::InvalidArgument(str_error.str());
+_missing_cols.push_back(slot_desc->col_name());
}
}
return Status::OK();
@@ -103,10 +100,13 @@ int ArrowReaderWrap::get_column_index(std::string
column_name) {
Status ArrowReaderWrap::get_next_block(vectorized::Block* block, bool* eof) {
size_t rows = 0;
+bool tmp_eof = false;
do {
if (_batch == nullptr || _arrow_batch_cur_idx >= _batch->num_rows()) {
-RETURN_IF_ERROR(next_batch(&_batch, eof));
-if (*eof) {
+RETURN_IF_ERROR(next_batch(&_batch, &tmp_eof));
+// We need to make sure the eof is set to true iff block is empty.
+if (tmp_eof) {
+*eof = (rows == 0);
return Status::OK();
}
}
@@ -128,7 +128,7 @@ Status ArrowReaderWrap::get_next_block(vectorized::Block*
block, bool* eof) {
}
rows += num_elements;
_arrow_batch_cur_idx += num_elements;
-} while (!(*eof) && rows < _state->batch_size());
+} while (!tmp_eof && rows < _state->batch_size());
return Status::OK();
}
@@ -138,7 +138,6 @@ Status
ArrowReaderWrap::next_batch(std::shared_ptr* batch, b
if (_batch_eof) {
_include_column_ids.clear();
*eof = true;
-_batch_eof = false;
return Status::OK();
}
_queue_reader_cond.wait_for(lock, std::chrono::seconds(1));
diff --git a/be/src/exec/arrow/arrow_reader.h b/be/src/exec/arrow/arrow_reader.h
index 35703e4bbd..2d83a1be01 100644
--- a/be/src/exec/arrow/arrow_reader.h
+++ b/be/src/exec/arrow/arrow_reader.h
@@ -137,6 +137,8 @@ protected:
// The following fields are only valid when using "get_block()" interface.
std::shared_ptr _batch;
size_t _arrow_batch_cur_idx = 0;
+// Save col names which need to be read but does not exist in file
+std::vector _missing_cols;
};
} // namespace doris
diff --git a/be/src/exec/arrow/orc_reader.cpp b/be/src/exec/arrow/orc_reader.cpp
index 65a67909ba..8f46a9bf21 100644
--- a/be/src/exec/arrow/orc_reader.cpp
+++ b/be/src/exec/arrow/orc_reader.cpp
@@ -26,6 +26,7 @@
#include "runtime/runtime_state.h
[GitHub] [doris] dataroaring merged pull request #12953: [feature-wip](new-scan) support more load situation
dataroaring merged PR #12953: URL: https://github.com/apache/doris/pull/12953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #12975: [enhancement](load) avoid duplicate reduce on same TabletsChannel
dataroaring merged PR #12975: URL: https://github.com/apache/doris/pull/12975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](load) avoid duplicate reduce on same TabletsChannel #12975
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d8ec53c83f [enhancement](load) avoid duplicate reduce on same
TabletsChannel #12975
d8ec53c83f is described below
commit d8ec53c83f4726abc15808b6c58e6ef3a1080d28
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Tue Sep 27 22:03:08 2022 +0800
[enhancement](load) avoid duplicate reduce on same TabletsChannel #12975
In the policy changed by PR #12716, when reaching the hard limit, there
might be multiple threads can pick same LoadChannel and call reduce_mem_usage
on same TabletsChannel. Although there's a lock and condition variable can
prevent multiple threads to reduce mem usage concurrently, but they still can
do same reduce-work on that channel multiple times one by one, even it's just
reduced.
---
be/src/runtime/load_channel_mgr.cpp | 9 -
be/src/runtime/load_channel_mgr.h | 9 ++---
be/src/runtime/tablets_channel.cpp | 22 --
be/src/runtime/tablets_channel.h| 2 +-
4 files changed, 27 insertions(+), 15 deletions(-)
diff --git a/be/src/runtime/load_channel_mgr.cpp
b/be/src/runtime/load_channel_mgr.cpp
index 2405a28ae7..e6f908f69c 100644
--- a/be/src/runtime/load_channel_mgr.cpp
+++ b/be/src/runtime/load_channel_mgr.cpp
@@ -112,15 +112,6 @@ Status LoadChannelMgr::open(const
PTabletWriterOpenRequest& params) {
return Status::OK();
}
-void LoadChannelMgr::_try_to_wait_flushing() {
-std::unique_lock l(_lock);
-while (_should_wait_flush) {
-LOG(INFO) << "Reached the load channel manager mem limit " <<
_mem_tracker->limit()
- << ", waiting for flush";
-_wait_flush_cond.wait(l);
-}
-}
-
static void dummy_deleter(const CacheKey& key, void* value) {}
void LoadChannelMgr::_finish_load_channel(const UniqueId load_id) {
diff --git a/be/src/runtime/load_channel_mgr.h
b/be/src/runtime/load_channel_mgr.h
index fb4a5d3592..686322b076 100644
--- a/be/src/runtime/load_channel_mgr.h
+++ b/be/src/runtime/load_channel_mgr.h
@@ -68,7 +68,6 @@ private:
// If yes, it will pick a load channel to try to reduce memory consumption.
template
Status _handle_mem_exceed_limit(TabletWriterAddResult* response);
-void _try_to_wait_flushing();
Status _start_bg_worker();
@@ -152,7 +151,6 @@ Status LoadChannelMgr::add_batch(const
TabletWriterAddRequest& request,
template
Status LoadChannelMgr::_handle_mem_exceed_limit(TabletWriterAddResult*
response) {
-_try_to_wait_flushing();
// Check the soft limit.
DCHECK(_load_soft_mem_limit > 0);
DCHECK(_process_soft_mem_limit > 0);
@@ -163,7 +161,12 @@ Status
LoadChannelMgr::_handle_mem_exceed_limit(TabletWriterAddResult* response)
// Pick load channel to reduce memory.
std::shared_ptr channel;
{
-std::lock_guard l(_lock);
+std::unique_lock l(_lock);
+while (_should_wait_flush) {
+LOG(INFO) << "Reached the load hard limit " <<
_mem_tracker->limit()
+ << ", waiting for flush";
+_wait_flush_cond.wait(l);
+}
// Some other thread is flushing data, and not reached hard limit now,
// we don't need to handle mem limit in current thread.
if (_reduce_memory_channel != nullptr &&
!_mem_tracker->limit_exceeded() &&
diff --git a/be/src/runtime/tablets_channel.cpp
b/be/src/runtime/tablets_channel.cpp
index 7fb8322b12..fa72d23ad0 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -196,7 +196,12 @@ void TabletsChannel::_close_wait(DeltaWriter* writer,
template
Status TabletsChannel::reduce_mem_usage(TabletWriterAddResult* response) {
-_try_to_wait_flushing();
+if (_try_to_wait_flushing()) {
+// `_try_to_wait_flushing()` returns true means other thread already
+// reduced the mem usage, and current thread do not need to reduce
again.
+return Status::OK();
+}
+
std::vector writers_to_flush;
{
std::lock_guard l(_lock);
@@ -335,11 +340,24 @@ Status TabletsChannel::_open_all_writers(const
PTabletWriterOpenRequest& request
return Status::OK();
}
-void TabletsChannel::_try_to_wait_flushing() {
+bool TabletsChannel::_try_to_wait_flushing() {
+bool duplicate_work = false;
std::unique_lock l(_lock);
+// NOTE: we call `reduce_mem_usage()` because we think it's necessary
+// to reduce it's memory and should not write more data into this
+// tablets channel. If there's already some other thead doing the
+// reduce-memory work, the only choice for current thread is to wait
+// here.
+// If current thread do not wait, it has two options:
+// 1. continue to write data to current channel.
+// 2. pick anot
[GitHub] [doris] dutyu closed issue #13010: [Bug] show proc "/cluster_health/tablet_health" returns wrong result
dutyu closed issue #13010: [Bug] show proc "/cluster_health/tablet_health" returns wrong result URL: https://github.com/apache/doris/issues/13010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive commented on issue #12760: [Bug] 使用insert-values语句导致be节点挂了
dataalive commented on issue #12760: URL: https://github.com/apache/doris/issues/12760#issuecomment-1259561690 so @deaftstill u could try on 1.1-lts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #13020: [Opt](Vec) Support const column check nullable and remove nullable
HappenLee opened a new pull request, #13020: URL: https://github.com/apache/doris/pull/13020 # Proposed changes 1. Support const column check nullable 2. Support const column remove nullable ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13016: [chore](third-party) Fix compilation errors reported by clang-15
github-actions[bot] commented on PR #13016: URL: https://github.com/apache/doris/pull/13016#issuecomment-1259652118 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13016: [chore](third-party) Fix compilation errors reported by clang-15
github-actions[bot] commented on PR #13016: URL: https://github.com/apache/doris/pull/13016#issuecomment-1259652063 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dinggege1024 commented on a diff in pull request #2763: [Alter]Add wait time while doing alter options
dinggege1024 commented on code in PR #2763:
URL: https://github.com/apache/doris/pull/2763#discussion_r981380592
##
fe/src/main/java/org/apache/doris/alter/Alter.java:
##
@@ -273,6 +276,18 @@ public void processAlterTable(AlterTableStmt stmt) throws
UserException {
throw new DdlException("table with empty parition cannot do
schema change. [" + tableName + "]");
}
+// if table state is unhealthy, change table repair priority, and
wait until repair finish or exceed timeout
+if (olapTable.getState() != OlapTableState.NORMAL) {
Review Comment:
> First, `olapTable.getState() != OlapTableState.NORMAL` does not mean the
table is unhealthy. Second, it's not good to sleep here, it will block the
client connection for a very long time. Third, I don't think its good idea to
add priority repair here. We should solve more why the table is often
unhealthy, rather than increase the complexity here.
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 merged pull request #13016: [chore](third-party) Fix compilation errors reported by clang-15
adonis0147 merged PR #13016: URL: https://github.com/apache/doris/pull/13016 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (d8ec53c83f -> 2dafbda9de)
This is an automated email from the ASF dual-hosted git repository. adonisling pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d8ec53c83f [enhancement](load) avoid duplicate reduce on same TabletsChannel #12975 add 2dafbda9de [chore](third-party) Fix compilation errors reported by clang-15 (#13016) No new revisions were added by this update. Summary of changes: thirdparty/build-thirdparty.sh | 22 +- 1 file changed, 13 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12981: [improvement](memory) set TCMALLOC_HEAP_LIMIT_MB to control memory co…
github-actions[bot] commented on PR #12981:
URL: https://github.com/apache/doris/pull/12981#issuecomment-1259775333
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/3135573969";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In bin/start_be.sh line 210:
mem_limit_str=$(grep ^mem_limit ${DORIS_HOME}/conf/be.conf)
^---^ SC2086 (info): Double
quote to prevent globbing and word splitting.
Did you mean:
mem_limit_str=$(grep ^mem_limit "${DORIS_HOME}"/conf/be.conf)
For more information:
https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12921: [Improvement](sort) Reuse memory in sort node
github-actions[bot] commented on PR #12921: URL: https://github.com/apache/doris/pull/12921#issuecomment-1259928485 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12921: [Improvement](sort) Reuse memory in sort node
github-actions[bot] commented on PR #12921: URL: https://github.com/apache/doris/pull/12921#issuecomment-1259928527 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13007: [Bug](function) core dump on substr
github-actions[bot] commented on PR #13007: URL: https://github.com/apache/doris/pull/13007#issuecomment-1259929813 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13007: [Bug](function) core dump on substr
github-actions[bot] commented on PR #13007: URL: https://github.com/apache/doris/pull/13007#issuecomment-1259929877 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13007: [Bug](function) core dump on substr
yiguolei merged PR #13007: URL: https://github.com/apache/doris/pull/13007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] hf200012 merged pull request #67: [Fix] add miss pr4
hf200012 merged PR #67: URL: https://github.com/apache/doris-flink-connector/pull/67 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] hf200012 merged pull request #64: [Feature] Support Light Schema change for flink1.14
hf200012 merged PR #64: URL: https://github.com/apache/doris-flink-connector/pull/64 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch schemachange-1.14 updated: [Feature] Support Light Schema change for flink1.14 (#64)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch schemachange-1.14
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/schemachange-1.14 by this push:
new b407905 [Feature] Support Light Schema change for flink1.14 (#64)
b407905 is described below
commit b40790537b0fe89d2c4c0aac75ff7476c0345ee2
Author: wudi <676366...@qq.com>
AuthorDate: Wed Sep 28 08:59:55 2022 +0800
[Feature] Support Light Schema change for flink1.14 (#64)
Support Light Schema change for flink1.14
---
flink-doris-connector/pom.xml | 45 ++--
.../doris/flink/cfg/DorisExecutionOptions.java | 2 +-
.../org/apache/doris/flink/rest/RestService.java | 2 +-
.../apache/doris/flink/sink/HttpGetWithEntity.java | 36 +++
.../apache/doris/flink/sink/HttpPutBuilder.java| 12 +-
.../doris/flink/sink/committer/DorisCommitter.java | 1 +
.../doris/flink/sink/writer/DorisStreamLoad.java | 3 +
.../doris/flink/sink/writer/DorisWriter.java | 7 +-
.../sink/writer/JsonDebeziumSchemaSerializer.java | 264 +
.../flink/table/DorisDynamicTableFactory.java | 2 +-
.../apache/doris/flink/CDCSchemaChangeExample.java | 87 +++
.../org/apache/doris/flink/DorisSinkExample.java | 12 +-
.../apache/doris/flink/DorisSinkSQLExample.java| 7 +-
.../apache/doris/flink/DorisSourceSinkExample.java | 40 ++--
.../writer/TestJsonDebeziumSchemaSerializer.java | 151
.../doris/flink/source/DorisSourceExampleTest.java | 1 -
.../flink/source/reader/DorisSourceReaderTest.java | 2 +
.../doris/flink/utils/DateToStringConverter.java | 147
18 files changed, 759 insertions(+), 62 deletions(-)
diff --git a/flink-doris-connector/pom.xml b/flink-doris-connector/pom.xml
index 6206b4a..bde8d37 100644
--- a/flink-doris-connector/pom.xml
+++ b/flink-doris-connector/pom.xml
@@ -62,9 +62,9 @@ under the License.
-${env.scala.version}
-${env.flink.version}
-${env.flink.minor.version}
+2.12
+1.14.4
+1.14
0.13.0
5.0.0
3.8.1
@@ -251,33 +251,20 @@ under the License.
2.13.3
-org.apache.logging.log4j
-log4j-web
-${log4j2.version}
-
-
-
-org.apache.logging.log4j
-log4j-api
-${log4j2.version}
-
-
-
-org.apache.logging.log4j
-log4j-core
-${log4j2.version}
-
-
-
-org.apache.logging.log4j
-log4j-slf4j-impl
-${log4j2.version}
+org.slf4j
+slf4j-api
+1.7.25
-
org.slf4j
slf4j-log4j12
-1.7.9
+1.7.25
+test
+
+
+log4j
+log4j
+1.2.17
@@ -310,6 +297,12 @@ under the License.
test
+
+com.ververica
+flink-connector-mysql-cdc
+2.2.1
+test
+
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
index 2daf5e1..102a7ee 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
@@ -73,7 +73,7 @@ public class DorisExecutionOptions implements Serializable {
public static DorisExecutionOptions defaults() {
Properties properties = new Properties();
properties.setProperty("format", "json");
-properties.setProperty("strip_outer_array", "true");
+properties.setProperty("read_json_by_line", "true");
return new Builder().setStreamLoadProp(properties).build();
}
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
index 734bfdb..5732dc8 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
@@ -442,7 +442,7 @@ public class RestService implements Serializable {
public static boolean isUniqueKeyType(DorisOptions options,
DorisReadOptions readOptions, Logger logger)
throws DorisRuntimeException {
try {
-return "UNIQUE_KEYS_TYPE".equals(getSchema(options, readOptions,
logger).getKeysType());
+return UNIQUE_KEYS_TYPE.equals(getSchema(options, readOptions
[doris-flink-connector] branch branch-for-flink-before-1.13 updated: update (#67)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch branch-for-flink-before-1.13
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/branch-for-flink-before-1.13
by this push:
new 154bcf4 update (#67)
154bcf4 is described below
commit 154bcf4161ba29a90d9d0603c241dda85ff85901
Author: wudi <676366...@qq.com>
AuthorDate: Wed Sep 28 08:59:12 2022 +0800
update (#67)
Co-authored-by: wudi <>
---
.../java/org/apache/doris/flink/cfg/GenericDorisSinkFunction.java| 2 +-
.../java/org/apache/doris/flink/table/DorisDynamicTableSink.java | 5 +++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/GenericDorisSinkFunction.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/GenericDorisSinkFunction.java
index 0ea11d0..a055195 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/GenericDorisSinkFunction.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/GenericDorisSinkFunction.java
@@ -59,7 +59,7 @@ public class GenericDorisSinkFunction extends
RichSinkFunction
@Override
public void snapshotState(FunctionSnapshotContext context) throws
Exception {
-
+outputFormat.flush();
}
@Override
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicTableSink.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicTableSink.java
index cccdb45..813669b 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicTableSink.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicTableSink.java
@@ -19,10 +19,11 @@ package org.apache.doris.flink.table;
import org.apache.doris.flink.cfg.DorisExecutionOptions;
import org.apache.doris.flink.cfg.DorisOptions;
import org.apache.doris.flink.cfg.DorisReadOptions;
+import org.apache.doris.flink.cfg.GenericDorisSinkFunction;
import org.apache.flink.table.api.TableSchema;
import org.apache.flink.table.connector.ChangelogMode;
import org.apache.flink.table.connector.sink.DynamicTableSink;
-import org.apache.flink.table.connector.sink.OutputFormatProvider;
+import org.apache.flink.table.connector.sink.SinkFunctionProvider;
import org.apache.flink.types.RowKind;
/**
@@ -65,7 +66,7 @@ public class DorisDynamicTableSink implements
DynamicTableSink {
.setExecutionOptions(executionOptions)
.setFieldDataTypes(tableSchema.getFieldDataTypes())
.setFieldNames(tableSchema.getFieldNames());
-return OutputFormatProvider.of(builder.build());
+return SinkFunctionProvider.of(new
GenericDorisSinkFunction(builder.build()));
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #13008: [fix](join)report 'natural join is not supported' instead of getting wrong result
yiguolei merged PR #13008: URL: https://github.com/apache/doris/pull/13008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 merged pull request #12921: [Improvement](sort) Reuse memory in sort node
Gabriel39 merged PR #12921: URL: https://github.com/apache/doris/pull/12921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improvement](sort) Reuse memory in sort node (#12921)
This is an automated email from the ASF dual-hosted git repository.
gabriellee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1ba9e4b568 [Improvement](sort) Reuse memory in sort node (#12921)
1ba9e4b568 is described below
commit 1ba9e4b5684c59db248b9bf17ab0fe54680bf9ed
Author: Gabriel
AuthorDate: Wed Sep 28 09:44:35 2022 +0800
[Improvement](sort) Reuse memory in sort node (#12921)
---
.../aggregate_functions/aggregate_function_sort.h | 2 +-
be/src/vec/common/sort/heap_sorter.cpp | 2 +-
be/src/vec/common/sort/heap_sorter.h | 2 +-
be/src/vec/common/sort/sorter.cpp | 50 ++
be/src/vec/common/sort/sorter.h| 24 ---
be/src/vec/common/sort/topn_sorter.cpp | 28 +---
be/src/vec/common/sort/topn_sorter.h | 5 ++-
be/src/vec/core/sort_block.cpp | 24 ++-
be/src/vec/core/sort_block.h | 3 +-
be/src/vec/exec/vsort_node.cpp | 15 ---
be/src/vec/exec/vsort_node.h | 2 +
be/src/vec/utils/util.hpp | 12 ++
12 files changed, 96 insertions(+), 73 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_sort.h
b/be/src/vec/aggregate_functions/aggregate_function_sort.h
index 2db72a4c5c..201bd5df62 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_sort.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_sort.h
@@ -87,7 +87,7 @@ struct AggregateFunctionSortData {
}
}
-void sort() { sort_block(block, sort_desc, block.rows()); }
+void sort() { sort_block(block, block, sort_desc, block.rows()); }
};
template
diff --git a/be/src/vec/common/sort/heap_sorter.cpp
b/be/src/vec/common/sort/heap_sorter.cpp
index 795bd66941..6520b005a4 100644
--- a/be/src/vec/common/sort/heap_sorter.cpp
+++ b/be/src/vec/common/sort/heap_sorter.cpp
@@ -29,7 +29,7 @@ HeapSorter::HeapSorter(VSortExecExprs& vsort_exec_exprs, int
limit, int64_t offs
_topn_filter_rows(0),
_init_sort_descs(false) {}
-Status HeapSorter::append_block(Block* block, bool* mem_reuse) {
+Status HeapSorter::append_block(Block* block) {
DCHECK(block->rows() > 0);
{
SCOPED_TIMER(_materialize_timer);
diff --git a/be/src/vec/common/sort/heap_sorter.h
b/be/src/vec/common/sort/heap_sorter.h
index f725d585c2..6f644a9d92 100644
--- a/be/src/vec/common/sort/heap_sorter.h
+++ b/be/src/vec/common/sort/heap_sorter.h
@@ -63,7 +63,7 @@ public:
_materialize_timer = ADD_TIMER(runtime_profile, "MaterializeTime");
}
-Status append_block(Block* block, bool* mem_reuse) override;
+Status append_block(Block* block) override;
Status prepare_for_read() override;
diff --git a/be/src/vec/common/sort/sorter.cpp
b/be/src/vec/common/sort/sorter.cpp
index 9b5641075d..5de7499a2e 100644
--- a/be/src/vec/common/sort/sorter.cpp
+++ b/be/src/vec/common/sort/sorter.cpp
@@ -72,25 +72,27 @@ Status
MergeSorterState::merge_sort_read(doris::RuntimeState* state,
return Status::OK();
}
-Status Sorter::partial_sort(Block& block) {
-if (_vsort_exec_exprs.need_materialize_tuple()) {
+Status Sorter::partial_sort(Block& src_block, Block& dest_block) {
+size_t num_cols = src_block.columns();
+if (_materialize_sort_exprs) {
auto output_tuple_expr_ctxs =
_vsort_exec_exprs.sort_tuple_slot_expr_ctxs();
std::vector valid_column_ids(output_tuple_expr_ctxs.size());
for (int i = 0; i < output_tuple_expr_ctxs.size(); ++i) {
-RETURN_IF_ERROR(output_tuple_expr_ctxs[i]->execute(&block,
&valid_column_ids[i]));
+RETURN_IF_ERROR(output_tuple_expr_ctxs[i]->execute(&src_block,
&valid_column_ids[i]));
}
Block new_block;
for (auto column_id : valid_column_ids) {
-new_block.insert(block.get_by_position(column_id));
+new_block.insert(src_block.get_by_position(column_id));
}
-block.swap(new_block);
+dest_block.swap(new_block);
}
_sort_description.resize(_vsort_exec_exprs.lhs_ordering_expr_ctxs().size());
+Block* result_block = _materialize_sort_exprs ? &dest_block : &src_block;
for (int i = 0; i < _sort_description.size(); i++) {
const auto& ordering_expr =
_vsort_exec_exprs.lhs_ordering_expr_ctxs()[i];
-RETURN_IF_ERROR(ordering_expr->execute(&block,
&_sort_description[i].column_number));
+RETURN_IF_ERROR(ordering_expr->execute(result_block,
&_sort_description[i].column_number));
_sort_description[i].direction = _is_asc_order[i] ? 1 : -1;
_sort_description[i].nulls_direction =
@@ -99,7 +101,12 @@ Status Sorter::partial_sort(Block& block) {
{
SCOPED_TIMER(_partial_sort_timer);
-
[GitHub] [doris] morningman opened a new pull request, #13021: [improvement](scan) remove concurrency limit if scan has predicate
morningman opened a new pull request, #13021: URL: https://github.com/apache/doris/pull/13021 # Proposed changes Issue Number: close #xxx ## Problem summary If a scan node has predicate, we can not limit the concurrency of scanner. Because we don't know how much data need to be scan. If we limit the concurrency, this will cause query to be very slow. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #13022: [branch-1.1-lts](memory) Disable page cache and lower chunk allocator to avoid OOM
xinyiZzz opened a new pull request, #13022: URL: https://github.com/apache/doris/pull/13022 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12849: [feature](Nereids) use one stage aggregation if available
github-actions[bot] commented on PR #12849: URL: https://github.com/apache/doris/pull/12849#issuecomment-1260309075 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12849: [feature](Nereids) use one stage aggregation if available
github-actions[bot] commented on PR #12849: URL: https://github.com/apache/doris/pull/12849#issuecomment-1260309054 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #12886: [feature-wip](unique-key-merge-on-write) unique key with merge on write table support schema change
zhannngchen commented on PR #12886: URL: https://github.com/apache/doris/pull/12886#issuecomment-1260310394 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts updated: [Bug](function) core dump on substr #13007
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new a4b83dc2f1 [Bug](function) core dump on substr #13007
a4b83dc2f1 is described below
commit a4b83dc2f1d146f2d2b2b04e6b83237774dc22fe
Author: Pxl
AuthorDate: Wed Sep 28 08:54:49 2022 +0800
[Bug](function) core dump on substr #13007
---
be/src/vec/functions/function_string.h | 4
1 file changed, 4 insertions(+)
diff --git a/be/src/vec/functions/function_string.h
b/be/src/vec/functions/function_string.h
index 0200adf495..f0566b2d80 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -188,6 +188,10 @@ private:
}
int fixed_pos = start[i];
+if (fixed_pos < -(int)index.size()) {
+StringOP::push_empty_string(i, res_chars, res_offsets);
+continue;
+}
if (fixed_pos < 0) {
fixed_pos = index.size() + fixed_pos + 1;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #12849: [feature](Nereids) use one stage aggregation if available
morrySnow merged PR #12849: URL: https://github.com/apache/doris/pull/12849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (1ba9e4b568 -> eef9367705)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1ba9e4b568 [Improvement](sort) Reuse memory in sort node (#12921) add eef9367705 [feature](Nereids) use one stage aggregation if available (#12849) No new revisions were added by this update. Summary of changes: .../glue/translator/PhysicalPlanTranslator.java| 9 +-- .../jobs/batch/NereidsRewriteJobExecutor.java | 2 - .../nereids/jobs/cascades/CostAndEnforcerJob.java | 4 ++ .../properties/ChildrenPropertiesRegulator.java| 13 + .../nereids/properties/RequestPropertyDeriver.java | 2 +- .../org/apache/doris/nereids/rules/RuleSet.java| 2 + .../rules/rewrite/AggregateDisassemble.java| 68 -- .../rewrite/logical/MergeConsecutiveProjects.java | 12 +++- .../rules/rewrite/logical/NormalizeAggregate.java | 38 .../doris/nereids/trees/expressions/Add.java | 6 ++ .../expressions/functions/ExecutableFunctions.java | 20 +++ .../trees/plans/logical/LogicalAggregate.java | 9 +-- .../doris/nereids/parser/HavingClauseTest.java | 5 +- .../properties/RequestPropertyDeriverTest.java | 2 +- .../rules/expression/rewrite/FoldConstantTest.java | 12 ++-- .../logical/MergeConsecutiveProjectsTest.java | 10 ++-- .../rewrite/logical/NormalizeAggregateTest.java| 19 -- .../nereids/trees/plans/PlanToStringTest.java | 2 +- .../nereids/util/AnalyzeWhereSubqueryTest.java | 3 +- 19 files changed, 155 insertions(+), 83 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
