[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
hello-stephen commented on PR #13246: URL: https://github.com/apache/doris/pull/13246#issuecomment-1279685483 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221015153413_clickbench_pr_29739.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13338: [Opt](fun) simd the substring function and use stack buf to speed up
github-actions[bot] commented on PR #13338: URL: https://github.com/apache/doris/pull/13338#issuecomment-1279693732 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13338: [Opt](fun) simd the substring function and use stack buf to speed up
github-actions[bot] commented on PR #13338: URL: https://github.com/apache/doris/pull/13338#issuecomment-1279693738 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13339: [Opt](vec) opt runtime filter for TPCH Q22
github-actions[bot] commented on PR #13339: URL: https://github.com/apache/doris/pull/13339#issuecomment-1279694115 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13339: [Opt](vec) opt runtime filter for TPCH Q22
github-actions[bot] commented on PR #13339: URL: https://github.com/apache/doris/pull/13339#issuecomment-1279694121 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13285: [improvement](memory) disable page cache and chunk allocator, optimize memory allocate size
github-actions[bot] commented on PR #13285: URL: https://github.com/apache/doris/pull/13285#issuecomment-1279704279 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13285: [improvement](memory) disable page cache and chunk allocator, optimize memory allocate size
github-actions[bot] commented on PR #13285: URL: https://github.com/apache/doris/pull/13285#issuecomment-1279704272 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz merged pull request #13285: [improvement](memory) disable page cache and chunk allocator, optimize memory allocate size
xinyiZzz merged PR #13285: URL: https://github.com/apache/doris/pull/13285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](memory) disable page cache and chunk allocator, optimize memory allocate size (#13285)
This is an automated email from the ASF dual-hosted git repository.
zouxinyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a5f3880649 [improvement](memory) disable page cache and chunk
allocator, optimize memory allocate size (#13285)
a5f3880649 is described below
commit a5f3880649b094b58061f25c15dccdb50a4a2973
Author: yiguolei <676222...@qq.com>
AuthorDate: Sat Oct 15 17:27:17 2022 +0800
[improvement](memory) disable page cache and chunk allocator, optimize
memory allocate size (#13285)
disable page cache by default
disable chunk allocator by default
not use chunk allocator for vectorized allocator by default
add a new config memory_linear_growth_threshold = 128Mb, not allocate
memory by RoundUpToPowerOf2 if the allocated size is larger than this
threshold. This config is added to MemPool, ChunkAllocator, PodArray, Arena.
---
be/src/common/config.h| 14 +---
be/src/runtime/exec_env_init.cpp | 15 ++--
be/src/runtime/mem_pool.cpp | 15 ++--
be/src/runtime/mem_pool.h | 6 ++--
be/src/runtime/memory/chunk_allocator.cpp | 57 ---
be/src/util/bit_util.h| 10 --
be/src/vec/common/arena.h | 11 --
be/src/vec/common/pod_array.h | 22
8 files changed, 79 insertions(+), 71 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 021f5cd680..0b8ac4af9f 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -239,7 +239,7 @@ CONF_Int32(storage_page_cache_shard_size, "16");
// all storage page cache will be divided into data_page_cache and
index_page_cache
CONF_Int32(index_page_cache_percentage, "10");
// whether to disable page cache feature in storage
-CONF_Bool(disable_storage_page_cache, "false");
+CONF_Bool(disable_storage_page_cache, "true");
CONF_Bool(enable_storage_vectorization, "true");
@@ -439,14 +439,20 @@ CONF_Bool(disable_mem_pools, "false");
// increase this variable can improve performance,
// but will acquire more free memory which can not be used by other modules.
CONF_mString(chunk_reserved_bytes_limit, "10%");
-// 1024, The minimum chunk allocator size (in bytes)
-CONF_Int32(min_chunk_reserved_bytes, "1024");
+
+// Whether using chunk allocator to cache memory chunk
+CONF_Bool(disable_chunk_allocator, "true");
// Disable Chunk Allocator in Vectorized Allocator, this will reduce memory
cache.
// For high concurrent queries, using Chunk Allocator with vectorized
Allocator can reduce the impact
// of gperftools tcmalloc central lock.
// Jemalloc or google tcmalloc have core cache, Chunk Allocator may no longer
be needed after replacing
// gperftools tcmalloc.
-CONF_mBool(disable_chunk_allocator_in_vec, "false");
+CONF_mBool(disable_chunk_allocator_in_vec, "true");
+
+// Both MemPool and vectorized engine's podarray allocator, vectorized
engine's arena will try to allocate memory as power of two.
+// But if the memory is very large then power of two is also very large. This
config means if the allocated memory's size is larger
+// than this limit then all allocators will not use RoundUpToPowerOfTwo to
allocate memory.
+CONF_mInt64(memory_linear_growth_threshold, "134217728"); // 128Mb
// The probing algorithm of partitioned hash table.
// Enable quadratic probing hash table
diff --git a/be/src/runtime/exec_env_init.cpp b/be/src/runtime/exec_env_init.cpp
index a55f547778..327a0307e7 100644
--- a/be/src/runtime/exec_env_init.cpp
+++ b/be/src/runtime/exec_env_init.cpp
@@ -199,9 +199,7 @@ Status ExecEnv::_init_mem_tracker() {
if (global_memory_limit_bytes > MemInfo::physical_mem()) {
LOG(WARNING) << "Memory limit "
<< PrettyPrinter::print(global_memory_limit_bytes,
TUnit::BYTES)
- << " exceeds physical memory of "
- << PrettyPrinter::print(MemInfo::physical_mem(),
TUnit::BYTES)
- << ". Using physical memory instead";
+ << " exceeds physical memory, using physical memory
instead";
global_memory_limit_bytes = MemInfo::physical_mem();
}
_process_mem_tracker =
@@ -307,13 +305,6 @@ Status ExecEnv::_init_mem_tracker() {
RETURN_IF_ERROR(_disk_io_mgr->init(global_memory_limit_bytes));
RETURN_IF_ERROR(_tmp_file_mgr->init());
-// 5. init chunk allocator
-if (!BitUtil::IsPowerOf2(config::min_chunk_reserved_bytes)) {
-ss << "Config min_chunk_reserved_bytes must be a power-of-two: "
- << config::min_chunk_reserved_bytes;
-return Status::InternalError(ss.str());
-}
-
int64_t chunk_reserved_bytes_limit =
ParseUtil::parse_mem_spec(config::chunk_reserved_bytes_limit,
global_memory_limit_bytes,
Me
[GitHub] [doris] morningman commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
morningman commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996279665
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -2743,17 +2745,17 @@ drop_stmt ::=
// Recover statement
recover_stmt ::=
-KW_RECOVER KW_DATABASE ident:dbName
+KW_RECOVER KW_DATABASE ident:dbName opt_id:dbId opt_table_alias:alias
Review Comment:
if `id` is specified, no need to specify `name`
so the syntax can be:
```
recover database ident_or_id opt_alias
```
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -3619,6 +3621,10 @@ show_param ::=
{:
RESULT = new ShowAnalyzeStmt(tbl, parser.where, orderByClause,
limitClause);
:}
+| KW_DROPPED opt_wild_where
Review Comment:
How about:
```
show catalog recycle bin where xxx?
```
`dropped` is too colloquial
##
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##
@@ -76,46 +81,56 @@ public synchronized boolean recycleDatabase(Database db,
Set tableNames)
// db should be empty. all tables are recycled before
Preconditions.checkState(db.getTables().isEmpty());
-// erase db with same name
-eraseDatabaseWithSameName(db.getFullName());
+// erase db with same id
+eraseDatabaseWithSameId(db.getId());
// recycle db
-RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames);
+RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames, tableIds);
idToDatabase.put(db.getId(), databaseInfo);
-idToRecycleTime.put(db.getId(), System.currentTimeMillis());
+if (!isReplay || replayRecycleTime == 0) {
+recycleTime = System.currentTimeMillis();
Review Comment:
Looks like the `recycleTime` is not written into edit log,
so it is not persisted?
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDroppedStmt.java:
##
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+
+import java.util.function.Predicate;
+
+public class ShowDroppedStmt extends ShowStmt {
+public static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+
.add("Type").add("Name").add("DbId").add("TableId").add("PartitionId").add("DropTime")
Review Comment:
We can know what id it is using `type` column.
So no need to use 2 columns to distinguish `db/tbl/partition` ids,
##
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##
@@ -76,46 +81,56 @@ public synchronized boolean recycleDatabase(Database db,
Set tableNames)
// db should be empty. all tables are recycled before
Preconditions.checkState(db.getTables().isEmpty());
-// erase db with same name
-eraseDatabaseWithSameName(db.getFullName());
+// erase db with same id
+eraseDatabaseWithSameId(db.getId());
Review Comment:
Why does the same id appear?
##
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##
@@ -76,46 +81,56 @@ public synchronized boolean recycleDatabase(Database db,
Set tableNames)
// db should be empty. all tables are recycled before
Preconditions.checkState(db.getTables().isEmpty());
-// erase db with same name
-eraseDatabaseWithSameName(db.getFullName());
+// erase db with same id
+eraseDatabaseWithSameId(db.getId());
// recycle db
-RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames);
+RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames, tableIds);
Review Comment:
if `tableIds` is saved, no n
[GitHub] [doris] morningman commented on pull request #13240: [fix](ddl) check view name by table name regex when create
morningman commented on PR #13240: URL: https://github.com/apache/doris/pull/13240#issuecomment-1279713987 Please add some unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #13387: [Enhancement] replace assertTrue(X.equals(X)) with assertEquals
jackwener commented on PR #13387: URL: https://github.com/apache/doris/pull/13387#issuecomment-1279720063 I suggest the you can use `intellij IDEA` open project. And `IDEA` will suggest it and auto fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13362: [Enhancement](function) refactor of date function
github-actions[bot] commented on PR #13362: URL: https://github.com/apache/doris/pull/13362#issuecomment-1279728971 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13362: [Enhancement](function) refactor of date function
github-actions[bot] commented on PR #13362: URL: https://github.com/apache/doris/pull/13362#issuecomment-1279728974 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataalive closed pull request #13385: [typo](docs)fix wrong url link
dataalive closed pull request #13385: [typo](docs)fix wrong url link URL: https://github.com/apache/doris/pull/13385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SpringHuBiWei opened a new issue, #13390: Broker Load get a mistake "ErrorMsg: type:LOAD_RUN_FAIL; msg:errCode = 2, detailMessage = coordinator could not finished before job timeout"
SpringHuBiWei opened a new issue, #13390:
URL: https://github.com/apache/doris/issues/13390
### Search before asking
- [X] I had searched in the
[issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and
found no similar issues.
### Version
1.1.2
### What's Wrong?
mysql> show load order by createtime desc limit 1\G;
*** 1. row ***
JobId: 13010
Label: label_20220404412
State: CANCELLED
Progress: ETL:N/A; LOAD:N/A
Type: BROKER
EtlInfo: NULL
TaskInfo: cluster:N/A; timeout(s):1200; max_filter_ratio:0.1
ErrorMsg: type:LOAD_RUN_FAIL; msg:errCode = 2, detailMessage =
coordinator could not finished before job timeout
CreateTime: 2022-10-15 20:48:51
EtlStartTime: 2022-10-15 21:08:51
EtlFinishTime: 2022-10-15 21:08:51
LoadStartTime: 2022-10-15 21:08:51
LoadFinishTime: 2022-10-15 21:08:52
URL: NULL
JobDetails: {"Unfinished
backends":{"a27627eda75d4775-b3c6b93a8c787505":[11008]},"ScannedRows":0,"TaskNumber":1,"LoadBytes":0,"All
backends":{"a27627eda75d4775-b3c6b93a8c787505":[11008]},"FileNumber":1,"FileSize":11238369}
TransactionId: 1004
ErrorTablets: {}
1 row in set (0.00 sec)
### What You Expected?
I don't know how to deal with it
### How to Reproduce?
_No response_
### Anything Else?
I have look up for many logs(ps: be fe broker ) but don't have some usage
message
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SpringHuBiWei commented on issue #13390: Broker Load get a mistake "ErrorMsg: type:LOAD_RUN_FAIL; msg:errCode = 2, detailMessage = coordinator could not finished before job timeout"
SpringHuBiWei commented on issue #13390: URL: https://github.com/apache/doris/issues/13390#issuecomment-1279747290 The file size on the hdfs has been identified, indicating that the hdfs can be connected, but it is always displayed in the loading process, and this error will be reported later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13314: [Improvement](like) Change `like` function to batch call
hello-stephen commented on PR #13314: URL: https://github.com/apache/doris/pull/13314#issuecomment-1279751163 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221015215615_clickbench_pr_29762.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus closed pull request #13386: [typo](docs) fix thrift_client_timeout_ms's incorrect description
smallhibiscus closed pull request #13386: [typo](docs) fix thrift_client_timeout_ms's incorrect description URL: https://github.com/apache/doris/pull/13386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] smallhibiscus opened a new pull request, #13391: [typo](docs) Fixed thrift_client_timeout_ms's incorrect description
smallhibiscus opened a new pull request, #13391: URL: https://github.com/apache/doris/pull/13391 …f en docs. # Proposed changes Issue Number: close #xxx ## Problem summary Fixed thrift_client_timeout_ms's incorrect description of en docs. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r57388 - /dev/doris/1.1.3-rc02/ /release/doris/1.1/1.1.3-rc02/
Author: morningman Date: Sat Oct 15 14:15:18 2022 New Revision: 57388 Log: move doris 1.1.3-rc02 to release Added: release/doris/1.1/1.1.3-rc02/ - copied from r57387, dev/doris/1.1.3-rc02/ Removed: dev/doris/1.1.3-rc02/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13388: [chore](build release) remove doris home and user info from doris_be --version output (#13344)
github-actions[bot] commented on PR #13388: URL: https://github.com/apache/doris/pull/13388#issuecomment-1279754797 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13388: [chore](build release) remove doris home and user info from doris_be --version output (#13344)
github-actions[bot] commented on PR #13388: URL: https://github.com/apache/doris/pull/13388#issuecomment-1279754813 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996317416
##
fe/fe-core/src/main/cup/sql_parser.cup:
##
@@ -3619,6 +3621,10 @@ show_param ::=
{:
RESULT = new ShowAnalyzeStmt(tbl, parser.where, orderByClause,
limitClause);
:}
+| KW_DROPPED opt_wild_where
Review Comment:
ok, i will fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996317757
##
fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java:
##
@@ -76,46 +81,56 @@ public synchronized boolean recycleDatabase(Database db,
Set tableNames)
// db should be empty. all tables are recycled before
Preconditions.checkState(db.getTables().isEmpty());
-// erase db with same name
-eraseDatabaseWithSameName(db.getFullName());
+// erase db with same id
+eraseDatabaseWithSameId(db.getId());
// recycle db
-RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames);
+RecycleDatabaseInfo databaseInfo = new RecycleDatabaseInfo(db,
tableNames, tableIds);
idToDatabase.put(db.getId(), databaseInfo);
-idToRecycleTime.put(db.getId(), System.currentTimeMillis());
+if (!isReplay || replayRecycleTime == 0) {
+recycleTime = System.currentTimeMillis();
Review Comment:
yes, sometimes recycleTime can change when fe restart before saving
recycleTime to new image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996319079
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDroppedStmt.java:
##
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+
+import java.util.function.Predicate;
+
+public class ShowDroppedStmt extends ShowStmt {
+public static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+
.add("Type").add("Name").add("DbId").add("TableId").add("PartitionId").add("DropTime")
Review Comment:
这个地方主要想让用户可以在一个查询中就能获取当前id的所属父id是谁，方便用户去区分同名咨询。比如说两个不同的database，每个database都删除过表test，通过上边的命令，就能看到两个不同的表test属于哪个database。否则很难通过一个过期的id去知道这个表曾经的信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996319079
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDroppedStmt.java:
##
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+
+import java.util.function.Predicate;
+
+public class ShowDroppedStmt extends ShowStmt {
+public static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+
.add("Type").add("Name").add("DbId").add("TableId").add("PartitionId").add("DropTime")
Review Comment:
这个地方主要想让用户可以在一个查询中就能获取当前id的所属父id是谁，方便用户去区分同名资源。比如说两个不同的database，每个database都删除过表test，通过上边的命令，就能看到两个不同的表test属于哪个database。否则很难通过一个过期的id去知道这个表曾经的信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on a diff in pull request #13067: [feature-wip](recover) new recover ddl and support show dropped
nextdreamblue commented on code in PR #13067:
URL: https://github.com/apache/doris/pull/13067#discussion_r996319079
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDroppedStmt.java:
##
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.UserException;
+import org.apache.doris.qe.ShowResultSetMetaData;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+
+import java.util.function.Predicate;
+
+public class ShowDroppedStmt extends ShowStmt {
+public static final ImmutableList TITLE_NAMES = new
ImmutableList.Builder()
+
.add("Type").add("Name").add("DbId").add("TableId").add("PartitionId").add("DropTime")
Review Comment:
这个地方主要想让用户可以在一个查询中就能获取当前id的所属父id是谁，方便用户去区分同名资源。比如说两个不同的database，每个database都删除过表test，通过上边的命令，就能看到两个不同的表test属于哪个database。否则很难通过一个过期的(被drop的)id去知道这个表曾经的信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #13056: [Feature](Retention) support retention function
dataroaring commented on code in PR #13056:
URL: https://github.com/apache/doris/pull/13056#discussion_r996316301
##
be/src/vec/aggregate_functions/aggregate_function_retention.h:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/AggregateFunctionRetention.h
+// and modified by Doris
+
+#pragma once
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/io/var_int.h"
+
+namespace doris::vectorized {
+struct RetentionState {
+static constexpr size_t max_events = 32;
+uint8_t events[max_events] = {0};
+
+RetentionState() {}
+
+void reset() {
+for (int64_t i = 0; i < max_events; i++) {
+events[i] = 0;
+}
+}
+
+void add(int event) { events[event] = 1; }
Review Comment:
set is a better name than add here.
##
be/src/vec/aggregate_functions/aggregate_function_retention.h:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This file is copied from
+//
https://github.com/ClickHouse/ClickHouse/blob/master/AggregateFunctionRetention.h
+// and modified by Doris
+
+#pragma once
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/io/var_int.h"
+
+namespace doris::vectorized {
+struct RetentionState {
+static constexpr size_t max_events = 32;
+uint8_t events[max_events] = {0};
+
+RetentionState() {}
+
+void reset() {
+for (int64_t i = 0; i < max_events; i++) {
+events[i] = 0;
+}
+}
+
+void add(int event) { events[event] = 1; }
+
+void merge(const RetentionState& other) {
+for (int64_t i = 0; i < max_events; i++) {
+events[i] |= other.events[i];
+}
+}
+
+void write(BufferWritable& out) const {
+for (int64_t i = 0; i < max_events; i++) {
+int64_t event = events[i];
+write_var_int(event, out);
Review Comment:
We can put these 32 '0|1's in a int.
uint64_t serialized_events = 0;
for (int64_t i = 0; i < max_events; i++) {
serialized_events |= events[i];
serialized_events <<= 1;
}
write_var_int(serialized_events, out)
##
be/src/vec/aggregate_functions/aggregate_function_retention.h:
##
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific l
[GitHub] [doris] hello-stephen opened a new pull request, #13392: [thirdparty](zstd)update dist info and thirdparty change log
hello-stephen opened a new pull request, #13392: URL: https://github.com/apache/doris/pull/13392 # Proposed changes Issue Number: close #xxx ## Problem summary 1. change dist/LICENSE-dist.txt，thirdparty/CHANGELOG.md 2. add zstd-1.5.2.tar.gz into https://doris-thirdparty-hk-1308700295.cos.ap-hongkong.myqcloud.com/thirdparty/ 3. add zstd-1.5.2.tar.gz into https://doris-thirdparty-1308700295.cos.ap-beijing.myqcloud.com/thirdparty/ ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13392: [thirdparty](zstd)update dist info and thirdparty change log
hello-stephen commented on PR #13392: URL: https://github.com/apache/doris/pull/13392#issuecomment-1279767218 @morningman Please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #13393: [feature-wip](unique-key-merge-on-write) fix that delete the bitmap of stale rowset
liaoxin01 opened a new pull request, #13393: URL: https://github.com/apache/doris/pull/13393 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13354: [feature-wip](new-scan) Support stream load with csv in new scan framework
hello-stephen commented on PR #13354: URL: https://github.com/apache/doris/pull/13354#issuecomment-1279809343 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221016025825_clickbench_pr_29807.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #8438: [Feature] [vectorized]support vectorized compaction
github-actions[bot] commented on PR #8438: URL: https://github.com/apache/doris/pull/8438#issuecomment-1279856777 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring opened a new pull request, #13394: [fix](jsonreader) teach jsonreader to release memory (#13336)
dataroaring opened a new pull request, #13394: URL: https://github.com/apache/doris/pull/13394 Allocator of rapidjson does not release memory, this fix use allocator with local buffer and call Clear to release memory allocated beyond local buffer. # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13354: [feature-wip](new-scan) Support stream load with csv in new scan framework
hello-stephen commented on PR #13354: URL: https://github.com/apache/doris/pull/13354#issuecomment-1279868419 https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221016093707_clickbench_pr_29833.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zxealous commented on pull request #13377: [Test](cache)Add remote cache ut
zxealous commented on PR #13377: URL: https://github.com/apache/doris/pull/13377#issuecomment-1279874828 > https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221015010249_clickbench_pr_29479.html @hello-stephen What's the meaning of this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13056: [Feature](Retention) support retention function
github-actions[bot] commented on PR #13056: URL: https://github.com/apache/doris/pull/13056#issuecomment-1279875175 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13056: [Feature](Retention) support retention function
github-actions[bot] commented on PR #13056: URL: https://github.com/apache/doris/pull/13056#issuecomment-1279875179 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on pull request #13240: [fix](ddl) check view name by table name regex when create
nextdreamblue commented on PR #13240: URL: https://github.com/apache/doris/pull/13240#issuecomment-1279878098 > Please add some unit test add unit test done, please review again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #13395: [fix](regression-test) fix that multiple cases conflict with the same table name
liaoxin01 opened a new pull request, #13395: URL: https://github.com/apache/doris/pull/13395 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #13338: [Opt](fun) simd the substring function and use stack buf to speed up
HappenLee merged PR #13338: URL: https://github.com/apache/doris/pull/13338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Opt](fun) simd the substring function and use stack buf to speed up (#13338)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 144486e220 [Opt](fun) simd the substring function and use stack buf to
speed up (#13338)
144486e220 is described below
commit 144486e220d85f587b505757c163fc5f12821c66
Author: HappenLee
AuthorDate: Sun Oct 16 11:48:34 2022 +0800
[Opt](fun) simd the substring function and use stack buf to speed up
(#13338)
---
be/src/vec/functions/function_string.h | 41 ++
1 file changed, 22 insertions(+), 19 deletions(-)
diff --git a/be/src/vec/functions/function_string.h
b/be/src/vec/functions/function_string.h
index 09672f2018..b4ef258639 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -22,6 +22,7 @@
#include
#include
+#include
#include
#include "exprs/math_functions.h"
@@ -50,18 +51,14 @@ namespace doris::vectorized {
inline size_t get_utf8_byte_length(unsigned char byte) {
size_t char_size = 0;
-if (byte >= 0xFC) {
-char_size = 6;
-} else if (byte >= 0xF8) {
-char_size = 5;
+if (byte < 0xC0) {
+char_size = 1;
} else if (byte >= 0xF0) {
char_size = 4;
} else if (byte >= 0xE0) {
char_size = 3;
-} else if (byte >= 0xC0) {
-char_size = 2;
} else {
-char_size = 1;
+char_size = 2;
}
return char_size;
}
@@ -144,6 +141,7 @@ struct SubstringUtil {
assert_cast*>(argument_columns[1].get());
auto specific_len_column =
assert_cast*>(argument_columns[2].get());
+
vector(specific_str_column->get_chars(),
specific_str_column->get_offsets(),
specific_start_column->get_data(),
specific_len_column->get_data(),
null_map->get_data(), res->get_chars(), res->get_offsets());
@@ -160,18 +158,24 @@ private:
int size = offsets.size();
res_offsets.resize(size);
res_chars.reserve(chars.size());
-std::vector index;
+std::array buf;
+std::pmr::monotonic_buffer_resource pool {buf.data(), buf.size()};
+std::pmr::vector index {&pool};
+
+std::pmr::vector> strs(&pool);
+strs.resize(size);
+auto* __restrict data_ptr = chars.data();
+auto* __restrict offset_ptr = offsets.data();
for (int i = 0; i < size; ++i) {
-auto* raw_str = reinterpret_cast(&chars[offsets[i - 1]]);
-int str_size = offsets[i] - offsets[i - 1];
+strs[i].first = data_ptr + offset_ptr[i - 1];
+strs[i].second = offset_ptr[i] - offset_ptr[i - 1];
+}
+
+for (int i = 0; i < size; ++i) {
+auto [raw_str, str_size] = strs[i];
// return empty string if start > src.length
-if (start[i] > str_size) {
-StringOP::push_empty_string(i, res_chars, res_offsets);
-continue;
-}
-// return "" if len < 0 or str == 0 or start == 0
-if (len[i] <= 0 || str_size == 0 || start[i] == 0) {
+if (start[i] > str_size || str_size == 0 || start[i] == 0 ||
len[i] <= 0) {
StringOP::push_empty_string(i, res_chars, res_offsets);
continue;
}
@@ -306,9 +310,8 @@ public:
size_t result, size_t input_rows_count) override {
auto int_type = std::make_shared();
size_t num_columns_without_result = block.columns();
-block.insert({int_type->create_column_const(input_rows_count,
to_field(1))
- ->convert_to_full_column_if_const(),
- int_type, "const 1"});
+block.insert({int_type->create_column_const(input_rows_count,
to_field(1)), int_type,
+ "const 1"});
ColumnNumbers temp_arguments(3);
temp_arguments[0] = arguments[0];
temp_arguments[1] = num_columns_without_result;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13365: [fix](array-type) check value valid while insert data into array column
github-actions[bot] commented on PR #13365: URL: https://github.com/apache/doris/pull/13365#issuecomment-1279889010 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13365: [fix](array-type) check value valid while insert data into array column
github-actions[bot] commented on PR #13365: URL: https://github.com/apache/doris/pull/13365#issuecomment-1279889021 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13361: [fix](array-type) bugfix for array column with delete condition
github-actions[bot] commented on PR #13361: URL: https://github.com/apache/doris/pull/13361#issuecomment-1279889524 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13361: [fix](array-type) bugfix for array column with delete condition
github-actions[bot] commented on PR #13361: URL: https://github.com/apache/doris/pull/13361#issuecomment-1279889529 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13314: [Improvement](like) Change `like` function to batch call
github-actions[bot] commented on PR #13314: URL: https://github.com/apache/doris/pull/13314#issuecomment-1279891290 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13314: [Improvement](like) Change `like` function to batch call
github-actions[bot] commented on PR #13314: URL: https://github.com/apache/doris/pull/13314#issuecomment-1279891295 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #13396: [branch-1.1-lts](memtracker) Fix mem hook cancel query
xinyiZzz opened a new pull request, #13396: URL: https://github.com/apache/doris/pull/13396 # Proposed changes Issue Number: close #xxx ## Problem summary Allow independent enable based on `proc/mem_info` check mem limit and cancel query ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] mrhhsg opened a new pull request, #13397: [improvement](scan) speed up inserting strings into ColumnString
mrhhsg opened a new pull request, #13397: URL: https://github.com/apache/doris/pull/13397 # Proposed changes Test sql: ```sql select count(distinct url) from hits where UserID % 5 = 0; ``` without this opt: https://user-images.githubusercontent.com/1179834/196019282-32dd905d-5cfd-40e7-8de0-cb402652c903.png";> with this opt: https://user-images.githubusercontent.com/1179834/196019272-5d169f5a-4a82-4efd-baf8-98d2972fc51b.png";> ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #13362: [Enhancement](function) refactor of date function
BiteThet merged PR #13362: URL: https://github.com/apache/doris/pull/13362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](function) refactor of date function (#13362)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 632670a49c [Enhancement](function) refactor of date function (#13362)
632670a49c is described below
commit 632670a49c2eb28790510c52d8f5630f141e9c28
Author: Pxl
AuthorDate: Sun Oct 16 14:31:26 2022 +0800
[Enhancement](function) refactor of date function (#13362)
refactor of date function
---
.../exec/format/parquet/vparquet_page_reader.cpp | 1 +
be/src/vec/functions/date_time_transforms.h| 242 -
.../function_date_or_datetime_to_something.h | 3 +-
.../function_date_or_datetime_to_string.cpp| 16 +-
.../function_date_or_datetime_to_string.h | 4 +-
be/src/vec/functions/time_of_function.cpp | 59 ++---
be/src/vec/functions/to_time_function.cpp | 115 +++---
be/src/vec/runtime/vdatetime_value.cpp | 7 +-
be/src/vec/runtime/vdatetime_value.h | 32 ++-
tools/tpch-tools/queries/q9.sql| 2 +-
10 files changed, 188 insertions(+), 293 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/vparquet_page_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_page_reader.cpp
index 00e2ef0926..cd018f4107 100644
--- a/be/src/vec/exec/format/parquet/vparquet_page_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_page_reader.cpp
@@ -19,6 +19,7 @@
#include
+#include "common/config.h"
#include "util/thrift_util.h"
namespace doris::vectorized {
diff --git a/be/src/vec/functions/date_time_transforms.h
b/be/src/vec/functions/date_time_transforms.h
index 8255713221..5fca7b1ce3 100644
--- a/be/src/vec/functions/date_time_transforms.h
+++ b/be/src/vec/functions/date_time_transforms.h
@@ -34,27 +34,20 @@
namespace doris::vectorized {
-#define TIME_FUNCTION_IMPL(CLASS, UNIT, FUNCTION)
\
-template
\
-struct CLASS {
\
-using ARG_TYPE = ArgType;
\
-static constexpr auto name = #UNIT;
\
-
\
-static inline auto execute(const ARG_TYPE& t, bool& is_null) {
\
-const auto& date_time_value = (DateValueType&)(t);
\
-is_null = !date_time_value.is_valid_date();
\
-return date_time_value.FUNCTION;
\
-}
\
-
\
-static DataTypes get_variadic_argument_types() {
\
-if constexpr (std::is_same_v) {
\
-return {std::make_shared()};
\
-} else if constexpr (std::is_same_v>) { \
-return {std::make_shared()};
\
-} else {
\
-return {std::make_shared()};
\
-}
\
-}
\
+#define TIME_FUNCTION_IMPL(CLASS, UNIT, FUNCTION)
\
+template
\
+struct CLASS {
\
+using OpArgType = ArgType;
\
+static constexpr auto name = #UNIT;
\
+
\
+static inline auto execute(const ArgType& t) {
\
+const auto& date_time_value = (typename
DateTraits::T&)(t); \
+return date_time_value.FUNCTION;
\
+}
\
+
\
+static DataTypes get_variadic_argument_types() {
\
+return {std::make_shared::DateType>()}; \
+}
