[GitHub] [incubator-doris] morningman commented on pull request #8442: [Vectorized] Support function tuple is null
morningman commented on pull request #8442: URL: https://github.com/apache/incubator-doris/pull/8442#issuecomment-1064904209 Merge it for quick test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8442: [Vectorized] Support function tuple is null
morningman merged pull request #8442: URL: https://github.com/apache/incubator-doris/pull/8442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8441: [Feature] Some feature of vec exec engine should be done
morningman closed issue #8441: URL: https://github.com/apache/incubator-doris/issues/8441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (37c201a -> 68dd799)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 37c201a [doc] fix metadata operation links to external docs (#8403)
add 68dd799 [improvement](vectorized) Support function tuple is null
(#8442)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/expr.h| 1 -
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/columns/column.h| 3 +
be/src/vec/columns/column_nullable.cpp | 5 ++
be/src/vec/columns/column_nullable.h | 3 +
be/src/vec/columns/column_vector.cpp | 3 +-
be/src/vec/exec/join/vhash_join_node.cpp | 7 +-
be/src/vec/exprs/vexpr.cpp | 5 ++
be/src/vec/exprs/vtuple_is_null_predicate.cpp | 86 ++
.../{vslot_ref.h => vtuple_is_null_predicate.h}| 42 +--
.../doris/analysis/TupleIsNullPredicate.java | 27 ---
.../apache/doris/planner/SingleNodePlanner.java| 5 +-
12 files changed, 147 insertions(+), 41 deletions(-)
create mode 100644 be/src/vec/exprs/vtuple_is_null_predicate.cpp
copy be/src/vec/exprs/{vslot_ref.h => vtuple_is_null_predicate.h} (52%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8444: [fix](report) fix bug that tablet may already be delete when reporting
github-actions[bot] commented on pull request #8444: URL: https://github.com/apache/incubator-doris/pull/8444#issuecomment-1064911344 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch dev created (now 68dd799)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch dev in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. at 68dd799 [improvement](vectorized) Support function tuple is null (#8442) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #8444: [fix](report) fix bug that tablet may already be delete when reporting
caiconghui commented on a change in pull request #8444:
URL: https://github.com/apache/incubator-doris/pull/8444#discussion_r824528071
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##
@@ -6938,6 +6938,14 @@ public void
replayBackendReplicasInfo(BackendReplicasInfo backendReplicasInfo) {
List replicaInfos =
backendReplicasInfo.getReplicaReportInfos();
for (BackendReplicasInfo.ReplicaReportInfo info : replicaInfos) {
+if (tabletInvertedIndex.getTabletMeta(info.tabletId) == null) {
Review comment:
can we invoke check replica or meta exist only once? it seems duplicate
with following check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8366: [Feature] Support user level memory limit settings
morningman closed issue #8366: URL: https://github.com/apache/incubator-doris/issues/8366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8365: [feature](user-property) Support user level exec_mem_limit and load_mem_limit
morningman merged pull request #8365: URL: https://github.com/apache/incubator-doris/pull/8365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new issue #8447: [Feature][Vectorized] support lateral view
BiteThet opened a new issue #8447: URL: https://github.com/apache/incubator-doris/issues/8447 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description support vectorized lateral view. ### Use case ```sql MySQL [test]> select * from test_lv_str lateral view explode_split(k2, ',') tmp1 as e1 lateral view explode_split(k2+k2, ',') tmp2 as e2; +--+---+--+--+ | k1 | k2| e2 | e1 | +--+---+--+--+ |1 | a,b,c | NULL | a| |1 | a,b,c | NULL | b| |1 | a,b,c | NULL | c| +--+---+--+--+ ``` ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt opened a new pull request #8448: [Feature][Vectorized] support lateral view
BiteThet opened a new pull request #8448: URL: https://github.com/apache/incubator-doris/pull/8448 # Proposed changes Issue Number: close #8447 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8357: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
morningman closed issue #8357: URL: https://github.com/apache/incubator-doris/issues/8357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8373: [Refactor] to_bitmap(-1) return NULL instead of return parse failed error_message
morningman merged pull request #8373: URL: https://github.com/apache/incubator-doris/pull/8373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8436: [improvement] Avoid print large string in error log
morningman merged pull request #8436: URL: https://github.com/apache/incubator-doris/pull/8436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8444: [fix](report) fix bug that tablet may already be delete when reporting
morningman merged pull request #8444: URL: https://github.com/apache/incubator-doris/pull/8444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (68dd799 -> e403dbc)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 68dd799 [improvement](vectorized) Support function tuple is null (#8442) add e403dbc [feature](user-property) Support user level exec_mem_limit and load_mem_limit (#8365) No new revisions were added by this update. Summary of changes: .../Account Management/SET PROPERTY.md | 10 ++ .../Account Management/SET PROPERTY.md | 12 ++- .../mysql/privilege/CommonUserProperties.java | 22 .../org/apache/doris/mysql/privilege/PaloAuth.java | 18 ++ .../apache/doris/mysql/privilege/UserProperty.java | 40 ++ .../doris/mysql/privilege/UserPropertyMgr.java | 22 ++-- .../main/java/org/apache/doris/qe/Coordinator.java | 23 ++--- 7 files changed, 139 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (e403dbc -> e0ef9b8)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from e403dbc [feature](user-property) Support user level exec_mem_limit
and load_mem_limit (#8365)
add e0ef9b8 [refactor](vectorized) to_bitmap(-1) return NULL instead of
return parse failed error_message (#8373)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/bitmap_function.cpp | 49 +++---
be/src/exprs/hll_function.cpp | 2 +-
.../aggregate_function_bitmap.h| 17 +-
.../aggregate_function_hll_union_agg.cpp | 18 +-
.../aggregate_function_hll_union_agg.h | 94 ++-
.../aggregate_function_reader.cpp | 2 +-
.../aggregate_function_simple_factory.cpp | 2 +-
be/src/vec/functions/function_bit.cpp | 25 +++
be/src/vec/functions/function_bitmap.cpp | 187 ++---
be/src/vec/functions/function_encryption.cpp | 2 +-
be/src/vec/functions/function_string.cpp | 4 +-
be/src/vec/sink/vtablet_sink.cpp | 2 +-
be/test/exprs/bitmap_function_test.cpp | 7 +-
be/test/vec/function/function_string_test.cpp | 15 ++
docs/.vuepress/sidebar/en.js | 3 +-
docs/.vuepress/sidebar/zh-CN.js| 3 +-
.../sql-functions/bitmap-functions/to_bitmap.md| 11 +-
.../sql-functions/bitwise-functions/bit_length.md} | 26 +--
.../sql-functions/bitmap-functions/to_bitmap.md| 8 +
.../now.md => bitwise-functions/bit_length.md} | 26 +--
.../apache/doris/catalog/AggregateFunction.java| 2 +-
gensrc/script/doris_builtins_functions.py | 14 +-
22 files changed, 338 insertions(+), 181 deletions(-)
copy docs/{zh-CN/sql-reference/sql-functions/date-time-functions/now.md =>
en/sql-reference/sql-functions/bitwise-functions/bit_length.md} (68%)
copy docs/zh-CN/sql-reference/sql-functions/{date-time-functions/now.md =>
bitwise-functions/bit_length.md} (69%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (e0ef9b8 -> a76889b)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from e0ef9b8 [refactor](vectorized) to_bitmap(-1) return NULL instead of return parse failed error_message (#8373) add a76889b [improvement] Avoid print large string in error log (#8436) No new revisions were added by this update. Summary of changes: be/src/vec/common/string_ref.h | 3 ++ be/src/vec/sink/vtablet_sink.cpp | 8 +-- docs/.vuepress/sidebar/en.js | 1 - docs/.vuepress/sidebar/zh-CN.js| 1 - .../sql-functions/string-functions/coalesce.md | 62 - .../conditional-functions/coalesce.md | 4 +- .../sql-functions/string-functions/coalesce.md | 63 -- .../apache/doris/analysis/CreateFunctionStmt.java | 2 +- 8 files changed, 10 insertions(+), 134 deletions(-) delete mode 100644 docs/en/sql-reference/sql-functions/string-functions/coalesce.md delete mode 100644 docs/zh-CN/sql-reference/sql-functions/string-functions/coalesce.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (a76889b -> ffddebf)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from a76889b [improvement] Avoid print large string in error log (#8436) add ffddebf [fix](report) fix bug that tablet may already be delete when reporting (#8444) No new revisions were added by this update. Summary of changes: be/src/olap/tablet.cpp | 27 ++ be/src/olap/tablet.h | 6 +++-- .../java/org/apache/doris/catalog/Catalog.java | 8 +++ 3 files changed, 30 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on a change in pull request #8444: [fix](report) fix bug that tablet may already be delete when reporting
caiconghui commented on a change in pull request #8444:
URL: https://github.com/apache/incubator-doris/pull/8444#discussion_r824528071
##
File path: fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
##
@@ -6938,6 +6938,14 @@ public void
replayBackendReplicasInfo(BackendReplicasInfo backendReplicasInfo) {
List replicaInfos =
backendReplicasInfo.getReplicaReportInfos();
for (BackendReplicasInfo.ReplicaReportInfo info : replicaInfos) {
+if (tabletInvertedIndex.getTabletMeta(info.tabletId) == null) {
Review comment:
can we invoke check replica or meta exist only once? it seems duplicate
with following check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8423: [Fix](transaction) Fix committed transaction couldn't be finished when table is dropped
github-actions[bot] commented on pull request #8423: URL: https://github.com/apache/incubator-doris/pull/8423#issuecomment-1064937676 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8426: [Bug] Transaction may not be finished when table does not exist
morningman closed issue #8426: URL: https://github.com/apache/incubator-doris/issues/8426 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8423: [Fix](transaction) Fix committed transaction couldn't be finished when table is dropped
morningman merged pull request #8423: URL: https://github.com/apache/incubator-doris/pull/8423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (ffddebf -> 4a38f2d)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from ffddebf [fix](report) fix bug that tablet may already be delete when reporting (#8444) add 4a38f2d [fix](transaction) Fix committed transaction couldn't be finished when table is dropped (#8423) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/catalog/Database.java| 19 ++- .../main/java/org/apache/doris/catalog/Table.java | 9 ++--- .../apache/doris/common/util/MetaLockUtils.java| 12 +++ .../apache/doris/service/FrontendServiceImpl.java | 2 +- .../doris/transaction/DatabaseTransactionMgr.java | 38 +- .../doris/transaction/PublishVersionDaemon.java| 7 ++-- 6 files changed, 53 insertions(+), 34 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8387: [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex
github-actions[bot] commented on pull request #8387: URL: https://github.com/apache/incubator-doris/pull/8387#issuecomment-1064945856 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 09/09: [fix](report) fix bug that tablet may already be delete when reporting (#8444)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 1681e4f104b599b432e41621d137ac0bcb1d2577
Author: Mingyu Chen
AuthorDate: Fri Mar 11 17:24:20 2022 +0800
[fix](report) fix bug that tablet may already be delete when reporting
(#8444)
1.
This bug was introduced by #8209.
Error in fe.warn.log:
```
java.lang.IllegalStateException: 560278
at
com.google.common.base.Preconditions.checkState(Preconditions.java:508)
~[spark-dpp-0.15-SNAPSHOT.jar:0.15-SNAPSHOT]
at
org.apache.doris.catalog.TabletInvertedIndex.getReplica(TabletInvertedIndex.java:462)
~[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.catalog.Catalog.replayBackendReplicasInfo(Catalog.java:6941)
~[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:626)
[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2446)
[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:116)
[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:74)
[palo-fe.jar:0.15-SNAPSHOT]
at
org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58)
[palo-fe.jar:0.15-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116)
[palo-fe.jar:0.15-SNAPSHOT]
```
Since the reporting of a tablet and the deletion of a tablet are two
independent events
and are not mutually exclusive, it may happen that the tablet is deleted
first and the reporting is done later.
2.
Change the tablet report info. Now, the version of a tablet report from BE
is the largest continuous version.
Eg, versions: [1,2,3,5,7], the report version of this tablet will be 3.
---
be/src/olap/tablet.cpp | 27 ++
be/src/olap/tablet.h | 6 +++--
.../java/org/apache/doris/catalog/Catalog.java | 8 +++
3 files changed, 30 insertions(+), 11 deletions(-)
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index a6f5648..862dccb 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -807,12 +807,12 @@ void Tablet::calc_missed_versions_unlocked(int64_t
spec_version,
}
}
-void Tablet::max_continuous_version_from_beginning(Version* version) {
+void Tablet::max_continuous_version_from_beginning(Version* version, Version*
max_version) {
ReadLock rdlock(&_meta_lock);
-_max_continuous_version_from_beginning_unlocked(version);
+_max_continuous_version_from_beginning_unlocked(version, max_version);
}
-void Tablet::_max_continuous_version_from_beginning_unlocked(Version* version)
const {
+void Tablet::_max_continuous_version_from_beginning_unlocked(Version* version,
Version* max_version) const {
std::vector existing_versions;
for (auto& rs : _tablet_meta->all_rs_metas()) {
existing_versions.emplace_back(rs->version());
@@ -833,6 +833,9 @@ void
Tablet::_max_continuous_version_from_beginning_unlocked(Version* version) c
max_continuous_version = existing_versions[i];
}
*version = max_continuous_version;
+if (max_version != nullptr) {
+*max_version = existing_versions.back();
+}
}
void Tablet::calculate_cumulative_point() {
@@ -1247,9 +1250,15 @@ void Tablet::build_tablet_report_info(TTabletInfo*
tablet_info) {
tablet_info->row_count = _tablet_meta->num_rows();
tablet_info->data_size = _tablet_meta->tablet_footprint();
-tablet_info->__set_version_miss(false);
-auto max_rowset = rowset_with_max_version();
-if (max_rowset == nullptr) {
+// Here we need to report to FE if there are any missing versions of
tablet.
+// We start from the initial version and traverse backwards until we meet
a discontinuous version.
+Version cversion;
+Version max_version;
+_max_continuous_version_from_beginning_unlocked(&cversion, &max_version);
+tablet_info->__set_version_miss(cversion.second < max_version.second);
+// find rowset with max version
+auto iter = _rs_version_map.find(max_version);
+if (iter == _rs_version_map.end()) {
// If the tablet is in running state, it must not be doing
schema-change. so if we can not
// access its rowsets, it means that the tablet is bad and needs to be
reported to the FE
// for subsequent repairs (through the cloning task)
@@ -1260,10 +1269,10 @@ void Tablet::build_tablet_report_info(TTabletInfo*
tablet_info) {
// still sets the state to normal when reporting. Note that every task
has an timeout,
// so if the task corresponding to this change hangs, when the task
timeout, FE will know
// and perform state
[incubator-doris] 08/09: [improvement] Avoid print large string in error log (#8436)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 3d1c4cf63f804a5ec1d1ae262334ee7c2f25d6f1
Author: Mingyu Chen
AuthorDate: Fri Mar 11 17:23:47 2022 +0800
[improvement] Avoid print large string in error log (#8436)
1. Avoid print large string in error log
If user load a unqualified large string, the all string will be saved
in error log,
so the error log is too big that can not be shown be using `show load
warnings on "url"`.
Err: `Got packet bigger than 'max_allowed_packet' bytes`
2. Remove duplicate help doc
Do not allow doc with same title, or error thrown when starting FE:
`java.lang.IllegalArgumentException: Multiple entries with same key:`
---
be/src/vec/common/string_ref.h | 3 ++
be/src/vec/sink/vtablet_sink.cpp | 8 +--
docs/.vuepress/sidebar/en.js | 1 -
docs/.vuepress/sidebar/zh-CN.js| 1 -
.../sql-functions/string-functions/coalesce.md | 62 -
.../sql-functions/string-functions/coalesce.md | 63 --
.../apache/doris/analysis/CreateFunctionStmt.java | 2 +-
7 files changed, 8 insertions(+), 132 deletions(-)
diff --git a/be/src/vec/common/string_ref.h b/be/src/vec/common/string_ref.h
index ebc4dcc..8f83d07 100644
--- a/be/src/vec/common/string_ref.h
+++ b/be/src/vec/common/string_ref.h
@@ -55,6 +55,9 @@ struct StringRef {
std::string to_string() const { return std::string(data, size); }
std::string_view to_string_view() const { return std::string_view(data,
size); }
+// this is just for show, eg. print data to error log, to avoid print
large string.
+std::string to_prefix(size_t length) const { return std::string(data,
std::min(length, size)); }
+
explicit operator std::string() const { return to_string(); }
StringVal to_string_val() {
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index 25d6117..fc5079d 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -203,7 +203,7 @@ Status VOlapTableSink::_validate_data(RuntimeState* state,
vectorized::Block* bl
fmt::format_to(error_msg, "{}",
"the length of input is too long
than schema. ");
fmt::format_to(error_msg, "column_name: {}; ",
desc->col_name());
-fmt::format_to(error_msg, "input str: [{}] ",
str_val.to_string());
+fmt::format_to(error_msg, "input str: [{}] ",
str_val.to_prefix(10));
fmt::format_to(error_msg, "schema length: {}; ",
desc->type().len);
fmt::format_to(error_msg, "actual length: {}; ",
str_val.size);
} else if (str_val.size > MAX_SIZE_OF_VEC_STRING) {
@@ -211,9 +211,9 @@ Status VOlapTableSink::_validate_data(RuntimeState* state,
vectorized::Block* bl
error_msg, "{}",
"the length of input string is too long
than vec schema. ");
fmt::format_to(error_msg, "column_name: {}; ",
desc->col_name());
-fmt::format_to(error_msg, "input str: [{}] ",
str_val.to_string());
-fmt::format_to(error_msg, "schema length: {}; ",
- MAX_SIZE_OF_VEC_STRING);
+fmt::format_to(error_msg, "input str: [{}] ",
str_val.to_prefix(10));
+fmt::format_to(error_msg, "schema length: {}; ",
desc->type().len);
+fmt::format_to(error_msg, "limit length: {}; ",
MAX_SIZE_OF_VEC_STRING);
fmt::format_to(error_msg, "actual length: {}; ",
str_val.size);
}
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index f64decb..941be11 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -341,7 +341,6 @@ module.exports = [
"ascii",
"bit_length",
"char_length",
- "coalesce",
"concat",
"concat_ws",
"ends_with",
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index 8a4d21f..582407b 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -345,7 +345,6 @@ module.exports = [
"ascii",
"bit_length",
"char_length",
- "coalesce",
"concat",
"concat_ws",
"ends_with",
diff --git a/docs/en/sql-reference/sql-functions/string-functions/coalesce.md
b/docs/en/sql-refer
[incubator-doris] 06/09: [refactor](vectorized) to_bitmap(-1) return NULL instead of return parse failed error_message (#8373)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit dbcb1f0def297630f7638693e87974a2e892ef9a
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Fri Mar 11 17:21:47 2022 +0800
[refactor](vectorized) to_bitmap(-1) return NULL instead of return parse
failed error_message (#8373)
---
be/src/exprs/bitmap_function.cpp | 49 +++---
be/src/exprs/hll_function.cpp | 2 +-
.../aggregate_function_bitmap.h| 17 +-
.../aggregate_function_hll_union_agg.cpp | 18 +-
.../aggregate_function_hll_union_agg.h | 94 ++-
.../aggregate_function_reader.cpp | 2 +-
.../aggregate_function_simple_factory.cpp | 2 +-
be/src/vec/functions/function_bit.cpp | 25 +++
be/src/vec/functions/function_bitmap.cpp | 187 ++---
be/src/vec/functions/function_encryption.cpp | 2 +-
be/src/vec/functions/function_string.cpp | 4 +-
be/src/vec/sink/vtablet_sink.cpp | 2 +-
be/test/exprs/bitmap_function_test.cpp | 7 +-
be/test/vec/function/function_string_test.cpp | 15 ++
docs/.vuepress/sidebar/en.js | 3 +-
docs/.vuepress/sidebar/zh-CN.js| 3 +-
.../sql-functions/bitmap-functions/to_bitmap.md| 11 +-
.../bit_length.md} | 35 ++--
.../sql-functions/bitmap-functions/to_bitmap.md| 8 +
.../sql-functions/bitwise-functions/bit_length.md} | 35 ++--
.../apache/doris/catalog/AggregateFunction.java| 2 +-
gensrc/script/doris_builtins_functions.py | 14 +-
22 files changed, 344 insertions(+), 193 deletions(-)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index aa68880..7e405f3 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -306,7 +306,8 @@ void BitmapFunctions::bitmap_union(FunctionContext* ctx,
const StringVal& src, S
// the dst value could be null
void BitmapFunctions::nullable_bitmap_init(FunctionContext* ctx, StringVal*
dst) {
-dst->is_null = true;
+dst->ptr = nullptr;
+dst->len = 0;
}
void BitmapFunctions::bitmap_intersect(FunctionContext* ctx, const StringVal&
src, StringVal* dst) {
@@ -314,7 +315,7 @@ void BitmapFunctions::bitmap_intersect(FunctionContext*
ctx, const StringVal& sr
return;
}
// if dst is null, the src input is the first value
-if (dst->is_null) {
+if (UNLIKELY(dst->ptr == nullptr)) {
dst->is_null = false;
dst->len = sizeof(BitmapValue);
dst->ptr = (uint8_t*)new BitmapValue((char*)src.ptr);
@@ -358,21 +359,17 @@ BigIntVal BitmapFunctions::bitmap_min(FunctionContext*
ctx, const StringVal& src
StringVal BitmapFunctions::to_bitmap(doris_udf::FunctionContext* ctx,
const doris_udf::StringVal& src) {
-BitmapValue bitmap;
-if (!src.is_null) {
-StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
-uint64_t int_value = StringParser::string_to_unsigned_int(
-reinterpret_cast(src.ptr), src.len, &parse_result);
-if (UNLIKELY(parse_result != StringParser::PARSE_SUCCESS)) {
-std::stringstream error_msg;
-error_msg << "The input: " <<
std::string(reinterpret_cast(src.ptr), src.len)
- << " is not valid, to_bitmap only support bigint value
from 0 to "
- "18446744073709551615 currently";
-ctx->set_error(error_msg.str().c_str());
-return StringVal::null();
-}
-bitmap.add(int_value);
+if (src.is_null) {
+return StringVal::null();
+}
+StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+uint64_t int_value = StringParser::string_to_unsigned_int(
+reinterpret_cast(src.ptr), src.len, &parse_result);
+if (UNLIKELY(parse_result != StringParser::PARSE_SUCCESS)) {
+return StringVal::null();
}
+BitmapValue bitmap;
+bitmap.add(int_value);
return serialize(ctx, &bitmap);
}
@@ -473,8 +470,8 @@ StringVal BitmapFunctions::bitmap_or(FunctionContext* ctx,
const StringVal& lhs,
return serialize(ctx, &bitmap);
}
-StringVal BitmapFunctions::bitmap_or(FunctionContext* ctx, const StringVal&
lhs,
- int num_args, const StringVal*
bitmap_strs) {
+StringVal BitmapFunctions::bitmap_or(FunctionContext* ctx, const StringVal&
lhs, int num_args,
+ const StringVal* bitmap_strs) {
DCHECK_GE(num_args, 1);
if (lhs.is_null || bitmap_strs->is_null) {
return StringVal::null();
@@ -518,8 +515,8 @@ StringVal BitmapFunctions::bitmap_and(Fun
[incubator-doris] 05/09: [feature](user-property) Support user level exec_mem_limit and load_mem_limit (#8365)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 3f64a9355cf5e95847cdc1d13e509d96855d2a12
Author: Mingyu Chen
AuthorDate: Fri Mar 11 17:20:09 2022 +0800
[feature](user-property) Support user level exec_mem_limit and
load_mem_limit (#8365)
```
SET PROPERTY FOR 'jack' 'exec_mem_limit' = '2147483648';
SET PROPERTY FOR 'jack' 'load_mem_limit' = '2147483648';
```
The user level property will overwrite the value in session variables.
---
.../Account Management/SET PROPERTY.md | 10 ++
.../Account Management/SET PROPERTY.md | 12 ++-
.../mysql/privilege/CommonUserProperties.java | 22
.../org/apache/doris/mysql/privilege/PaloAuth.java | 18 ++
.../apache/doris/mysql/privilege/UserProperty.java | 40 ++
.../doris/mysql/privilege/UserPropertyMgr.java | 22 ++--
.../main/java/org/apache/doris/qe/Coordinator.java | 23 ++---
7 files changed, 139 insertions(+), 8 deletions(-)
diff --git a/docs/en/sql-reference/sql-statements/Account Management/SET
PROPERTY.md b/docs/en/sql-reference/sql-statements/Account Management/SET
PROPERTY.md
index f2be712..d0ac580 100644
--- a/docs/en/sql-reference/sql-statements/Account Management/SET PROPERTY.md
+++ b/docs/en/sql-reference/sql-statements/Account Management/SET PROPERTY.md
@@ -42,10 +42,14 @@ max_user_connections: Maximum number of connections.
max_query_instances: Maximum number of query instance user can use when query.
sql_block_rules: set sql block rules。After setting, if the query user execute
match the rules, it will be rejected.
cpu_resource_limit: limit the cpu resource usage of a query. See session
variable `cpu_resource_limit`.
+exec_mem_limit: Limit the memory usage of the query. See the description of
the session variable `exec_mem_limit` for details. -1 means not set.
+load_mem_limit: Limit memory usage for imports. See the introduction of the
session variable `load_mem_limit` for details. -1 means not set.
resource.cpu_share: cpu resource assignment.(Derepcated)
Load_cluster. {cluster_name}. priority: assigns priority to a specified
cluster, which can be HIGH or NORMAL
resource_tags: Specify the user's resource tag permissions.
+> Notice: The `cpu_resource_limit`, `exec_mem_limit`, and `load_mem_limit`
properties default to the values in the session variables if they are not set.
+
Ordinary user rights:
Quota.normal: Resource allocation at the normal level.
Quota.high: Resource allocation at the high level.
@@ -93,6 +97,12 @@ SET PROPERTY FOR 'jack' 'cpu_resource_limit' = '2';
11. Modify user's resource tag permission
SET PROPERTY FOR 'jack' 'resource_tags.location' = 'group_a, group_b';
+12. modify the user's query memory usage limit in bytes
+SET PROPERTY FOR 'jack' 'exec_mem_limit' = '2147483648';
+
+13. modify the user's import memory usage limit in bytes
+SET PROPERTY FOR 'jack' 'load_mem_limit' = '2147483648';
+
## keyword
SET, PROPERTY
diff --git a/docs/zh-CN/sql-reference/sql-statements/Account Management/SET
PROPERTY.md b/docs/zh-CN/sql-reference/sql-statements/Account Management/SET
PROPERTY.md
index 3885d57..fb9d34e 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Account Management/SET
PROPERTY.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Account Management/SET
PROPERTY.md
@@ -41,11 +41,15 @@ under the License.
max_user_connections: 最大连接数。
max_query_instances: 用户同一时间点执行查询可以使用的instance个数。
sql_block_rules: 设置 sql block rules。设置后，该用户发送的查询如果匹配规则，则会被拒绝。
-cpu_resource_limit: 限制查询的cpu资源。详见会话变量 `cpu_resource_limit` 的介绍。
+cpu_resource_limit: 限制查询的cpu资源。详见会话变量 `cpu_resource_limit` 的介绍。-1
表示未设置。
+exec_mem_limit: 限制查询的内存使用。详见会话变量 `exec_mem_limit` 的介绍。-1 表示未设置。
+load_mem_limit: 限制导入的内存使用。详见会话变量 `load_mem_limit` 的介绍。-1 表示未设置。
resource.cpu_share: cpu资源分配。（已废弃）
load_cluster.{cluster_name}.priority: 为指定的cluster分配优先级，可以为 HIGH 或
NORMAL
resource_tags：指定用户的资源标签权限。
+注：`cpu_resource_limit`, `exec_mem_limit`, `load_mem_limit`
三个属性如果未设置，则默认使用会话变量中值。
+
普通用户权限：
quota.normal: normal级别的资源分配。
quota.high: high级别的资源分配。
@@ -93,6 +97,12 @@ under the License.
11. 修改用户的资源标签权限
SET PROPERTY FOR 'jack' 'resource_tags.location' = 'group_a, group_b';
+12. 修改用户的查询内存使用限制，单位字节
+SET PROPERTY FOR 'jack' 'exec_mem_limit' = '2147483648';
+
+13. 修改用户的导入内存使用限制，单位字节
+SET PROPERTY FOR 'jack' 'load_mem_limit' = '2147483648';
+
## keyword
SET, PROPERTY
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/CommonUserProperties.java
index 2b88909..ecaf436 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mysql
[incubator-doris] 04/09: [improvement](vectorized) Support function tuple is null (#8442)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 7fdd24889cc2ec83d7a163ef21d4cc46a3988162
Author: HappenLee
AuthorDate: Fri Mar 11 16:54:37 2022 +0800
[improvement](vectorized) Support function tuple is null (#8442)
---
be/src/exprs/expr.h| 1 -
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/columns/column.h| 3 +
be/src/vec/columns/column_nullable.cpp | 5 ++
be/src/vec/columns/column_nullable.h | 3 +
be/src/vec/columns/column_vector.cpp | 3 +-
be/src/vec/exec/join/vhash_join_node.cpp | 7 +-
be/src/vec/exprs/vexpr.cpp | 5 ++
be/src/vec/exprs/vtuple_is_null_predicate.cpp | 86 ++
be/src/vec/exprs/vtuple_is_null_predicate.h| 48
.../doris/analysis/TupleIsNullPredicate.java | 27 ---
.../apache/doris/planner/SingleNodePlanner.java| 5 +-
12 files changed, 175 insertions(+), 19 deletions(-)
diff --git a/be/src/exprs/expr.h b/be/src/exprs/expr.h
index fdb4755..f8b4aa2 100644
--- a/be/src/exprs/expr.h
+++ b/be/src/exprs/expr.h
@@ -50,7 +50,6 @@ class RuntimeState;
class TColumnValue;
class TExpr;
class TExprNode;
-class SetVar;
class TupleIsNullPredicate;
class VectorizedRowBatch;
class Literal;
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 8365714..6bea9a6 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -99,6 +99,7 @@ set(VEC_FILES
exprs/vexpr_context.cpp
exprs/vliteral.cpp
exprs/vin_predicate.cpp
+ exprs/vtuple_is_null_predicate.cpp
exprs/vslot_ref.cpp
exprs/vcast_expr.cpp
exprs/vcase_expr.cpp
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index cb04a1c..fdfd85b 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -162,6 +162,9 @@ public:
/// Appends a batch elements from other column with the same type
/// indices_begin + indices_end represent the row indices of column src
+/// Warning:
+/// if *indices == -1 means the row is null, only use in outer join,
do not use in any other place
+/// insert -1 in null map to hint the null is produced by outer join
virtual void insert_indices_from(const IColumn& src, const int*
indices_begin, const int* indices_end) = 0;
/// Appends data located in specified memory chunk if it is possible
(throws an exception if it cannot be implemented).
diff --git a/be/src/vec/columns/column_nullable.cpp
b/be/src/vec/columns/column_nullable.cpp
index bae8f9a..9877903 100644
--- a/be/src/vec/columns/column_nullable.cpp
+++ b/be/src/vec/columns/column_nullable.cpp
@@ -112,6 +112,11 @@ StringRef
ColumnNullable::serialize_value_into_arena(size_t n, Arena& arena,
return StringRef(nested_ref.data - s, nested_ref.size + s);
}
+void ColumnNullable::insert_join_null_data() {
+get_nested_column().insert_default();
+get_null_map_data().push_back(-1);
+}
+
const char* ColumnNullable::deserialize_and_insert_from_arena(const char* pos)
{
UInt8 val = *reinterpret_cast(pos);
pos += sizeof(val);
diff --git a/be/src/vec/columns/column_nullable.h
b/be/src/vec/columns/column_nullable.h
index 21c67a5..030ca13 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -80,6 +80,9 @@ public:
/// Will insert null value if pos=nullptr
void insert_data(const char* pos, size_t length) override;
+/// -1 in null map means null is generated by join, only use in tuple is
null
+void insert_join_null_data();
+
StringRef serialize_value_into_arena(size_t n, Arena& arena, char const*&
begin) const override;
const char* deserialize_and_insert_from_arena(const char* pos) override;
void insert_range_from(const IColumn& src, size_t start, size_t length)
override;
diff --git a/be/src/vec/columns/column_vector.cpp
b/be/src/vec/columns/column_vector.cpp
index 132d359..3188a93 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -231,7 +231,8 @@ void ColumnVector::insert_indices_from(const IColumn&
src, const int* indices
// Now Uint8 use to identify null and non null
// 1. nullable column : offset == -1 means is null at the here,
set true here
// 2. real data column : offset == -1 what at is meaningless
-data[origin_size + i] = (offset == -1) ? T{1} :
src_vec.get_element(offset);
+// 3. -1 only use in outer join to hint the null is produced by
outer join
+data[origin_size + i] = (offset == -1) ? UInt8(-1) :
src_vec.get_element(offset);
} else {
data[origin_size + i] = (offset == -1) ? T{0} :
src_vec.get_element(offs
[incubator-doris] 07/09: [fix](transaction) Fix committed transaction couldn't be finished when table is dropped (#8423)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8a2da556e755e6ec4f5afb83b8807c6cbb7874b1
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Fri Mar 11 17:36:23 2022 +0800
[fix](transaction) Fix committed transaction couldn't be finished when
table is dropped (#8423)
Issue Number: close #8426
---
.../java/org/apache/doris/catalog/Database.java| 19 ++-
.../main/java/org/apache/doris/catalog/Table.java | 9 ++---
.../apache/doris/common/util/MetaLockUtils.java| 12 +++
.../apache/doris/service/FrontendServiceImpl.java | 2 +-
.../doris/transaction/DatabaseTransactionMgr.java | 38 +-
.../doris/transaction/PublishVersionDaemon.java| 7 ++--
6 files changed, 53 insertions(+), 34 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
index db93dd9..c1d55c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
@@ -411,14 +411,16 @@ public class Database extends MetaObject implements
Writable {
return views;
}
-public List getTablesOnIdOrderOrThrowException(List
tableIdList) throws MetaNotFoundException {
-List tableList = Lists.newArrayList();
+/**
+ * this method is used for get existed table list by table id list, if
table not exist, just ignore it.
+ */
+public List getTablesOnIdOrderIfExist(List tableIdList) {
+List tableList =
Lists.newArrayListWithCapacity(tableIdList.size());
for (Long tableId : tableIdList) {
Table table = idToTable.get(tableId);
-if (table == null) {
-throw new MetaNotFoundException("unknown table, tableId=" +
tableId);
+if (table != null) {
+tableList.add(table);
}
-tableList.add(table);
}
if (tableList.size() > 1) {
return
tableList.stream().sorted(Comparator.comparing(Table::getId)).collect(Collectors.toList());
@@ -426,13 +428,12 @@ public class Database extends MetaObject implements
Writable {
return tableList;
}
-public List getTablesOnIdOrderWithIgnoringWrongTableId(List
tableIdList) {
-List tableList = Lists.newArrayList();
+public List getTablesOnIdOrderOrThrowException(List
tableIdList) throws MetaNotFoundException {
+List tableList =
Lists.newArrayListWithCapacity(tableIdList.size());
for (Long tableId : tableIdList) {
Table table = idToTable.get(tableId);
if (table == null) {
-LOG.warn("unknown table, tableId=" + tableId);
-continue;
+throw new MetaNotFoundException("unknown table, tableId=" +
tableId);
}
tableList.add(table);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
index a0c1fb1..58cf08a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
@@ -170,11 +170,12 @@ public class Table extends MetaObject implements Writable
{
}
public boolean writeLockIfExist() {
-if (!isDropped) {
-this.rwLock.writeLock().lock();
-return true;
+this.rwLock.writeLock().lock();
+if (isDropped) {
+this.rwLock.writeLock().unlock();
+return false;
}
-return false;
+return true;
}
public boolean tryWriteLock(long timeout, TimeUnit unit) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java
index 42346a9..42fa8cd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/MetaLockUtils.java
@@ -21,6 +21,8 @@ import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Table;
import org.apache.doris.common.MetaNotFoundException;
+import com.google.common.collect.Lists;
+
import java.util.List;
import java.util.concurrent.TimeUnit;
@@ -61,6 +63,16 @@ public class MetaLockUtils {
}
}
+public static List writeLockTablesIfExist(List tableList) {
+List lockedTablesList =
Lists.newArrayListWithCapacity(tableList.size());
+for (Table table : tableList) {
+if (table.writeLockIfExist()) {
+lockedTablesList.add(table);
+}
+}
+return lockedTablesList;
+}
+
public static void writeLockTablesOrMetaException(List tableList)
[incubator-doris] 03/09: [fix](vectorized) core dump on runtime filter insert because of block address change (#8429)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 39d1ff6999f924495bac1de847e4a28083afa064
Author: Pxl <952130...@qq.com>
AuthorDate: Thu Mar 10 19:02:19 2022 +0800
[fix](vectorized) core dump on runtime filter insert because of block
address change (#8429)
---
be/src/vec/exec/join/vhash_join_node.cpp | 5 +
1 file changed, 5 insertions(+)
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp
b/be/src/vec/exec/join/vhash_join_node.cpp
index 9f93f96..e4f8058 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -602,6 +602,11 @@ HashJoinNode::HashJoinNode(ObjectPool* pool, const
TPlanNode& tnode, const Descr
_is_outer_join(_match_all_build || _match_all_probe) {
_runtime_filter_descs = tnode.runtime_filters;
init_join_op();
+
+// avoid vector expand change block address.
+// one block can store 4g data, _build_blocks can store 128*4g data.
+// if probe data bigger than 512g, runtime filter maybe will core dump
when insert data.
+_build_blocks.reserve(128);
}
HashJoinNode::~HashJoinNode() = default;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 02/09: [feature](function) support vectorized digital_masking (#8409)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 13ad45e298f0d2fc134fb2b7e490910bb7b96e0f
Author: Pxl <952130...@qq.com>
AuthorDate: Thu Mar 10 09:07:07 2022 +0800
[feature](function) support vectorized digital_masking (#8409)
---
.../org/apache/doris/catalog/AliasFunction.java| 35 ++
.../java/org/apache/doris/catalog/Function.java| 2 +-
2 files changed, 23 insertions(+), 14 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/AliasFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/AliasFunction.java
index c212976..1203b7f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/AliasFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/AliasFunction.java
@@ -73,7 +73,7 @@ public class AliasFunction extends Function {
}
public static AliasFunction createFunction(FunctionName functionName,
Type[] argTypes, Type retType,
- boolean hasVarArgs,
List parameters, Expr originFunction) {
+boolean hasVarArgs, List parameters, Expr originFunction) {
AliasFunction aliasFunction = new AliasFunction(functionName,
Arrays.asList(argTypes), retType, hasVarArgs);
aliasFunction.setBinaryType(TFunctionBinaryType.NATIVE);
aliasFunction.setUserVisible(true);
@@ -89,7 +89,11 @@ public class AliasFunction extends Function {
* Please ensure that the condition checks in {@link #analyze} are
satisfied
*/
functionSet.addBuiltin(createBuiltin(DIGITAL_MASKING,
Lists.newArrayList(Type.BIGINT), Type.VARCHAR,
-false, Lists.newArrayList("id"), getExpr(oriStmt), true));
+false, Lists.newArrayList("id"), getExpr(oriStmt), true,
false));
+
+functionSet.addBuiltin(createBuiltin(DIGITAL_MASKING,
Lists.newArrayList(Type.BIGINT), Type.VARCHAR,
+false, Lists.newArrayList("id"), getExpr(oriStmt), true,
true));
+
} catch (AnalysisException e) {
LOG.error("Add builtin alias function error {}", e);
}
@@ -110,7 +114,7 @@ public class AliasFunction extends Function {
LOG.info("analysis exception happened when parsing stmt {}, error:
{}",
sql, syntaxError, e);
if (syntaxError == null) {
-throw e;
+throw e;
} else {
throw new AnalysisException(syntaxError, e);
}
@@ -126,13 +130,14 @@ public class AliasFunction extends Function {
}
private static AliasFunction createBuiltin(String name, ArrayList
argTypes, Type retType,
- boolean hasVarArgs, List
parameters, Expr originFunction,
- boolean userVisible) {
+boolean hasVarArgs, List parameters, Expr originFunction,
+boolean userVisible, boolean isVectorized) {
AliasFunction aliasFunction = new AliasFunction(new
FunctionName(name), argTypes, retType, hasVarArgs);
aliasFunction.setBinaryType(TFunctionBinaryType.BUILTIN);
aliasFunction.setUserVisible(userVisible);
aliasFunction.originFunction = originFunction;
aliasFunction.parameters = parameters;
+aliasFunction.vectorized = isVectorized;
return aliasFunction;
}
@@ -154,7 +159,8 @@ public class AliasFunction extends Function {
public void analyze() throws AnalysisException {
if (parameters.size() != getArgs().length) {
-throw new AnalysisException("Alias function [" + functionName() +
"] args number is not equal to parameters number");
+throw new AnalysisException(
+"Alias function [" + functionName() + "] args number is
not equal to parameters number");
}
List exprs;
if (originFunction instanceof FunctionCallExpr) {
@@ -188,7 +194,8 @@ public class AliasFunction extends Function {
Set set = new HashSet<>();
for (String str : parameters) {
if (!set.add(str)) {
-throw new AnalysisException("Alias function [" +
functionName() + "] has duplicate parameter [" + str + "].");
+throw new AnalysisException(
+"Alias function [" + functionName() + "] has duplicate
parameter [" + str + "].");
}
boolean existFlag = false;
// check exprs
@@ -200,7 +207,8 @@ public class AliasFunction extends Function {
existFlag |= typeDefParam.equals(str);
}
if (!existFlag) {
-throw new AnalysisException("Alias function [" +
functionName() + "] do not contain parameter [" + str + "].");
+throw new Ana
[incubator-doris] branch dev-1.0.0 updated (4e7ea50 -> 1681e4f)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 4e7ea50 [fix](vectorized) fix arithmetic calculate get wrong
result(#8226)
new aad6cb5 [fix](replica) handle replica version missing info to avoid
-214 error (#8209)
new 13ad45e [feature](function) support vectorized digital_masking (#8409)
new 39d1ff6 [fix](vectorized) core dump on runtime filter insert because
of block address change (#8429)
new 7fdd248 [improvement](vectorized) Support function tuple is null
(#8442)
new 3f64a93 [feature](user-property) Support user level exec_mem_limit
and load_mem_limit (#8365)
new dbcb1f0 [refactor](vectorized) to_bitmap(-1) return NULL instead of
return parse failed error_message (#8373)
new 8a2da55 [fix](transaction) Fix committed transaction couldn't be
finished when table is dropped (#8423)
new 3d1c4cf [improvement] Avoid print large string in error log (#8436)
new 1681e4f [fix](report) fix bug that tablet may already be delete when
reporting (#8444)
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exprs/bitmap_function.cpp | 49 +++---
be/src/exprs/expr.h| 1 -
be/src/exprs/hll_function.cpp | 2 +-
be/src/olap/tablet.cpp | 40 +++--
be/src/olap/tablet.h | 14 +-
be/src/vec/CMakeLists.txt | 1 +
.../aggregate_function_bitmap.h| 17 +-
.../aggregate_function_hll_union_agg.cpp | 18 +-
.../aggregate_function_hll_union_agg.h | 94 ++-
.../aggregate_function_reader.cpp | 2 +-
.../aggregate_function_simple_factory.cpp | 2 +-
be/src/vec/columns/column.h| 3 +
be/src/vec/columns/column_nullable.cpp | 5 +
be/src/vec/columns/column_nullable.h | 3 +
be/src/vec/columns/column_vector.cpp | 3 +-
be/src/vec/common/string_ref.h | 3 +
be/src/vec/exec/join/vhash_join_node.cpp | 12 +-
be/src/vec/exprs/vexpr.cpp | 5 +
be/src/vec/exprs/vtuple_is_null_predicate.cpp | 86 ++
.../{vslot_ref.h => vtuple_is_null_predicate.h}| 42 +++--
be/src/vec/functions/function_bit.cpp | 25 +++
be/src/vec/functions/function_bitmap.cpp | 187 ++---
be/src/vec/functions/function_encryption.cpp | 2 +-
be/src/vec/functions/function_string.cpp | 4 +-
be/src/vec/sink/vtablet_sink.cpp | 10 +-
be/test/exprs/bitmap_function_test.cpp | 7 +-
be/test/vec/function/function_string_test.cpp | 15 ++
docs/.vuepress/sidebar/en.js | 4 +-
docs/.vuepress/sidebar/zh-CN.js| 4 +-
.../sql-functions/bitmap-functions/to_bitmap.md| 11 +-
.../sql-functions/bitwise-functions/bit_length.md} | 26 +--
.../sql-functions/string-functions/coalesce.md | 62 ---
.../Account Management/SET PROPERTY.md | 10 ++
.../sql-functions/bitmap-functions/to_bitmap.md| 8 +
.../now.md => bitwise-functions/bit_length.md} | 26 +--
.../sql-functions/string-functions/coalesce.md | 63 ---
.../Account Management/SET PROPERTY.md | 12 +-
.../apache/doris/analysis/CreateFunctionStmt.java | 2 +-
.../doris/analysis/TupleIsNullPredicate.java | 27 +--
.../apache/doris/catalog/AggregateFunction.java| 2 +-
.../org/apache/doris/catalog/AliasFunction.java| 35 ++--
.../java/org/apache/doris/catalog/Catalog.java | 41 +
.../java/org/apache/doris/catalog/Database.java| 19 ++-
.../java/org/apache/doris/catalog/EsTable.java | 3 +-
.../java/org/apache/doris/catalog/Function.java| 2 +-
.../java/org/apache/doris/catalog/Replica.java | 58 ---
.../main/java/org/apache/doris/catalog/Table.java | 9 +-
.../apache/doris/catalog/TabletInvertedIndex.java | 10 +-
.../apache/doris/common/util/MetaLockUtils.java| 12 ++
.../doris/external/elasticsearch/EsRestClient.java | 17 +-
.../org/apache/doris/journal/JournalEntity.java| 6 +
.../main/java/org/apache/doris/load/LoadJob.java | 145
.../org/apache/doris/master/ReportHandler.java | 42 ++---
.../mysql/privilege/CommonUserProperties.java | 22 +++
.../org/apache/doris/mysql/privilege/PaloAuth.java | 18 ++
.../apache/doris/mysql/privilege/UserProperty.java | 40 +
.../doris/mysql/privilege/UserPropertyMgr.java | 22 ++-
.../apache/doris/
[incubator-doris] 01/09: [fix](replica) handle replica version missing info to avoid -214 error (#8209)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit aad6cb57d458b790b6e0dced75f86b6499b39b1c
Author: Mingyu Chen
AuthorDate: Wed Mar 9 13:03:22 2022 +0800
[fix](replica) handle replica version missing info to avoid -214 error
(#8209)
In the original tablet reporting information, the version missing
information is done by combining
two pieces of information as follows:
1. the maximum consecutive version number
2. the `version_miss` field
The logic of this approach is confusing and inconsistent with the logic of
checking for missing versions when querying.
After the change, we directly use the version checking logic used in the
query, and set `version_miss` to true
if a missing version is found
and on the FE processing side. Originally, only the **bad replica**
information was syncronized among FEs,
but not the **version missing** information. As a result, the non-master FE
is not aware of the missing version information.
In the new design, we deprecate the original log persistence class
`BackendTabletsInfo` and use the new
`BackendReplicasInfo` to record replica reporting information and write
both **bad** and **version missing**
information to metadata so that other FEs can synchronize these information.
---
be/src/olap/tablet.cpp | 23 ++--
be/src/olap/tablet.h | 8 +-
.../java/org/apache/doris/catalog/Catalog.java | 33 +
.../java/org/apache/doris/catalog/EsTable.java | 3 +-
.../java/org/apache/doris/catalog/Replica.java | 58 +
.../apache/doris/catalog/TabletInvertedIndex.java | 10 +-
.../doris/external/elasticsearch/EsRestClient.java | 17 ++-
.../org/apache/doris/journal/JournalEntity.java| 6 +
.../main/java/org/apache/doris/load/LoadJob.java | 145 +++--
.../org/apache/doris/master/ReportHandler.java | 42 ++
.../apache/doris/persist/BackendReplicasInfo.java | 105 +++
.../apache/doris/persist/BackendTabletsInfo.java | 2 +
.../java/org/apache/doris/persist/EditLog.java | 10 ++
.../org/apache/doris/persist/OperationType.java| 2 +
.../doris/persist/BackendReplicaInfosTest.java | 81
15 files changed, 391 insertions(+), 154 deletions(-)
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index f9856d7..a6f5648 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -552,10 +552,11 @@ bool Tablet::_reconstruct_version_tracker_if_necessary() {
}
OLAPStatus Tablet::capture_consistent_versions(const Version& spec_version,
- std::vector*
version_path) const {
+ std::vector*
version_path,
+ bool quiet) const {
OLAPStatus status =
_timestamped_version_tracker.capture_consistent_versions(spec_version,
version_path);
-if (status != OLAP_SUCCESS) {
+if (status != OLAP_SUCCESS && !quiet) {
std::vector missed_versions;
calc_missed_versions_unlocked(spec_version.second, &missed_versions);
if (missed_versions.empty()) {
@@ -577,9 +578,9 @@ OLAPStatus Tablet::capture_consistent_versions(const
Version& spec_version,
return status;
}
-OLAPStatus Tablet::check_version_integrity(const Version& version) {
+OLAPStatus Tablet::check_version_integrity(const Version& version, bool quiet)
{
ReadLock rdlock(&_meta_lock);
-return capture_consistent_versions(version, nullptr);
+return capture_consistent_versions(version, nullptr, quiet);
}
// If any rowset contains the specific version, it means the version already
exist
@@ -1245,14 +1246,10 @@ void Tablet::build_tablet_report_info(TTabletInfo*
tablet_info) {
tablet_info->schema_hash = _tablet_meta->schema_hash();
tablet_info->row_count = _tablet_meta->num_rows();
tablet_info->data_size = _tablet_meta->tablet_footprint();
-Version version = {-1, 0};
-_max_continuous_version_from_beginning_unlocked(&version);
+
+tablet_info->__set_version_miss(false);
auto max_rowset = rowset_with_max_version();
-if (max_rowset != nullptr) {
-if (max_rowset->version() != version) {
-tablet_info->__set_version_miss(true);
-}
-} else {
+if (max_rowset == nullptr) {
// If the tablet is in running state, it must not be doing
schema-change. so if we can not
// access its rowsets, it means that the tablet is bad and needs to be
reported to the FE
// for subsequent repairs (through the cloning task)
@@ -1263,8 +1260,10 @@ void Tablet::build_tablet_report_info(TTabletInfo*
tablet_info) {
// still sets the state to normal when reporting. Note that e
[GitHub] [incubator-doris] caiconghui merged pull request #8387: [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex
caiconghui merged pull request #8387: URL: https://github.com/apache/incubator-doris/pull/8387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (4a38f2d -> c86d469)
This is an automated email from the ASF dual-hosted git repository. caiconghui pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 4a38f2d [fix](transaction) Fix committed transaction couldn't be finished when table is dropped (#8423) add c86d469 [Refactor](storage_engine) Use std::shared_mutex to replace RWMutex (#8387) No new revisions were added by this update. Summary of changes: be/src/agent/task_worker_pool.cpp | 2 +- be/src/agent/topic_subscriber.cpp | 6 +- be/src/exec/olap_scanner.cpp | 2 +- be/src/olap/compaction.cpp | 4 +- be/src/olap/data_dir.cpp | 6 +- be/src/olap/data_dir.h | 3 +- be/src/olap/delta_writer.cpp | 4 +- be/src/olap/push_handler.cpp | 16 +- be/src/olap/reader.cpp | 12 +- be/src/olap/schema_change.cpp | 214 ++--- be/src/olap/snapshot_manager.cpp | 4 +- be/src/olap/storage_engine.cpp | 18 +- be/src/olap/tablet.cpp | 44 ++--- be/src/olap/tablet.h | 17 +- be/src/olap/tablet_manager.cpp | 101 +- be/src/olap/tablet_manager.h | 16 +- be/src/olap/tablet_meta.cpp| 4 +- be/src/olap/tablet_meta.h | 3 +- be/src/olap/task/engine_checksum_task.cpp | 2 +- be/src/olap/task/engine_clone_task.cpp | 160 --- be/src/olap/task/engine_storage_migration_task.cpp | 45 ++--- be/src/olap/txn_manager.cpp| 30 +-- be/src/olap/txn_manager.h | 6 +- be/src/util/mutex.cpp | 63 -- be/src/util/mutex.h| 96 + be/src/util/thread_group.h | 16 +- 26 files changed, 367 insertions(+), 527 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wangshuo128 commented on a change in pull request #8318: [improvement](storage) Low cardinality string optimization in storage layer
wangshuo128 commented on a change in pull request #8318:
URL: https://github.com/apache/incubator-doris/pull/8318#discussion_r824578717
##
File path: be/src/vec/columns/column_dictionary.h
##
@@ -0,0 +1,381 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+
+#include "gutil/hash/string_hash.h"
+#include "olap/decimal12.h"
+#include "olap/uint24.h"
+#include "runtime/string_value.h"
+#include "util/slice.h"
+#include "vec/columns/column.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_impl.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/predicate_column.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+/**
+ * For low cardinality string columns, using ColumnDictionary can reducememory
+ * usage and improve query efficiency.
+ * For equal predicate comparisons, convert the predicate constant to encodings
+ * according to the dictionary, so that encoding comparisons are used instead
+ * of string comparisons to improve performance.
+ * For range comparison predicates, it is necessary to sort the dictionary
+ * contents, convert the encoding column, and then compare the encoding
directly.
+ * If the read data page contains plain-encoded data pages, the dictionary
+ * columns are converted into PredicateColumn for processing.
+ * Currently ColumnDictionary is only used for storage layer.
+ */
+template
+class ColumnDictionary final : public COWHelper> {
+private:
+friend class COWHelper;
+
+ColumnDictionary() {}
+ColumnDictionary(const size_t n) : codes(n) {}
+ColumnDictionary(const ColumnDictionary& src) : codes(src.codes.begin(),
src.codes.end()) {}
+
+public:
+using Self = ColumnDictionary;
+using value_type = T;
+using Container = PaddedPODArray;
+using DictContainer = PaddedPODArray;
+
+bool is_numeric() const override { return false; }
+
+bool is_predicate_column() const override { return false; }
+
+bool is_column_dictionary() const override { return true; }
+
+size_t size() const override { return codes.size(); }
+
+[[noreturn]] StringRef get_data_at(size_t n) const override {
+LOG(FATAL) << "get_data_at not supported in ColumnDictionary";
+}
+
+void insert_from(const IColumn& src, size_t n) override {
+LOG(FATAL) << "insert_from not supported in ColumnDictionary";
+}
+
+void insert_range_from(const IColumn& src, size_t start, size_t length)
override {
+LOG(FATAL) << "insert_range_from not supported in ColumnDictionary";
+}
+
+void insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) override {
+LOG(FATAL) << "insert_indices_from not supported in ColumnDictionary";
+}
+
+void pop_back(size_t n) override { LOG(FATAL) << "pop_back not supported
in ColumnDictionary"; }
+
+void update_hash_with_value(size_t n, SipHash& hash) const override {
+LOG(FATAL) << "update_hash_with_value not supported in
ColumnDictionary";
+}
+
+void insert_data(const char* pos, size_t /*length*/) override {
+codes.push_back(unaligned_load(pos));
+}
+
+void insert_data(const T value) { codes.push_back(value); }
+
+void insert_default() override { codes.push_back(T()); }
+
+void clear() override { codes.clear(); }
+
+// TODO: Make dict memory usage more precise
+size_t byte_size() const override { return codes.size() *
sizeof(codes[0]); }
+
+size_t allocated_bytes() const override { return byte_size(); }
+
+void protect() override {}
+
+void get_permutation(bool reverse, size_t limit, int nan_direction_hint,
+ IColumn::Permutation& res) const override {
+LOG(FATAL) << "get_permutation not supported in ColumnDictionary";
+}
+
+void reserve(size_t n) override { codes.reserve(n); }
+
+[[noreturn]] const char* get_family_name() const override {
+LOG(FATAL) << "get_family_name not supported in ColumnDictionary";
+}
+
+[[noreturn]] MutableColumnPtr clone_resized(size_t size) const override {
+LOG(FATAL) << "
[GitHub] [incubator-doris] HappenLee opened a new issue #8449: [Bug] BetweenPredicate failed when enable the fold const expr
HappenLee opened a new issue #8449: URL: https://github.com/apache/incubator-doris/issues/8449 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ``` mysql> select 2 between NULL and 1,2 between 3 AND NULL,NULL between 1 and 2,2 between NULL and 3, 2 between 1 AND NULL; ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: BetweenPredicate needs to be rewritten into a CompoundPredicate. ``` ### What You Expected? get the right result ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] HappenLee opened a new pull request #8450: [Bug] Support BetweenPredicate enable fold const expr
HappenLee opened a new pull request #8450: URL: https://github.com/apache/incubator-doris/pull/8450 # Proposed changes Issue Number: close #8449 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 merged pull request #8424: [docs]Update load-manual.md
hf200012 merged pull request #8424: URL: https://github.com/apache/incubator-doris/pull/8424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch branch-0.11 updated: Update load-manual.md (#8424)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch branch-0.11 in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/branch-0.11 by this push: new 39bb335 Update load-manual.md (#8424) 39bb335 is described below commit 39bb335e9cf142cd6245df4769b634d6c587911c Author: yangjun <32293670+yangjun...@users.noreply.github.com> AuthorDate: Fri Mar 11 20:48:58 2022 +0800 Update load-manual.md (#8424) --- docs/zh-CN/administrator-guide/load-data/load-manual.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/zh-CN/administrator-guide/load-data/load-manual.md b/docs/zh-CN/administrator-guide/load-data/load-manual.md index 2eee650..4e02a0b 100644 --- a/docs/zh-CN/administrator-guide/load-data/load-manual.md +++ b/docs/zh-CN/administrator-guide/load-data/load-manual.md @@ -85,9 +85,9 @@ Doris 支持多种导入方式。建议先完整阅读本文档，再根据所 ### Label 和 原子性 -Doris 对所有导入方式提供原子性保证。既保证同一个导入作业内的数据，原子生效。不会出现仅导入部分数据的情况。 +Doris 对所有导入方式提供原子性保证。即保证同一个导入作业内的数据，原子生效。不会出现仅导入部分数据的情况。 -同时，每一个导入作业都有一个由用户指定或者系统自动生成的 Label。Label 在一个 Database 内唯一。当一个 Label 对应的导入作业成功够，不可在重复使用该 Label 提交导入作业。如果 Label 对应的导入作业失败，则可以重复使用。 +同时，每一个导入作业都有一个由用户指定或者系统自动生成的 Label。Label 在一个 Database 内唯一。当一个 Label 对应的导入作业成功过，不可再重复使用该 Label 提交导入作业。如果 Label 对应的导入作业失败，则可以重复使用。 用户可以通过 Label 机制，来保证 Label 对应的数据最多被导入一次，级 At-Most-Once 语义。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r824717556
##
File path:
fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
##
@@ -386,25 +386,18 @@ public void analyze(Analyzer analyzer) throws
UserException {
}
}
-if (columnDef.getType().isHllType()) {
-hasHll = true;
-}
-
-if (columnDef.getAggregateType() == AggregateType.BITMAP_UNION) {
-hasBitmap = columnDef.getType().isBitmapType();
+if (columnDef.getType().isObjectStored()) {
+hasObjectStored = true;
+objectStoredColumn = columnDef.getName();
}
if (!columnSet.add(columnDef.getName())) {
ErrorReport.reportAnalysisException(ErrorCode.ERR_DUP_FIELDNAME,
columnDef.getName());
}
}
-if (hasHll && keysDesc.getKeysType() != KeysType.AGG_KEYS) {
-throw new AnalysisException("HLL must be used in AGG_KEYS");
-}
-
-if (hasBitmap && keysDesc.getKeysType() != KeysType.AGG_KEYS) {
-throw new AnalysisException("BITMAP_UNION must be used in
AGG_KEYS");
+if (hasObjectStored && keysDesc.getKeysType() != KeysType.AGG_KEYS) {
+throw new AnalysisException("object stored column must be used in
AGG_KEYS table, object stored column:" + objectStoredColumn);
Review comment:
ok i will remove the object stored key words just keep the column name
in message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8322: [refactor] Impl of MemTracker, and related use
morningman merged pull request #8322: URL: https://github.com/apache/incubator-doris/pull/8322 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8322: [refactor] Impl of MemTracker, and related use
github-actions[bot] commented on pull request #8322: URL: https://github.com/apache/incubator-doris/pull/8322#issuecomment-1065142696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8450: [Bug] Support BetweenPredicate enable fold const expr
morningman commented on a change in pull request #8450: URL: https://github.com/apache/incubator-doris/pull/8450#discussion_r824748122 ## File path: be/src/vec/columns/column_vector.cpp ## @@ -231,8 +231,8 @@ void ColumnVector::insert_indices_from(const IColumn& src, const int* indices // Now Uint8 use to identify null and non null // 1. nullable column : offset == -1 means is null at the here, set true here // 2. real data column : offset == -1 what at is meaningless -// 3. -1 only use in outer join to hint the null is produced by outer join -data[origin_size + i] = (offset == -1) ? UInt8(-1) : src_vec.get_element(offset); +// 3. 2 only use in outer join to hint the null is produced by outer join +data[origin_size + i] = (offset == -1) ? 2 : src_vec.get_element(offset); Review comment: define `2` somewhere as a constant, to avoid magic number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8439: [refactor] use c++ 14 deprecated instead of comment, this detect usage of deprecated var or func at compile time
morningman commented on a change in pull request #8439:
URL: https://github.com/apache/incubator-doris/pull/8439#discussion_r824750657
##
File path: be/src/exec/tablet_sink.h
##
@@ -165,6 +165,7 @@ class NodeChannel {
Status add_row(Tuple* tuple, int64_t tablet_id);
+// [[deprecated]]
Review comment:
This method is not deprecated. The previous comment is wrong.
##
File path: be/src/agent/task_worker_pool.h
##
@@ -96,16 +92,16 @@ class TaskWorkerPool {
return "REALTIME_PUSH";
case PUBLISH_VERSION:
return "PUBLISH_VERSION";
-case CLEAR_ALTER_TASK:
-return "CLEAR_ALTER_TASK";
+// case CLEAR_ALTER_TASK:
Review comment:
remove it.
##
File path: be/src/agent/task_worker_pool.h
##
@@ -51,12 +51,12 @@ class TaskWorkerPool {
REALTIME_PUSH,
PUBLISH_VERSION,
// Deprecated
-CLEAR_ALTER_TASK,
+CLEAR_ALTER_TASK [[deprecated]],
Review comment:
Just remove it?
##
File path: be/src/exec/tablet_sink.h
##
@@ -291,16 +292,18 @@ class IndexChannel {
Status init(RuntimeState* state, const std::vector&
tablets);
void add_row(Tuple* tuple, int64_t tablet_id);
-
+// [[deprecated]]
Review comment:
This method is not deprecated. The previous comment is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui merged pull request #8324: [Feature](proc) Support proc dir for showing tablet health status
caiconghui merged pull request #8324: URL: https://github.com/apache/incubator-doris/pull/8324 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui closed issue #8323: [Feature] Support proc dir for showing tablet health status
caiconghui closed issue #8323: URL: https://github.com/apache/incubator-doris/issues/8323 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (e17aef9 -> 23d0e7b)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from e17aef9 [refactor] refactor the implement of MemTracker, and related
usage (#8322)
add 23d0e7b [Feature](proc) Support proc dir for showing tablet health
status (#8324)
No new revisions were added by this update.
Summary of changes:
...nitorProcDir.java => ClusterHealthProcDir.java} | 50 ++--
.../common/proc/IncompleteTabletsProcNode.java | 25 +-
.../org/apache/doris/common/proc/ProcService.java | 3 +-
.../apache/doris/common/proc/StatisticProcDir.java | 263
.../doris/common/proc/StatisticProcNode.java | 151 ++
.../doris/common/proc/TabletHealthProcDir.java | 331 +
6 files changed, 522 insertions(+), 301 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/common/proc/{MonitorProcDir.java =>
ClusterHealthProcDir.java} (62%)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/common/proc/StatisticProcDir.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/common/proc/StatisticProcNode.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/common/proc/TabletHealthProcDir.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r824814803
##
File path: be/src/olap/aggregate_func.h
##
@@ -580,6 +581,53 @@ struct
AggregateFuncTraits {
};
+template <>
+struct AggregateFuncTraits {
+static void init(RowCursorCell* dst, const char* src, bool src_null,
MemPool* mem_pool,
+ ObjectPool* agg_pool) {
+DCHECK_EQ(src_null, false);
+dst->set_not_null();
+
+auto* src_slice = reinterpret_cast(src);
+auto* dst_slice = reinterpret_cast(dst->mutable_cell_ptr());
+
+// we use zero size represent this slice is a agg object
+dst_slice->size = 0;
+auto* quantile_state = new QuantileState(*src_slice);
+
+
mem_pool->mem_tracker()->Consume(quantile_state->get_serialized_size());
+
+dst_slice->data = reinterpret_cast(quantile_state);
+
+agg_pool->add(quantile_state);
+}
+
+static void update(RowCursorCell* dst, const RowCursorCell& src, MemPool*
mem_pool) {
+DCHECK_EQ(src.is_null(), false);
+
+auto* dst_slice = reinterpret_cast(dst->mutable_cell_ptr());
+auto* src_slice = reinterpret_cast(src.cell_ptr());
Review comment:
It will meet compile error cause `srr.cell_ptr` is const.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] spaces-X commented on a change in pull request #8234: [Feature] Support pre-aggregation for quantile type
spaces-X commented on a change in pull request #8234:
URL: https://github.com/apache/incubator-doris/pull/8234#discussion_r824834713
##
File path: be/src/exprs/quantile_function.cpp
##
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/quantile_function.h"
+#include "exprs/anyval_util.h"
+#include "gutil/strings/numbers.h"
+#include "gutil/strings/split.h"
+#include "util/quantile_state.h"
+#include "util/string_parser.hpp"
+#include "util/slice.h"
+
+namespace doris {
+
+using doris_udf::DoubleVal;
+using doris_udf::StringVal;
+using doris_udf::FloatVal;
+
+void QuantileStateFunctions::init(){}
+
+void QuantileStateFunctions::quantile_state_init(FunctionContext* ctx,
StringVal* dst) {
+dst->is_null = false;
+dst->len = sizeof(QuantileState);
+dst->ptr = (uint8_t*) new QuantileState();
+}
+
+static StringVal serialize(FunctionContext* ctx, QuantileState* value)
{
+StringVal result(ctx, value->get_serialized_size());
+value->serialize(result.ptr);
+return result;
+}
+
+StringVal QuantileStateFunctions::to_quantile_state(FunctionContext* ctx,
const StringVal& src) {
+QuantileState quantile_state;
+quantile_state.set_compression(2048);
+const AnyVal* digest_compression = ctx->get_constant_arg(1);
+if (digest_compression != nullptr) {
+float compression = reinterpret_cast(digest_compression)->val;
+if (compression >= 2048 && compression <= 1) {
+quantile_state.set_compression(compression);
+}
+}
+
+if(!src.is_null) {
+StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+double double_value = StringParser::string_to_float(
+reinterpret_cast(src.ptr), src.len, &parse_result);
+if (UNLIKELY(parse_result != StringParser::PARSE_SUCCESS)) {
+std::stringstream error_msg;
+error_msg << "The input: " <<
std::string(reinterpret_cast(src.ptr), src.len)
+ << " is not valid, to_bitmap only support bigint value
from 0 to "
+ "18446744073709551615 currently";
+ctx->set_error(error_msg.str().c_str());
+return StringVal::null();
+}
+quantile_state.add_value(double_value);
+}
+return serialize(ctx, &quantile_state);
+}
+
+void QuantileStateFunctions::quantile_union(FunctionContext* ctx, const
StringVal& src, StringVal* dst) {
+if(src.is_null) {
+return;
+}
+auto dst_quantile = reinterpret_cast*>(dst->ptr);
+if(src.len == 0) {
+
dst_quantile->merge(*reinterpret_cast*>(src.ptr));
+} else {
+QuantileState state(Slice(src.ptr, src.len));
+dst_quantile->merge(state);
+}
+}
+
+DoubleVal QuantileStateFunctions::quantile_percent(FunctionContext* ctx,
StringVal& src) {
+const AnyVal* percentile = ctx->get_constant_arg(1);
Review comment:
I did not get the benfit of adding a quantile_percent_prepare function.
May be this can be implemented in support vec-functions of quantile_state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xiaokang opened a new pull request #8451: fix olap table scan and sink memory usage problem
xiaokang opened a new pull request #8451: URL: https://github.com/apache/incubator-doris/pull/8451 # Proposed changes Issue Number: NA ## Problem Summary: Due to unlimited queue in OlapScanNode and NodeChannel, memory usage can be very large for reading and writting large table, e.g 'insert into tableB select * from tableA'. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments 1. Add bytes limit (2.5% of query_options.mem_limit) for _scan_row_batches queue in OlapScanNode. There was no limit for _scan_row_batches yet. 2. Add bytes limit (2.5% of exec_mem_limit) for _materialized_row_batches queue in OlapScanNode. There was only queue size limit for _materialized_row_batches before, and it's not enough for data with large average bytes of a single row. 3. Add bytes limit for reading a single batch. A new be config doris_scanner_row_bytes is added and its default value is 10MB. There was only row number limit before, and it's also not enough for data with large average bytes of a single row. 5. Add bytes limit (5% of load_mem_limit) for _max_pending_batches queue in NodeChannel. 6. Fixed an exceptional zero value of max_thread in OlapScanNode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8451: fix olap table scan and sink memory usage problem
morningman commented on pull request #8451: URL: https://github.com/apache/incubator-doris/pull/8451#issuecomment-1065287730 LGTM, but there are 2 more things: 1. It is better to provide some data to illustrate the memory usage before and after the modification in order to give more information to the reviewer. 2. Also need to modify the `src/vec/exec/volap_scan_node.cpp` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8448: [Feature][Vectorized] support lateral view
morningman commented on a change in pull request #8448:
URL: https://github.com/apache/incubator-doris/pull/8448#discussion_r824671065
##
File path: be/src/exec/table_function_node.h
##
@@ -62,8 +59,11 @@ class TableFunctionNode : public ExecNode {
bool _child_batch_exhausted = true;
std::vector _fn_ctxs;
+std::vector _vfn_ctxs;
+
std::vector _fns;
std::vector _fn_values;
+std::vector _fn_value_lengths;
Review comment:
use int64_t?
##
File path: be/src/vec/exec/vtable_function_node.cpp
##
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/exec/vtable_function_node.h"
+
+#include "exprs/expr.h"
+#include "exprs/expr_context.h"
+#include "exprs/table_function/table_function.h"
+#include "exprs/table_function/table_function_factory.h"
+#include "vec/exprs/vexpr.h"
+
+namespace doris::vectorized {
+
+VTableFunctionNode::VTableFunctionNode(ObjectPool* pool, const TPlanNode&
tnode,
+ const DescriptorTbl& descs)
+: TableFunctionNode(pool, tnode, descs) {}
+
+Status VTableFunctionNode::init(const TPlanNode& tnode, RuntimeState* state) {
+RETURN_IF_ERROR(ExecNode::init(tnode, state));
+
+for (const TExpr& texpr : tnode.table_function_node.fnCallExprList) {
+VExprContext* ctx = nullptr;
+RETURN_IF_ERROR(VExpr::create_expr_tree(_pool, texpr, &ctx));
+_vfn_ctxs.push_back(ctx);
+
+VExpr* root = ctx->root();
+const std::string& tf_name = root->fn().name.function_name;
+TableFunction* fn = nullptr;
+RETURN_IF_ERROR(TableFunctionFactory::get_fn("v" + tf_name, _pool,
&fn));
+fn->set_vexpr_context(ctx);
+_fns.push_back(fn);
+}
+_fn_num = _fns.size();
+_fn_values.resize(_fn_num);
+_fn_value_lengths.resize(_fn_num);
+
+// Prepare output slot ids
+RETURN_IF_ERROR(_prepare_output_slot_ids(tnode));
+return Status::OK();
+}
+
+Status VTableFunctionNode::prepare(RuntimeState* state) {
+SCOPED_TIMER(_runtime_profile->total_time_counter());
+RETURN_IF_ERROR(TableFunctionNode::prepare(state));
+RETURN_IF_ERROR(VExpr::prepare(_vfn_ctxs, state, _row_descriptor,
expr_mem_tracker()));
+
+// get current all output slots
+for (const auto& tuple_desc : this->row_desc().tuple_descriptors()) {
+for (const auto& slot_desc : tuple_desc->slots()) {
+_output_slots.push_back(slot_desc);
+}
+}
+
+// get all input slots
+for (const auto& child_tuple_desc :
child(0)->row_desc().tuple_descriptors()) {
+for (const auto& child_slot_desc : child_tuple_desc->slots()) {
+_child_slots.push_back(child_slot_desc);
+}
+}
+
+_child_block.reset(new Block());
+_cur_child_offset = -1;
+
+return Status::OK();
+}
+
+Status VTableFunctionNode::get_next(RuntimeState* state, Block* block, bool*
eos) {
+SCOPED_TIMER(_runtime_profile->total_time_counter());
+
+if (state == nullptr || block == nullptr || eos == nullptr) {
Review comment:
Is this check necessary?
##
File path: be/src/vec/exec/vrepeat_node.cpp
##
@@ -181,13 +187,9 @@ Status VRepeatNode::get_next(RuntimeState* state, Block*
block, bool* eos) {
// current child block has finished its repeat, get child's next block
if (_child_block->rows() == 0) {
-if (_child_eos) {
Review comment:
Is this a bug before?
##
File path: be/src/exprs/table_function/table_function_factory.cpp
##
@@ -21,25 +21,30 @@
#include "exprs/table_function/explode_bitmap.h"
#include "exprs/table_function/explode_json_array.h"
#include "exprs/table_function/explode_split.h"
+#include "vec/exprs/table_function/vexplode_split.h"
namespace doris {
-Status TableFunctionFactory::get_fn(const std::string& fn_name, ObjectPool*
pool, TableFunction** fn) {
+Status TableFunctionFactory::get_fn(const std::string& fn_name, ObjectPool*
pool,
+TableFunction** fn) {
if (fn_name == "explode_split") {
*fn = pool->add(new ExplodeSplitTableFunction());
-return Status::OK();
+return Status::O
[GitHub] [incubator-doris] xiaokang commented on pull request #8451: [improvement](memory) fix olap table scan and sink memory usage problem
xiaokang commented on pull request #8451:
URL: https://github.com/apache/incubator-doris/pull/8451#issuecomment-1065797757
Hi @morningman, Thanks for your quick response!
>
> 1. It is better to provide some data to illustrate the memory usage before
and after the modification in order to give more information to the reviewer.
Test environment:
A single node cluster with 32g memory and 8 cpu cores. All configuration is
default.
Table schema:
CREATE TABLE `tableA` (
`uuid` text NULL,
`no` text NULL,
`f` text NULL,
`m` text NULL,
`data` text NULL,
`time1` datetime NULL,
`time2` datetime NULL,
`p` text NOT NULL
) ENGINE=OLAP
DUPLICATE KEY(`uuid`)
PARTITION BY LIST(`p`)
(PARTITION p1 VALUES IN ("1"),
...
PARTITION p98 VALUES IN ("98"),
PARTITION p99 VALUES IN ("99"))
DISTRIBUTED BY HASH(`uuid`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
)
Test steps:
1. create tableA and tableB with 100 partition.
2. insert 1 million rows with large average size 100KB to tableA.
3. execute query: 'set batch_size = 128; set exec_mem_limit=8147483648;
insert into tableB select * from table A;'
Result:
- master branch: be process quickly consume all 32g memory in about 20
seconds and killed by Linux oom-killer.
- fix branch: The memory usage of be process is 14g memory at peak and fall
back to 8g when query finished. 8g is for be storage page cache (32g*20%=6g)
and chunk_reserved_bytes (2g).
> 2. Also need to modify the `src/vec/exec/volap_scan_node.cpp`
I will try to do the same things for the vectorized code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #8452: [Refactor](mutex) Use std::mutex to replace Mutex and refactor some lock logic
caiconghui opened a new pull request #8452: URL: https://github.com/apache/incubator-doris/pull/8452 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
