[GitHub] [incubator-doris] Gabriel39 commented on a change in pull request #6973: [Feature] Spark connector supports to specify fields to write
Gabriel39 commented on a change in pull request #6973:
URL: https://github.com/apache/incubator-doris/pull/6973#discussion_r739777540
##
File path:
extension/spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java
##
@@ -523,7 +470,13 @@ public static String randomBackend(SparkSettings
sparkSettings , Logger logger)
logger.error(SHOULD_NOT_HAPPEN_MESSAGE);
throw new ShouldNeverHappenException();
}
-List backendRows = backend.getRows().stream().filter(v ->
v.getAlive()).collect(Collectors.toList());
+List backendRows = backend.stream().map(array -> {
+BackendRow backendRow = new BackendRow();
+backendRow.setIP(array.get(2));
Review comment:
Why changed here? I think parsing to BackendRow is a more reasonable way
than List>. Parsing to BackendRow can eliminate hard code here.
##
File path:
extension/spark-doris-connector/src/test/scala/org/apache/doris/spark/sql/TestSparkConnector.scala
##
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.spark.sql
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.{SparkConf, SparkContext}
+import org.junit.Test
+
+class TestSparkConnector {
+ val dorisFeNodes = "your_fe_host:8030"
+ val dorisUser = "root"
+ val dorisPwd = ""
+ val dorisTable = "test.test_tbl"
+
+ val kafkaServers = ""
+ val kafkaTopics = ""
+
+ @Test
+ def rddReadTest(): Unit = {
+val sparkConf: SparkConf = new SparkConf().setMaster("local[*]")
+val sc = new SparkContext(sparkConf)
+//sc.setLogLevel("DEBUG")
Review comment:
comment line should be removed.
##
File path:
extension/spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java
##
@@ -541,7 +494,7 @@ public static String randomBackend(SparkSettings
sparkSettings , Logger logger)
*/
@VisibleForTesting
static List tabletsMapToPartition(Settings cfg,
Map> be2Tablets,
-String opaquedQueryPlan, String database, String table, Logger
logger)
+ String
opaquedQueryPlan, String database, String table, Logger logger)
Review comment:
The code style should be consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 commented on pull request #6973: [Feature] Spark connector supports to specify fields to write
Gabriel39 commented on pull request #6973: URL: https://github.com/apache/incubator-doris/pull/6973#issuecomment-955660653 one more question, do you think `doris.write.fields` sounds more reasonable than `doris.write.field` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chovy-3012 commented on a change in pull request #6973: [Feature] Spark connector supports to specify fields to write
chovy-3012 commented on a change in pull request #6973: URL: https://github.com/apache/incubator-doris/pull/6973#discussion_r739787630 ## File path: extension/spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java ## @@ -541,7 +494,7 @@ public static String randomBackend(SparkSettings sparkSettings , Logger logger) */ @VisibleForTesting static List tabletsMapToPartition(Settings cfg, Map> be2Tablets, -String opaquedQueryPlan, String database, String table, Logger logger) + String opaquedQueryPlan, String database, String table, Logger logger) Review comment: Sorry😅, this `RestService.java` file shuld not been commited, it's my local file . Because of the version of my doris cluster is 0.12 which doesn't have `HTTPV2` API yet. My fault!😇 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chovy-3012 commented on a change in pull request #6973: [Feature] Spark connector supports to specify fields to write
chovy-3012 commented on a change in pull request #6973:
URL: https://github.com/apache/incubator-doris/pull/6973#discussion_r739787732
##
File path:
extension/spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java
##
@@ -523,7 +470,13 @@ public static String randomBackend(SparkSettings
sparkSettings , Logger logger)
logger.error(SHOULD_NOT_HAPPEN_MESSAGE);
throw new ShouldNeverHappenException();
}
-List backendRows = backend.getRows().stream().filter(v ->
v.getAlive()).collect(Collectors.toList());
+List backendRows = backend.stream().map(array -> {
+BackendRow backendRow = new BackendRow();
+backendRow.setIP(array.get(2));
Review comment:
The same as above
##
File path:
extension/spark-doris-connector/src/test/scala/org/apache/doris/spark/sql/TestSparkConnector.scala
##
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.spark.sql
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.{SparkConf, SparkContext}
+import org.junit.Test
+
+class TestSparkConnector {
+ val dorisFeNodes = "your_fe_host:8030"
+ val dorisUser = "root"
+ val dorisPwd = ""
+ val dorisTable = "test.test_tbl"
+
+ val kafkaServers = ""
+ val kafkaTopics = ""
+
+ @Test
+ def rddReadTest(): Unit = {
+val sparkConf: SparkConf = new SparkConf().setMaster("local[*]")
+val sc = new SparkContext(sparkConf)
+//sc.setLogLevel("DEBUG")
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chovy-3012 commented on pull request #6973: [Feature] Spark connector supports to specify fields to write
chovy-3012 commented on pull request #6973: URL: https://github.com/apache/incubator-doris/pull/6973#issuecomment-955672095 > one more question, do you think `doris.write.fields` sounds more reasonable than `doris.write.field` ? As to the filed name , I just wanted to be consistent with the field `doris.read.field` http://doris.apache.org/master/en/extending-doris/spark-doris-connector.html#sql-dataframe-configuration -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zbtzbtzbt commented on a change in pull request #6964: Prevent invalid expr continue parse
zbtzbtzbt commented on a change in pull request #6964: URL: https://github.com/apache/incubator-doris/pull/6964#discussion_r739788456 ## File path: be/src/exprs/expr.cpp ## @@ -283,6 +283,9 @@ Status Expr::create_tree_from_thrift(ObjectPool* pool, const std::vectorhttps://github.com/apache/incubator-doris/pull/5799](https://github.com/apache/incubator-doris/pull/5799). so this pr can be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chovy-3012 commented on pull request #6973: [Feature] Spark connector supports to specify fields to write
chovy-3012 commented on pull request #6973: URL: https://github.com/apache/incubator-doris/pull/6973#issuecomment-955677582 1. fix option `doris.write.field` to `doris.write.fields` 2. remove comment line in `TestSparkConnector.scala` 3. update en/cn doc for `doris.write.fields` 4. pls remove `RestService.java` in second commit , it shuld not been commited. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6910: [Feature] Support bitmap_and_not & bitmap_and_not_count
github-actions[bot] commented on pull request #6910: URL: https://github.com/apache/incubator-doris/pull/6910#issuecomment-955694832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6976: generage compile_command.json
github-actions[bot] commented on pull request #6976: URL: https://github.com/apache/incubator-doris/pull/6976#issuecomment-955695670 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6975: [BUG][Schedule] Fix getMixLoadScore error.
github-actions[bot] commented on pull request #6975: URL: https://github.com/apache/incubator-doris/pull/6975#issuecomment-955696414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] GoGoWen commented on a change in pull request #6965: optimize for single rowset reading
GoGoWen commented on a change in pull request #6965:
URL: https://github.com/apache/incubator-doris/pull/6965#discussion_r739820481
##
File path: be/src/olap/reader.cpp
##
@@ -130,10 +130,16 @@ OLAPStatus Reader::init(const ReaderParams& read_params) {
<< ", version:" << read_params.version;
return res;
}
-
+// optimize for single rowset reading without do aggregation when reading
all columns,
+// and otherwise should use _agg_key_next_row for AGG_KEYS
if (_optimize_for_single_rowset(rs_readers)) {
-_next_row_func = _tablet->keys_type() == AGG_KEYS ?
&Reader::_direct_agg_key_next_row
- :
&Reader::_direct_next_row;
+if(_tablet->keys_type() == AGG_KEYS && _return_columns.size() ==
_tablet->tablet_schema().num_columns()) {
+_next_row_func = &Reader::_direct_agg_key_next_row;
+} else if (_tablet->keys_type() == AGG_KEYS) {
+_next_row_func = &Reader::_agg_key_next_row;
Review comment:
For example, the table test (k int ,k1 int ,k2 int ,k3 int, v int sum),
if we read all columns (k,k1,k2,k3,v), then it definitely no need to call
agg. However, if we just need (k,k1,v), then if we do not call agg, there
are many many duplicate-key rows for next caller(may be aggregation node),
it's not a good idea to pass these rows without aggregation to aggregation node
for a query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #6875: [Feature] I hope to support more bitmap functions with the same functions as Clickhouse
morningman closed issue #6875: URL: https://github.com/apache/incubator-doris/issues/6875 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6910: [Feature] Support bitmap_and_not & bitmap_and_not_count
morningman merged pull request #6910: URL: https://github.com/apache/incubator-doris/pull/6910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Feature] Support bitmap_and_not & bitmap_and_not_count (#6910)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2803029 [Feature] Support bitmap_and_not & bitmap_and_not_count
(#6910)
2803029 is described below
commit 28030294f707785249ff4932c9ea91cbdf5e88ec
Author: Pxl <952130...@qq.com>
AuthorDate: Mon Nov 1 10:11:54 2021 +0800
[Feature] Support bitmap_and_not & bitmap_and_not_count (#6910)
Support bitmap_and_not & bitmap_and_not_count.
---
be/src/exprs/bitmap_function.cpp | 10 +++
be/src/exprs/bitmap_function.h | 5 ++
be/test/exprs/bitmap_function_test.cpp | 81 ++
docs/.vuepress/sidebar/en.js | 2 +
docs/.vuepress/sidebar/zh-CN.js| 2 +
.../bitmap-functions/bitmap_and_not.md | 48 +
.../bitmap-functions/bitmap_and_not_count.md | 49 +
.../bitmap-functions/bitmap_and_not.md | 48 +
.../bitmap-functions/bitmap_and_not_count.md | 48 +
gensrc/script/doris_builtins_functions.py | 6 ++
10 files changed, 299 insertions(+)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index 5566109..7d03984 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -530,6 +530,16 @@ StringVal BitmapFunctions::bitmap_not(FunctionContext*
ctx, const StringVal& lhs
return serialize(ctx, &bitmap);
}
+StringVal BitmapFunctions::bitmap_and_not(FunctionContext* ctx, const
StringVal& lhs,
+ const StringVal& rhs) {
+return bitmap_xor(ctx, lhs, bitmap_and(ctx, lhs, rhs));
+}
+
+BigIntVal BitmapFunctions::bitmap_and_not_count(FunctionContext* ctx, const
StringVal& lhs,
+ const StringVal& rhs) {
+return bitmap_count(ctx, bitmap_and_not(ctx, lhs, rhs));
+}
+
StringVal BitmapFunctions::bitmap_to_string(FunctionContext* ctx, const
StringVal& input) {
if (input.is_null) {
return StringVal::null();
diff --git a/be/src/exprs/bitmap_function.h b/be/src/exprs/bitmap_function.h
index 5aca41f..a07af1c 100644
--- a/be/src/exprs/bitmap_function.h
+++ b/be/src/exprs/bitmap_function.h
@@ -55,6 +55,8 @@ public:
static void nullable_bitmap_init(FunctionContext* ctx, StringVal* dst);
static void bitmap_intersect(FunctionContext* ctx, const StringVal& src,
StringVal* dst);
static BigIntVal bitmap_count(FunctionContext* ctx, const StringVal& src);
+static BigIntVal bitmap_and_not_count(FunctionContext* ctx, const
StringVal& src,
+ const StringVal& dst);
static BigIntVal bitmap_min(FunctionContext* ctx, const StringVal& str);
static StringVal bitmap_serialize(FunctionContext* ctx, const StringVal&
src);
@@ -64,6 +66,9 @@ public:
static StringVal bitmap_xor(FunctionContext* ctx, const StringVal& src,
const StringVal& dst);
static StringVal bitmap_and(FunctionContext* ctx, const StringVal& src,
const StringVal& dst);
static StringVal bitmap_not(FunctionContext* ctx, const StringVal& src,
const StringVal& dst);
+static StringVal bitmap_and_not(FunctionContext* ctx, const StringVal& src,
+const StringVal& dst);
+
static StringVal bitmap_to_string(FunctionContext* ctx, const StringVal&
input);
// Convert a comma separated string to a Bitmap
// Example:
diff --git a/be/test/exprs/bitmap_function_test.cpp
b/be/test/exprs/bitmap_function_test.cpp
index 98bae2a..a4e2fb8 100644
--- a/be/test/exprs/bitmap_function_test.cpp
+++ b/be/test/exprs/bitmap_function_test.cpp
@@ -394,6 +394,87 @@ TEST_F(BitmapFunctionsTest, bitmap_not) {
ASSERT_EQ(expected, result);
}
+TEST_F(BitmapFunctionsTest, bitmap_and_not) {
+{
+BitmapValue bitmap1({1, 2, 3});
+BitmapValue bitmap2({3, 4, 5});
+
+StringVal bitmap_src = convert_bitmap_to_string(ctx, bitmap1);
+StringVal bitmap_dst = convert_bitmap_to_string(ctx, bitmap2);
+
+StringVal bitmap_str = BitmapFunctions::bitmap_and_not(ctx,
bitmap_src, bitmap_dst);
+BigIntVal result = BitmapFunctions::bitmap_count(ctx, bitmap_str);
+
+BigIntVal expected(2);
+ASSERT_EQ(expected, result);
+}
+{
+BitmapValue bitmap1({1, 2, 3});
+BitmapValue bitmap2({3, 2, 1});
+
+StringVal bitmap_src = convert_bitmap_to_string(ctx, bitmap1);
+StringVal bitmap_dst = convert_bitmap_to_string(ctx, bitmap2);
+
+StringVal bitmap_str = BitmapFunctions::bitmap_and_not(ctx,
bitmap_src, bitmap_dst);
+BigIntVal result = BitmapFunctions::bitmap_count(ctx, bitmap_str);
+
+BigIntVal expected(0);
+ASSERT_EQ(expected, result);
+}
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6947: [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch
github-actions[bot] commented on pull request #6947: URL: https://github.com/apache/incubator-doris/pull/6947#issuecomment-955870302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6962: [S3] Support path style endpoint
github-actions[bot] commented on pull request #6962: URL: https://github.com/apache/incubator-doris/pull/6962#issuecomment-955871346 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739921631
##
File path: be/src/util/brpc_stub_cache.h
##
@@ -34,25 +34,32 @@ class BrpcStubCache {
BrpcStubCache();
virtual ~BrpcStubCache();
-virtual PBackendService_Stub* get_stub(const butil::EndPoint& endpoint) {
-std::lock_guard l(_lock);
+virtual std::shared_ptr get_stub(const
butil::EndPoint& endpoint) {
+std::lock_guard l(_mutex);
auto stub_ptr = _stub_map.seek(endpoint);
if (stub_ptr != nullptr) {
-return *stub_ptr;
+if (available(*stub_ptr, endpoint)) {
Review comment:
Checking availability is a low-cost check. Checking and refreshing
before obtaining the stub is more appropriate than handling it after failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739922341
##
File path: be/src/exprs/runtime_filter_rpc.cpp
##
@@ -39,7 +39,12 @@ struct IRuntimeFilter::rpc_context {
Status IRuntimeFilter::push_to_remote(RuntimeState* state, const
TNetworkAddress* addr) {
DCHECK(is_producer());
DCHECK(_rpc_context == nullptr);
-PBackendService_Stub* stub =
state->exec_env()->brpc_stub_cache()->get_stub(*addr);
+std::shared_ptr stub(
+state->exec_env()->brpc_stub_cache()->get_stub(*addr));
+if (!stub) {
+LOG(WARNING) << "Get rpc stub failed, host=" << addr->hostname << ",
port=" << addr->port;
+return Status::InternalError("get rpc stub failed");
Review comment:
why get failed is logged in
state->exec_env()->brpc_stub_cache()->get_stub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on a change in pull request #6970: Add stream load info to system info of web site
tianhui5 commented on a change in pull request #6970:
URL: https://github.com/apache/incubator-doris/pull/6970#discussion_r739922335
##
File path:
fe/fe-core/src/main/java/org/apache/doris/common/proc/ProcService.java
##
@@ -38,7 +38,7 @@ private ProcService() {
root.register("backends", new
BackendsProcDir(Catalog.getCurrentSystemInfo()));
root.register("dbs", new DbsProcDir(Catalog.getCurrentCatalog()));
root.register("jobs", new JobsDbProcDir(Catalog.getCurrentCatalog()));
-root.register("statistic", new
StatisticProcDir(Catalog.getCurrentCatalog()));
Review comment:
That makes sense, I'll change it back right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739922341
##
File path: be/src/exprs/runtime_filter_rpc.cpp
##
@@ -39,7 +39,12 @@ struct IRuntimeFilter::rpc_context {
Status IRuntimeFilter::push_to_remote(RuntimeState* state, const
TNetworkAddress* addr) {
DCHECK(is_producer());
DCHECK(_rpc_context == nullptr);
-PBackendService_Stub* stub =
state->exec_env()->brpc_stub_cache()->get_stub(*addr);
+std::shared_ptr stub(
+state->exec_env()->brpc_stub_cache()->get_stub(*addr));
+if (!stub) {
+LOG(WARNING) << "Get rpc stub failed, host=" << addr->hostname << ",
port=" << addr->port;
+return Status::InternalError("get rpc stub failed");
Review comment:
The reason why get failed is logged in
`state->exec_env()->brpc_stub_cache()->get_stub`
##
File path: be/src/runtime/data_stream_sender.cpp
##
@@ -105,17 +105,18 @@ Status DataStreamSender::Channel::init(RuntimeState*
state) {
_brpc_request.set_be_number(_be_number);
_brpc_timeout_ms = std::min(3600, state->query_options().query_timeout) *
1000;
-if (_brpc_dest_addr.hostname == BackendOptions::get_localhost()) {
-_brpc_stub =
-state->exec_env()->brpc_stub_cache()->get_stub("127.0.0.1",
_brpc_dest_addr.port);
-} else {
-_brpc_stub =
state->exec_env()->brpc_stub_cache()->get_stub(_brpc_dest_addr);
-}
// In bucket shuffle join will set fragment_instance_id (-1, -1)
// to build a camouflaged empty channel. the ip and port is '0.0.0.0:0"
// so the empty channel not need call function close_internal()
_need_close = (_fragment_instance_id.hi != -1 && _fragment_instance_id.lo
!= -1);
+if (_need_close) {
+_brpc_stub =
state->exec_env()->brpc_stub_cache()->get_stub(_brpc_dest_addr);
+if (!_brpc_stub) {
+LOG(WARNING) << "Get rpc stub failed, dest_addr=" <<
_brpc_dest_addr;
+return Status::InternalError("get rpc stub failed");
Review comment:
The reason why get failed is logged in
state->exec_env()->brpc_stub_cache()->get_stub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739923930
##
File path: docs/zh-CN/administrator-guide/http-actions/check-reset-rpc-cache.md
##
@@ -0,0 +1,46 @@
+---
+{
+"title": "检查和重置连接缓存",
Review comment:
those files already in sidebar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739924897
##
File path: be/src/util/brpc_stub_cache.h
##
@@ -34,25 +34,32 @@ class BrpcStubCache {
BrpcStubCache();
virtual ~BrpcStubCache();
-virtual PBackendService_Stub* get_stub(const butil::EndPoint& endpoint) {
-std::lock_guard l(_lock);
+virtual std::shared_ptr get_stub(const
butil::EndPoint& endpoint) {
+std::lock_guard l(_mutex);
Review comment:
I think we should try to use the class in stl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739925901
##
File path: be/src/exprs/runtime_filter_rpc.cpp
##
@@ -39,7 +39,12 @@ struct IRuntimeFilter::rpc_context {
Status IRuntimeFilter::push_to_remote(RuntimeState* state, const
TNetworkAddress* addr) {
DCHECK(is_producer());
DCHECK(_rpc_context == nullptr);
-PBackendService_Stub* stub =
state->exec_env()->brpc_stub_cache()->get_stub(*addr);
+std::shared_ptr stub(
+state->exec_env()->brpc_stub_cache()->get_stub(*addr));
+if (!stub) {
+LOG(WARNING) << "Get rpc stub failed, host=" << addr->hostname << ",
port=" << addr->port;
+return Status::InternalError("get rpc stub failed");
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6973: [Feature] Spark connector supports to specify fields to write
github-actions[bot] commented on pull request #6973: URL: https://github.com/apache/incubator-doris/pull/6973#issuecomment-955879796 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed issue #6961: [S3] Support path style endpoint
yangzhg closed issue #6961: URL: https://github.com/apache/incubator-doris/issues/6961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #6962: [S3] Support path style endpoint
yangzhg merged pull request #6962: URL: https://github.com/apache/incubator-doris/pull/6962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (2803029 -> e8cabff)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 2803029 [Feature] Support bitmap_and_not & bitmap_and_not_count (#6910) add e8cabff [S3] Support path style endpoint (#6962) No new revisions were added by this update. Summary of changes: be/src/exec/tablet_sink.cpp| 6 +- be/src/olap/delta_writer.cpp | 4 +- be/src/runtime/load_channel.cpp| 3 +- be/src/runtime/tablets_channel.cpp | 5 +- be/src/runtime/tablets_channel.h | 2 +- be/src/service/internal_service.cpp| 3 +- be/src/util/s3_util.cpp| 11 ++- be/test/util/s3_storage_backend_test.cpp | 17 +++-- .../load-data/s3-load-manual.md| 21 +- .../load-data/s3-load-manual.md| 21 +- fe/fe-core/pom.xml | 2 +- .../java/org/apache/doris/backup/BlobStorage.java | 1 - .../java/org/apache/doris/backup/Repository.java | 14 ++-- .../java/org/apache/doris/backup/S3Storage.java| 59 +++- .../org/apache/doris/common/util/BrokerUtil.java | 15 +++-- .../java/org/apache/doris/common/util/S3URI.java | 68 +++ .../doris/httpv2/rest/manager/NodeAction.java | 10 +-- .../org/apache/doris/backup/S3StorageTest.java | 10 +-- .../org/apache/doris/common/util/S3URITest.java| 38 ++- .../doris/planner/StreamLoadScanNodeTest.java | 78 -- fe/pom.xml | 2 +- gensrc/proto/internal_service.proto| 2 + 22 files changed, 288 insertions(+), 104 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #6943: Docker 1.4.1 Compile Environment, First Compile Description
yangzhg merged pull request #6943: URL: https://github.com/apache/incubator-doris/pull/6943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (e8cabff -> 80f61c8)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from e8cabff [S3] Support path style endpoint (#6962) add 80f61c8 Docker 1.4.1 Compile Environment, First Compile Description (#6943) No new revisions were added by this update. Summary of changes: docs/en/installing/compilation.md| 8 docs/zh-CN/installing/compilation.md | 8 2 files changed, 16 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #6917: [Function] add BE bitmap function bitmap_subset_in_range
xinyiZzz commented on a change in pull request #6917:
URL: https://github.com/apache/incubator-doris/pull/6917#discussion_r739927720
##
File path: be/src/util/bitmap_value.h
##
@@ -1451,6 +1451,26 @@ class BitmapValue {
}
}
+/**
+ * Return new set with specified range (not include the range_end)
+ */
+int64_t sub_range(const int64_t& range_start, const int64_t& range_end,
BitmapValue* ret_bitmap) {
+int64_t count = 0;
+for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
+if (*it < range_start) {
+continue;
+}
+if (*it < range_end) {
+ret_bitmap->add(*it);
+++count;
+} else {
+break;
+}
+}
+return count;
Review comment:
If range_start is greater than bitmap max, count will be equal to 0, and
finally an empty string will be output, whether to unified into "null"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #6947: [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch
yangzhg merged pull request #6947: URL: https://github.com/apache/incubator-doris/pull/6947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (80f61c8 -> db1c281)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 80f61c8 Docker 1.4.1 Compile Environment, First Compile Description (#6943) add db1c281 [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch (#6947) No new revisions were added by this update. Summary of changes: be/src/common/config.h | 4 +- be/src/olap/memtable.cpp | 5 +++ be/src/olap/memtable.h | 4 ++ be/src/runtime/load_channel.cpp| 2 +- be/src/runtime/tablets_channel.cpp | 48 ++ be/src/runtime/tablets_channel.h | 2 +- docs/en/best-practices/star-schema-benchmark.md| 12 +++--- docs/zh-CN/best-practices/star-schema-benchmark.md | 16 .../org/apache/doris/analysis/OutFileClause.java | 3 +- 9 files changed, 70 insertions(+), 26 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] Gabriel39 commented on a change in pull request #6973: [Feature] Spark connector supports to specify fields to write
Gabriel39 commented on a change in pull request #6973:
URL: https://github.com/apache/incubator-doris/pull/6973#discussion_r739928533
##
File path:
extension/spark-doris-connector/src/main/java/org/apache/doris/spark/rest/RestService.java
##
@@ -523,7 +470,13 @@ public static String randomBackend(SparkSettings
sparkSettings , Logger logger)
logger.error(SHOULD_NOT_HAPPEN_MESSAGE);
throw new ShouldNeverHappenException();
}
-List backendRows = backend.getRows().stream().filter(v ->
v.getAlive()).collect(Collectors.toList());
+List backendRows = backend.stream().map(array -> {
+BackendRow backendRow = new BackendRow();
+backendRow.setIP(array.get(2));
Review comment:
emm, have you updated code here? I think this change should be reverted
to current version. Hard code like `array.get(2)` is unreasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] qidaye commented on a change in pull request #6917: [Function] add BE bitmap function bitmap_subset_in_range
qidaye commented on a change in pull request #6917:
URL: https://github.com/apache/incubator-doris/pull/6917#discussion_r739930721
##
File path: be/src/util/bitmap_value.h
##
@@ -1451,6 +1451,26 @@ class BitmapValue {
}
}
+/**
+ * Return new set with specified range (not include the range_end)
+ */
+int64_t sub_range(const int64_t& range_start, const int64_t& range_end,
BitmapValue* ret_bitmap) {
+int64_t count = 0;
+for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
+if (*it < range_start) {
+continue;
+}
+if (*it < range_end) {
+ret_bitmap->add(*it);
+++count;
+} else {
+break;
+}
+}
+return count;
Review comment:
When there is a error, it returns `null`. The case `range_start` greater
than `bitmap_max` is a normal case, I think it's ok to return an empty result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6917: [Function] add BE bitmap function bitmap_subset_in_range
morningman merged pull request #6917: URL: https://github.com/apache/incubator-doris/pull/6917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (db1c281 -> 65ded82)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from db1c281 [Enhance][Load] Reduce the number of segments when loading a
large volume data in one batch (#6947)
add 65ded82 [Function] add BE bitmap function bitmap_subset_in_range
(#6917)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/bitmap_function.cpp | 19 +++
be/src/exprs/bitmap_function.h | 2 +
be/src/util/bitmap_value.h | 20 +++
be/test/exprs/bitmap_function_test.cpp | 61 ++
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js| 1 +
.../{bitmap_max.md => bitmap_subset_in_range.md} | 32 ++--
.../{bitmap_max.md => bitmap_subset_in_range.md} | 32 ++--
gensrc/script/doris_builtins_functions.py | 3 ++
9 files changed, 141 insertions(+), 30 deletions(-)
copy docs/en/sql-reference/sql-functions/bitmap-functions/{bitmap_max.md =>
bitmap_subset_in_range.md} (61%)
copy docs/zh-CN/sql-reference/sql-functions/bitmap-functions/{bitmap_max.md =>
bitmap_subset_in_range.md} (61%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #6916: Added bprc stub cache check and reset api
yangzhg commented on a change in pull request #6916:
URL: https://github.com/apache/incubator-doris/pull/6916#discussion_r739933439
##
File path: be/src/exec/tablet_sink.cpp
##
@@ -368,6 +370,10 @@ void NodeChannel::try_send_batch() {
if (row_batch->num_rows() > 0) {
SCOPED_ATOMIC_TIMER(&_serialize_batch_ns);
row_batch->serialize(request.mutable_row_batch());
+if (request.row_batch().ByteSizeLong() >=
double(config::brpc_max_body_size) * 0.95f) {
Review comment:
`row_batch()` only contains primitive types, so there is no problem of
recursive calls. When it fails because the data sent is too large, it will only
report reset by peer errors. It is difficult to determine the specific reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6899: [BUG] fix Calc capacityCoefficient mistake
github-actions[bot] commented on pull request #6899: URL: https://github.com/apache/incubator-doris/pull/6899#issuecomment-955896183 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
morningman commented on pull request #6971: URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-955897530 Hi @zuochunwei Could you please create an issue to describe more detail about how this works and how much memory can be reduced after this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #6917: [Function] add BE bitmap function bitmap_subset_in_range
xinyiZzz commented on a change in pull request #6917:
URL: https://github.com/apache/incubator-doris/pull/6917#discussion_r739937010
##
File path: be/test/exprs/bitmap_function_test.cpp
##
@@ -552,6 +552,67 @@ TEST_F(BitmapFunctionsTest, bitmap_max) {
ASSERT_EQ(BigIntVal(1024), result);
}
+TEST_F(BitmapFunctionsTest, bitmap_subset_in_range) {
+// null
+StringVal res = BitmapFunctions::bitmap_subset_in_range(ctx,
StringVal::null(), BigIntVal(1), BigIntVal(3));
+ASSERT_TRUE(res.is_null);
+
+// empty
+BitmapValue bitmap0;
+StringVal empty_str = convert_bitmap_to_string(ctx, bitmap0);
+res = BitmapFunctions::bitmap_subset_in_range(ctx, empty_str,
BigIntVal(1), BigIntVal(3));
+BigIntVal result = BitmapFunctions::bitmap_count(ctx, res);
+ASSERT_EQ(BigIntVal(0), result);
+
+// normal
+BitmapValue
bitmap1({0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,208,23,24,25,26,27,28,29,30,31,32,33,100,200,500});
+
+StringVal bitmap_src = convert_bitmap_to_string(ctx, bitmap1);
+res = BitmapFunctions::bitmap_subset_in_range(ctx, bitmap_src,
BigIntVal(30), BigIntVal(200));
+result = BitmapFunctions::bitmap_count(ctx, res);
+ASSERT_EQ(BigIntVal(5), result);
+
+res = BitmapFunctions::bitmap_subset_in_range(ctx, bitmap_src,
BigIntVal(0), BigIntVal(1));
+result = BitmapFunctions::bitmap_count(ctx, res);
+ASSERT_EQ(BigIntVal(1), result);
+
+res = BitmapFunctions::bitmap_subset_in_range(ctx, bitmap_src,
BigIntVal(11), BigIntVal(15));
+result = BitmapFunctions::bitmap_count(ctx, res);
+ASSERT_EQ(BigIntVal(4), result);
+
+res = BitmapFunctions::bitmap_subset_in_range(ctx, bitmap_src,
BigIntVal(11), DecimalV2Value::MAX_INT64);
Review comment:
It seems to be better to use BigIntVal(INT64_MAX) and
BigIntVal(INT64_MIN) here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on a change in pull request #6917: [Function] add BE bitmap function bitmap_subset_in_range
xinyiZzz commented on a change in pull request #6917:
URL: https://github.com/apache/incubator-doris/pull/6917#discussion_r739937151
##
File path: be/test/exprs/bitmap_function_test.cpp
##
@@ -552,6 +552,67 @@ TEST_F(BitmapFunctionsTest, bitmap_max) {
ASSERT_EQ(BigIntVal(1024), result);
}
+TEST_F(BitmapFunctionsTest, bitmap_subset_in_range) {
+// null
+StringVal res = BitmapFunctions::bitmap_subset_in_range(ctx,
StringVal::null(), BigIntVal(1), BigIntVal(3));
+ASSERT_TRUE(res.is_null);
+
+// empty
+BitmapValue bitmap0;
+StringVal empty_str = convert_bitmap_to_string(ctx, bitmap0);
+res = BitmapFunctions::bitmap_subset_in_range(ctx, empty_str,
BigIntVal(1), BigIntVal(3));
+BigIntVal result = BitmapFunctions::bitmap_count(ctx, res);
+ASSERT_EQ(BigIntVal(0), result);
+
+// normal
+BitmapValue
bitmap1({0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,208,23,24,25,26,27,28,29,30,31,32,33,100,200,500});
Review comment:
208,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Doc] Update fe-idea developer guide for latest version (#6963)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 210625b [Doc] Update fe-idea developer guide for latest version (#6963) 210625b is described below commit 210625b358490f9858a54ff2175ffe8e5cd30f16 Author: wei zhao AuthorDate: Mon Nov 1 11:42:13 2021 +0800 [Doc] Update fe-idea developer guide for latest version (#6963) --- docs/en/developer-guide/fe-idea-dev.md| 10 ++ docs/zh-CN/developer-guide/fe-idea-dev.md | 14 -- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/en/developer-guide/fe-idea-dev.md b/docs/en/developer-guide/fe-idea-dev.md index 1bd6113..fac8f7c 100644 --- a/docs/en/developer-guide/fe-idea-dev.md +++ b/docs/en/developer-guide/fe-idea-dev.md @@ -43,12 +43,12 @@ If your are only interested in FE module, and for some reason you can't or don't the minimum tool required for FE module is `thrift`, so you can manually install `thrift` and copy or create a link of the executable `thrift` command to `./thirdparty/installed/bin`. -Doris build against `thrift` 0.9.3, and `thrift` 0.9.3.1 should also work well, but the newer version will not. +Doris build against `thrift` 0.13.0 ( note : `Doris` 0.15 and later version build against `thrift` 0.13.0 , the previous version is still `thrift` 0.9.3) -If your are using macOS, try `brew install thrift@0.9` and will get `thrift` 0.9.3.1 installed at `/usr/local/opt/thrift@0.9/bin/thrift`, +If your are using macOS, try `brew install thrift@0.13.0` and will get `thrift` 0.13.0 installed at `/usr/local/opt/thrift@0.13.0/bin/thrift`, then create a soft link to `./thirdparty/installed/bin/thrift`. -For Windows users, download `thrift` 0.9.3 from `http://archive.apache.org/dist/thrift/0.9.3/thrift-0.9.3.exe`, +For Windows users, download `thrift` 0.13.0 from `http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.exe`, and put it into `thirdparty/installed/bin/` folder. Go to `./fe` folder and run the following maven command to generate sources. @@ -74,10 +74,12 @@ is generate codes in Linux and copy them back. Using Docker should also be an op 1. Import `./fe` into IDEA -2. Follow the picture to create the folders and copy files under `webroot` into it. +2. Follow the picture to create the folders  +3. Build `ui` project , and copy files from directory `ui/dist` into directory `webroot` ( you can skip this step , if you don't need `Doris` UI ) + ## Custom FE configuration Copy below content into `conf/fe.conf` and tune it to fit your environment. diff --git a/docs/zh-CN/developer-guide/fe-idea-dev.md b/docs/zh-CN/developer-guide/fe-idea-dev.md index c51bbd7..9fae981 100644 --- a/docs/zh-CN/developer-guide/fe-idea-dev.md +++ b/docs/zh-CN/developer-guide/fe-idea-dev.md @@ -36,10 +36,10 @@ JDK1.8+, IntelliJ IDEA 3. 如果仅进行fe开发而没有编译过thirdparty，则需要安装thrift，并将thrift 复制或者连接到 `thirdparty/installed/bin` 目录下 -1. 安装 thrift 0.9.3 版本(注意：Doris 基于 thrift 0.9.3 构建, 使用新版本会导致编译不通过) - -MacOS: `brew install thrift@0.9` -Windows: `http://archive.apache.org/dist/thrift/0.9.3/thrift-0.9.3.exe` +安装 `thrift` 0.13.0 版本(注意：`Doris` 0.15 和最新的版本基于 `thrift` 0.13.0 构建, 之前的版本依然使用`thrift` 0.9.3 构建) + +MacOS: `brew install thrift@0.13.0` +Windows: `http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.exe` 4. 如果是Mac 或者 Linux 环境 可以通过 如下命令生成自动生成代码： @@ -62,12 +62,14 @@ JDK1.8+, IntelliJ IDEA ## 2.调试 -**1. 用idea导入fe工程；** +1. 用idea导入fe工程 -2.在fe目录下创建下面红框标出的目录，并将webroot里的内容拷贝进去 +2. 在fe目录下创建下面红框标出的目录  +3. 编译`ui`项目，将 `ui/dist/`目录中的文件拷贝到`webroot`中（如果你不需要看`Doris` UI，这一步可以跳过） + ## 3.配置conf/fe.conf 下面是我自己的配置，你可以根据自己的需要进行修改 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg merged pull request #6963: [Doc] Update fe-idea developer guide for latest version
yangzhg merged pull request #6963: URL: https://github.com/apache/incubator-doris/pull/6963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] zhangstar333 commented on pull request #6912: [Feature] add functions of bitmap_and/or_count
zhangstar333 commented on pull request #6912: URL: https://github.com/apache/incubator-doris/pull/6912#issuecomment-955903320 > LGTM, but please solve the conflict OK, have done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz opened a new pull request #6977: [Function] add BE bitmap function sub_bitmap
xinyiZzz opened a new pull request #6977: URL: https://github.com/apache/incubator-doris/pull/6977 ## Proposed changes Starting from the offset position, intercept the specified limit bitmap elements and return a bitmap subset. ## Types of changes What types of changes does your code introduce to Doris? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) - [ ] Code refactor (Modify the code structure, format the code, etc...) - [ ] Optimization. Including functional usability improvements and performance improvements. - [ ] Dependency. Such as changes related to third-party components. - [ ] Other. ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on (Fix #6875 ) and described the bug/feature there in detail - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If these changes need document changes, I have updated the document - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at d...@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch branch-0.15 updated (26a81f2 -> 4719771)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-0.15 in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 26a81f2 [Optimize] optimize some session variable and profile (#6920) new 36ae508 Modify Chinese comment (#6951) new a946895 [Docs] Fix error KEY url (#6955) new b58ae3e support use char like \x01 in flink-doris-sink column & line delimiter (#6937) new 1c8e962 Fix spark connector build error (#6948) new 162b9dc [Bug] Fix failure to stop sync job (#6950) new 5bec413 [S3] Support path style endpoint (#6962) new 4719771 [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch (#6947) The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/common/config.h | 4 +- be/src/exec/tablet_sink.cpp| 6 +- be/src/olap/delta_writer.cpp | 4 +- be/src/olap/memtable.cpp | 5 ++ be/src/olap/memtable.h | 4 ++ be/src/olap/olap_common.h | 19 ++--- be/src/olap/olap_cond.cpp | 38 +- be/src/olap/out_stream.cpp | 35 + be/src/olap/out_stream.h | 56 +++ be/src/olap/schema_change.cpp | 84 +++--- be/src/runtime/load_channel.cpp| 5 +- be/src/runtime/tablets_channel.cpp | 53 +++--- be/src/runtime/tablets_channel.h | 4 +- be/src/service/internal_service.cpp| 3 +- be/src/util/s3_util.cpp| 11 ++- be/test/util/s3_storage_backend_test.cpp | 17 +++-- .../load-data/s3-load-manual.md| 21 +- docs/en/best-practices/star-schema-benchmark.md| 12 ++-- docs/en/downloads/downloads.md | 2 +- docs/en/extending-doris/flink-doris-connector.md | 2 +- .../load-data/s3-load-manual.md| 21 +- docs/zh-CN/best-practices/star-schema-benchmark.md | 16 +++-- docs/zh-CN/downloads/downloads.md | 2 +- .../zh-CN/extending-doris/flink-doris-connector.md | 2 +- .../doris/flink/rest/models/RespContent.java | 4 ++ .../flink/table/DorisDynamicOutputFormat.java | 44 ++-- .../apache/doris/flink/table/DorisStreamLoad.java | 3 +- extension/spark-doris-connector/pom.xml| 2 + fe/fe-core/pom.xml | 2 +- .../org/apache/doris/analysis/OutFileClause.java | 3 +- .../java/org/apache/doris/backup/BlobStorage.java | 1 - .../java/org/apache/doris/backup/Repository.java | 14 ++-- .../java/org/apache/doris/backup/S3Storage.java| 59 ++- .../org/apache/doris/common/util/BrokerUtil.java | 15 ++-- .../java/org/apache/doris/common/util/S3URI.java | 68 ++ .../doris/httpv2/rest/manager/NodeAction.java | 10 +-- .../org/apache/doris/load/sync/SyncLifeCycle.java | 2 +- .../org/apache/doris/backup/S3StorageTest.java | 10 +-- .../org/apache/doris/common/util/S3URITest.java| 38 +- .../doris/planner/StreamLoadScanNodeTest.java | 78 ++-- fe/pom.xml | 2 +- gensrc/proto/internal_service.proto| 2 + 42 files changed, 526 insertions(+), 257 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 04/07: Fix spark connector build error (#6948)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 1c8e9629e82f78daf95308c241d56632ea4a7fa5
Author: jiafeng.zhang
AuthorDate: Fri Oct 29 14:59:05 2021 +0800
Fix spark connector build error (#6948)
pom.xml error
---
extension/spark-doris-connector/pom.xml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/extension/spark-doris-connector/pom.xml
b/extension/spark-doris-connector/pom.xml
index d8ebfe7..b76c823 100644
--- a/extension/spark-doris-connector/pom.xml
+++ b/extension/spark-doris-connector/pom.xml
@@ -150,6 +150,8 @@
${spark.version}
test
+
+
com.fasterxml.jackson.core
jackson-databind
2.10.0
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 03/07: support use char like \x01 in flink-doris-sink column & line delimiter (#6937)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b58ae3e02bf7ac0937cbfe50002a91abb8d8
Author: wunan1210
AuthorDate: Fri Oct 29 13:56:52 2021 +0800
support use char like \x01 in flink-doris-sink column & line delimiter
(#6937)
* support use char like \x01 in flink-doris-sink column & line delimiter
* extend imports
* add docs
---
docs/en/extending-doris/flink-doris-connector.md | 2 +-
.../zh-CN/extending-doris/flink-doris-connector.md | 2 +-
.../doris/flink/rest/models/RespContent.java | 4 ++
.../flink/table/DorisDynamicOutputFormat.java | 44 +++---
.../apache/doris/flink/table/DorisStreamLoad.java | 3 +-
5 files changed, 46 insertions(+), 9 deletions(-)
diff --git a/docs/en/extending-doris/flink-doris-connector.md
b/docs/en/extending-doris/flink-doris-connector.md
index c42d237..961da90 100644
--- a/docs/en/extending-doris/flink-doris-connector.md
+++ b/docs/en/extending-doris/flink-doris-connector.md
@@ -257,7 +257,7 @@ outputFormat.close();
| sink.batch.size| 100| Maximum number of
lines in a single write BE |
| sink.max-retries| 1| Number of retries
after writing BE failed |
| sink.batch.interval | 1s| The flush
interval, after which the asynchronous thread will write the data in the cache
to BE. The default value is 1 second, and the time units are ms, s, min, h, and
d. Set to 0 to turn off periodic writing. |
-| sink.properties.* | -- | The stream load
parameters.eg:sink.properties.column_separator' = ','. Support JSON
format import, you need to enable both 'sink.properties.format' ='json' and
'sink.properties.strip_outer_array' ='true'|
+| sink.properties.* | -- | The stream load
parameters.eg:sink.properties.column_separator' = ','. Setting
'sink.properties.escape_delimiters' = 'true' if you want to use a control char
as a separator, so that such as '\\x01' will translate to binary 0x01
Support JSON format import, you need to enable both 'sink.properties.format'
='json' and 'sink.properties.strip_outer_array' ='true'|
## Doris & Flink Column Type Mapping
diff --git a/docs/zh-CN/extending-doris/flink-doris-connector.md
b/docs/zh-CN/extending-doris/flink-doris-connector.md
index 9ea1eba..4a0a8cb 100644
--- a/docs/zh-CN/extending-doris/flink-doris-connector.md
+++ b/docs/zh-CN/extending-doris/flink-doris-connector.md
@@ -260,7 +260,7 @@ outputFormat.close();
| sink.batch.size | 100| 单次写BE的最大行数|
| sink.max-retries | 1| 写BE失败之后的重试次数 |
| sink.batch.interval | 1s| flush 间隔时间，超过该时间后异步线程将
缓存中数据写入BE。 默认值为1秒，支持时间单位ms、s、min、h和d。设置为0表示关闭定期写入。|
-| sink.properties.* | -- | Stream load
的导入参数。例如:'sink.properties.column_separator' = ','等。
支持JSON格式导入，需要同时开启'sink.properties.format' =
'json'和'sink.properties.strip_outer_array' = 'true' |
+| sink.properties.* | -- | Stream load
的导入参数。例如:'sink.properties.column_separator' = ','等。如果需要特殊字符作为分隔符,
可以加上参数'sink.properties.escape_delimiters' = 'true', '\\x01'会被转换为二进制的0x01
支持JSON格式导入，需要同时开启'sink.properties.format' =
'json'和'sink.properties.strip_outer_array' = 'true' |
diff --git
a/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/models/RespContent.java
b/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/models/RespContent.java
index b86b3dd..07a356c 100644
---
a/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/models/RespContent.java
+++
b/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/models/RespContent.java
@@ -93,4 +93,8 @@ public class RespContent {
}
}
+
+public String getErrorURL() {
+return ErrorURL;
+}
}
diff --git
a/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicOutputFormat.java
b/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicOutputFormat.java
index 0fd154a..f4f49bd 100644
---
a/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicOutputFormat.java
+++
b/extension/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisDynamicOutputFormat.java
@@ -38,11 +38,14 @@ import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
import java.util.StringJoiner;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
+impor
[incubator-doris] 01/07: Modify Chinese comment (#6951)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 36ae508076dee8daa75c0f548821c979a26bcdb6
Author: jiafeng.zhang
AuthorDate: Thu Oct 28 13:56:59 2021 +0800
Modify Chinese comment (#6951)
Modify Chinese comment
---
be/src/olap/olap_common.h | 19 +-
be/src/olap/olap_cond.cpp | 38 ++--
be/src/olap/out_stream.cpp| 35 +-
be/src/olap/out_stream.h | 56 ++---
be/src/olap/schema_change.cpp | 84 +--
5 files changed, 117 insertions(+), 115 deletions(-)
diff --git a/be/src/olap/olap_common.h b/be/src/olap/olap_common.h
index 0769870..7fe970e 100644
--- a/be/src/olap/olap_common.h
+++ b/be/src/olap/olap_common.h
@@ -146,10 +146,10 @@ enum FieldType {
OLAP_FIELD_TYPE_STRING = 26
};
-// 定义Field支持的所有聚集方法
-// 注意，实际中并非所有的类型都能使用以下所有的聚集方法
-// 例如对于string类型使用SUM就是毫无意义的(但不会导致程序崩溃)
-// Field类的实现并没有进行这类检查，应该在创建表的时候进行约束
+// Define all aggregation methods supported by Field
+// Note that in practice, not all types can use all the following aggregation
methods
+// For example, it is meaningless to use SUM for the string type (but it will
not cause the program to crash)
+// The implementation of the Field class does not perform such checks, and
should be constrained when creating the table
enum FieldAggregationMethod {
OLAP_FIELD_AGGREGATION_NONE = 0,
OLAP_FIELD_AGGREGATION_SUM = 1,
@@ -163,11 +163,14 @@ enum FieldAggregationMethod {
OLAP_FIELD_AGGREGATION_REPLACE_IF_NOT_NULL = 8,
};
-// 压缩算法类型
+// Compression algorithm type
enum OLAPCompressionType {
-OLAP_COMP_TRANSPORT = 1, // 用于网络传输的压缩算法，压缩率低，cpu开销低
-OLAP_COMP_STORAGE = 2, // 用于硬盘数据的压缩算法，压缩率高，cpu开销大
-OLAP_COMP_LZ4 = 3, // 用于储存的压缩算法，压缩率低，cpu开销低
+// Compression algorithm used for network transmission, low compression
rate, low cpu overhead
+OLAP_COMP_TRANSPORT = 1,
+// Compression algorithm used for hard disk data, with high compression
rate and high CPU overhead
+OLAP_COMP_STORAGE = 2,
+// The compression algorithm used for storage, the compression rate is
low, and the cpu overhead is low
+OLAP_COMP_LZ4 = 3,
};
enum PushType {
diff --git a/be/src/olap/olap_cond.cpp b/be/src/olap/olap_cond.cpp
index c9184c3..4bc6b51 100644
--- a/be/src/olap/olap_cond.cpp
+++ b/be/src/olap/olap_cond.cpp
@@ -35,25 +35,25 @@ using std::vector;
using doris::ColumnStatistics;
-//此文件主要用于对用户发送的查询条件和删除条件进行处理，逻辑上二者都可以分为三层
+//This file is mainly used to process query conditions and delete conditions
sent by users. Logically, both can be divided into three layers
//Condition->Condcolumn->Cond
-//Condition表示用户发的单个条件
-//Condcolumn表示一列上所有条件的集合。
-//Conds表示一列上的单个条件.
-//对于查询条件而言，各层级的条件之间都是逻辑与的关系
-//对于delete条件则有不同。Cond和Condcolumn之间是逻辑与的关系，而Condtion之间是逻辑或的关系。
-
-//具体到实现。
-//eval是用来过滤查询条件，包括堆row、block、version的过滤，具体使用哪一层看具体的调用地方。
-// 1. 没有单独过滤行的过滤条件，这部分在查询层进行。
-// 2. 过滤block在SegmentReader里面。
-// 3. 过滤version在Reader里面。调用delta_pruing_filter
+//Condition represents a single condition sent by the user
+//Condcolumn represents the collection of all conditions on a column.
+//Conds represents a single condition on a column.
+//For query conditions, the conditions of each level are logical AND
relationships
+//There are different conditions for delete. The relationship between Cond and
Condcolumn is logical AND, and the relationship between Condtion is logical OR.
+
+//Specific to the realization.
+//eval is used to filter query conditions, including the filtering of heap
row, block, and version. Which layer is used depends on the specific calling
place.
+// 1. There is no filter condition to filter rows separately, this part is
carried out in the query layer.
+// 2. The filter block is in the SegmentReader.
+// 3. Filter version in Reader. Call delta_pruing_filter
//
-//del_eval用来过滤删除条件，包括堆block和version的过滤，但是这个过滤比eval多了一个状态，即部分过滤。
-// 1. 对行的过滤在DeleteHandler。
-// 这部分直接调用delete_condition_eval实现,内部调用eval函数，因为对row的过滤不涉及部分过滤这种状态。
-// 2. 过滤block是在SegmentReader里面,直接调用del_eval
-// 3. 过滤version实在Reader里面,调用rowset_pruning_filter
+//del_eval is used to filter deletion conditions, including the filtering of
heap block and version, but this filtering has one more state than eval, that
is, partial filtering.
+// 1. The filtering of rows is in DeleteHandler.
+// This part directly calls delete_condition_eval to achieve, and internally
calls the eval function, because the filtering of row does not involve partial
filtering.
+// 2. The filter block is in the SegmentReader, call del_eval directly
+// 3. The filter version is actually in Reader, call rowset_pruning_filter
namespace doris {
@@ -176,7 +176,7 @@ OLAPStatus Cond::init(const TCondition& tcond, const
TabletColumn& column) {
bool Cond::eval(const RowCursorCell& cell) const {
[incubator-doris] 07/07: [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch (#6947)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 4719771be8049e5b17fdfc0caaa9b727a1a2a854
Author: Mingyu Chen
AuthorDate: Mon Nov 1 10:51:50 2021 +0800
[Enhance][Load] Reduce the number of segments when loading a large volume
data in one batch (#6947)
## Case
In the load process, each tablet will have a memtable to save the incoming
data,
and if the data in a memtable is larger than 100MB, it will be flushed to
disk as a `segment` file. And then
a new memtable will be created to save the following data/
Assume that this is a table with N buckets(tablets). So the max size of all
memtables will be `N * 100MB`.
If N is large, it will cost too much memory.
So for memory limit purpose, when the size of all memtables reach a
threshold(2GB as default), Doris will
try to flush all current memtables to disk(even if their size are not reach
100MB).
So you will see that the memtable will be flushed when it's size reach
`2GB/N`, which maybe much smaller
than 100MB, resulting in too many small segment files.
## Solution
When decide to flush memtable to reduce memory consumption, NOT to flush
all memtable, but to flush part
of them.
For example, there are 50 tablets(with 50 memtables). The memory limit is
1GB, so when each memtable reach
20MB, the total size reach 1GB, and flush will occur.
If I only flush 25 of 50 memtables, then next time when the total size
reach 1GB, there will be 25 memtables with
size 10MB, and other 25 memtables with size 30MB. So I can flush those
memtables with size 30MB, which is larger
than 20MB.
The main idea is to introduce some jitter during flush to ensure the small
unevenness of each memtable, so as to ensure that flush will only be triggered
when the memtable is large enough.
In my test, loading a table with 48 buckets, mem limit 2G, in previous
version, the average memtable size is 44MB,
after modification, the average size is 82MB
---
be/src/common/config.h | 4 +-
be/src/olap/memtable.cpp | 5 +++
be/src/olap/memtable.h | 4 ++
be/src/runtime/load_channel.cpp| 2 +-
be/src/runtime/tablets_channel.cpp | 48 ++
be/src/runtime/tablets_channel.h | 2 +-
docs/en/best-practices/star-schema-benchmark.md| 12 +++---
docs/zh-CN/best-practices/star-schema-benchmark.md | 16
.../org/apache/doris/analysis/OutFileClause.java | 3 +-
9 files changed, 70 insertions(+), 26 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 42c43fb..977282c 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -175,7 +175,7 @@ CONF_mInt32(doris_max_pushdown_conjuncts_return_rate, "90");
// (Advanced) Maximum size of per-query receive-side buffer
CONF_mInt32(exchg_node_buffer_size_bytes, "10485760");
// push_write_mbytes_per_sec
-CONF_mInt32(push_write_mbytes_per_sec, "10");
+CONF_mInt32(push_write_mbytes_per_sec, "100");
CONF_mInt64(column_dictionary_key_ratio_threshold, "0");
CONF_mInt64(column_dictionary_key_size_threshold, "0");
@@ -440,7 +440,7 @@ CONF_mInt64(memory_maintenance_sleep_time_s, "10");
CONF_Int32(memory_max_alignment, "16");
// write buffer size before flush
-CONF_mInt64(write_buffer_size, "104857600");
+CONF_mInt64(write_buffer_size, "209715200");
// following 2 configs limit the memory consumption of load process on a
Backend.
// eg: memory limit to 80% of mem limit config but up to 100GB(default)
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index 2c6069f..3ce5db1 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -61,6 +61,7 @@ int MemTable::RowCursorComparator::operator()(const char*
left, const char* righ
}
void MemTable::insert(const Tuple* tuple) {
+_rows++;
bool overwritten = false;
uint8_t* _tuple_buf = nullptr;
if (_keys_type == KeysType::DUP_KEYS) {
@@ -120,6 +121,8 @@ void MemTable::_aggregate_two_row(const ContiguousRow&
src_row, TableKey row_in_
}
OLAPStatus MemTable::flush() {
+VLOG_CRITICAL << "begin to flush memtable for tablet: " << _tablet_id
+ << ", memsize: " << memory_usage() << ", rows: " << _rows;
int64_t duration_ns = 0;
{
SCOPED_RAW_TIMER(&duration_ns);
@@ -141,6 +144,8 @@ OLAPStatus MemTable::flush() {
}
DorisMetrics::instance()->memtable_flush_total->increment(1);
DorisMetrics::instance()->memtable_flush_duration_us->increment(duration_ns /
1000);
+VLOG_CRITICAL << "after flush memtable for tablet: " << _tablet_id
+ << ", flushsize: " << _flush_size;
return OLAP_SUCCESS;
}
diff --git a/be/sr
[incubator-doris] 06/07: [S3] Support path style endpoint (#6962)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5bec4132ffbec0d038bfc40b47803d07b554107b
Author: Mingyu Chen
AuthorDate: Mon Nov 1 10:48:10 2021 +0800
[S3] Support path style endpoint (#6962)
Add a use_path_style property for S3
Upgrade hadoop-common and hadoop-aws to 2.8.0 to support path style property
Fix some S3 URI bugs
Add some logs for tracing load process.
---
be/src/exec/tablet_sink.cpp| 6 +-
be/src/olap/delta_writer.cpp | 4 +-
be/src/runtime/load_channel.cpp| 3 +-
be/src/runtime/tablets_channel.cpp | 5 +-
be/src/runtime/tablets_channel.h | 2 +-
be/src/service/internal_service.cpp| 3 +-
be/src/util/s3_util.cpp| 11 ++-
be/test/util/s3_storage_backend_test.cpp | 17 +++--
.../load-data/s3-load-manual.md| 21 +-
.../load-data/s3-load-manual.md| 21 +-
fe/fe-core/pom.xml | 2 +-
.../java/org/apache/doris/backup/BlobStorage.java | 1 -
.../java/org/apache/doris/backup/Repository.java | 14 ++--
.../java/org/apache/doris/backup/S3Storage.java| 59 +++-
.../org/apache/doris/common/util/BrokerUtil.java | 15 +++--
.../java/org/apache/doris/common/util/S3URI.java | 68 +++
.../doris/httpv2/rest/manager/NodeAction.java | 10 +--
.../org/apache/doris/backup/S3StorageTest.java | 10 +--
.../org/apache/doris/common/util/S3URITest.java| 38 ++-
.../doris/planner/StreamLoadScanNodeTest.java | 78 --
fe/pom.xml | 2 +-
gensrc/proto/internal_service.proto| 2 +
22 files changed, 288 insertions(+), 104 deletions(-)
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index c5c02d2..b818918 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -86,6 +86,7 @@ Status NodeChannel::init(RuntimeState* state) {
_cur_add_batch_request.set_allocated_id(&_parent->_load_id);
_cur_add_batch_request.set_index_id(_index_id);
_cur_add_batch_request.set_sender_id(_parent->_sender_id);
+_cur_add_batch_request.set_backend_id(_node_id);
_cur_add_batch_request.set_eos(false);
_rpc_timeout_ms = state->query_options().query_timeout * 1000;
@@ -93,7 +94,7 @@ Status NodeChannel::init(RuntimeState* state) {
_load_info = "load_id=" + print_id(_parent->_load_id) +
", txn_id=" + std::to_string(_parent->_txn_id);
-_name = "NodeChannel[" + std::to_string(_index_id) + "-" +
std::to_string(_node_id) + "]";
+_name = fmt::format("NodeChannel[{}-{}]", _index_id, _node_id);
return Status::OK();
}
@@ -282,7 +283,8 @@ Status NodeChannel::close_wait(RuntimeState* state) {
SleepFor(MonoDelta::FromMilliseconds(1));
}
timer.stop();
-VLOG_CRITICAL << name() << " close_wait cost: " << timer.elapsed_time() /
100 << " ms";
+VLOG_CRITICAL << name() << " close_wait cost: " << timer.elapsed_time() /
100 << " ms"
+ << ", " << _load_info;
if (_add_batches_finished) {
{
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index 3868e88..afa014b 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -321,7 +321,9 @@ OLAPStatus
DeltaWriter::close_wait(google::protobuf::RepeatedPtrFieldget_stats();
-VLOG_CRITICAL << "close delta writer for tablet: " << _tablet->tablet_id()
<< ", stats: " << stat;
+VLOG_CRITICAL << "close delta writer for tablet: " << _tablet->tablet_id()
+ << ", load id: " << print_id(_req.load_id)
+ << ", stats: " << stat;
return OLAP_SUCCESS;
}
diff --git a/be/src/runtime/load_channel.cpp b/be/src/runtime/load_channel.cpp
index 43f8a88..01e8d66 100644
--- a/be/src/runtime/load_channel.cpp
+++ b/be/src/runtime/load_channel.cpp
@@ -94,7 +94,8 @@ Status LoadChannel::add_batch(const
PTabletWriterAddBatchRequest& request,
Status st;
if (request.has_eos() && request.eos()) {
bool finished = false;
-RETURN_IF_ERROR(channel->close(request.sender_id(), &finished,
request.partition_ids(),
+RETURN_IF_ERROR(channel->close(request.sender_id(),
request.backend_id(),
+ &finished, request.partition_ids(),
tablet_vec));
if (finished) {
std::lock_guard l(_lock);
diff --git a/be/src/runtime/tablets_channel.cpp
b/be/src/runtime/tablets_channel.cpp
index 531a3bd..db5e23b 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -126,7 +126,7 @@ Status TabletsChanne
[incubator-doris] 02/07: [Docs] Fix error KEY url (#6955)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch branch-0.15 in repository https://gitbox.apache.org/repos/asf/incubator-doris.git commit a9468959cef0d7b62cd8fa94fc9a88c4b14884f3 Author: EmmyMiao87 <522274...@qq.com> AuthorDate: Fri Oct 29 12:07:44 2021 +0800 [Docs] Fix error KEY url (#6955) --- docs/en/downloads/downloads.md| 2 +- docs/zh-CN/downloads/downloads.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/downloads/downloads.md b/docs/en/downloads/downloads.md index 6ed6502..57faba5 100644 --- a/docs/en/downloads/downloads.md +++ b/docs/en/downloads/downloads.md @@ -37,6 +37,6 @@ You can download source code from following links, then compile and install Dori | 0.10.0 | 2019-07-02 | [Source](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz) ([Signature](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz.asc) [SHA512](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz.sha512)) | | 0.9.0 | 2019-02-18 | [Source](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz) ([Signature](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.asc) [SHA512](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.sha512)) | -To verify the downloaded files, please read [Verify Apache Release](../community/verify-apache-release.html) and using these [KEYS](https://www.apache.org/dist/incubator/doris/KEYS). +To verify the downloaded files, please read [Verify Apache Release](../community/verify-apache-release.html) and using these [KEYS](https://downloads.apache.org/incubator/doris/KEYS). After verification, please read [Compilation](../installing/compilation.html) and [Installation and deployment](../installing/install-deploy.html) to compile and install Doris. diff --git a/docs/zh-CN/downloads/downloads.md b/docs/zh-CN/downloads/downloads.md index 6f7e141..0bc8291 100644 --- a/docs/zh-CN/downloads/downloads.md +++ b/docs/zh-CN/downloads/downloads.md @@ -18,6 +18,6 @@ | 0.10.0 | 2019-07-02 | [Source](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz) ([Signature](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz.asc) [SHA512](http://archive.apache.org/dist/incubator/doris/0.10.0-incubating/apache-doris-0.10.0-incubating-src.tar.gz.sha512)) | | 0.9.0 | 2019-02-18 | [Source](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz) ([Signature](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.asc) [SHA512](http://archive.apache.org/dist/incubator/doris/0.9.0-incubating/apache-doris-0.9.0-incubating-src.tar.gz.sha512)) | -关于如何校验下载文件，请参阅 [校验下载文件](../community/verify-apache-release.html)，并使用这些[KEYS](https://www.apache.org/dist/incubator/doris/KEYS)。 +关于如何校验下载文件，请参阅 [校验下载文件](../community/verify-apache-release.html)，并使用这些[KEYS](https://downloads.apache.org/incubator/doris/KEYS)。 校验完成后，可以参阅 [编译文档](../installing/compilation.html) 以及 [安装与部署文档](../installing/install-deploy.html) 进行 Doris 的编译、安装与部署。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 05/07: [Bug] Fix failure to stop sync job (#6950)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 162b9dc829b6ed4217868d5f6086868d9dc22b32
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Sat Oct 30 18:17:15 2021 +0800
[Bug] Fix failure to stop sync job (#6950)
---
fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncLifeCycle.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncLifeCycle.java
b/fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncLifeCycle.java
index 8d98c14..227f175 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncLifeCycle.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncLifeCycle.java
@@ -55,7 +55,7 @@ public abstract class SyncLifeCycle {
}
public void stop() {
-if (isStart()) {
+if (!isStart()) {
// Repeated stops are considered successful
return;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch tmp created (now 210625b)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch tmp in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. at 210625b [Doc] Update fe-idea developer guide for latest version (#6963) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6918: [Function] add bitmap function of bitmap_has_all
morningman merged pull request #6918: URL: https://github.com/apache/incubator-doris/pull/6918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (210625b -> c7a3116)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 210625b [Doc] Update fe-idea developer guide for latest version
(#6963)
add c7a3116 [Function] add bitmap function of bitmap_has_all (#6918)
No new revisions were added by this update.
Summary of changes:
be/src/exprs/bitmap_function.cpp | 18 +++
be/src/exprs/bitmap_function.h | 2 ++
be/test/exprs/bitmap_function_test.cpp | 35 ++
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js| 1 +
.../{bitmap_and.md => bitmap_has_all.md} | 19 ++--
.../{bitmap_contains.md => bitmap_has_all.md} | 19 ++--
gensrc/script/doris_builtins_functions.py | 3 ++
8 files changed, 80 insertions(+), 18 deletions(-)
copy docs/en/sql-reference/sql-functions/bitmap-functions/{bitmap_and.md =>
bitmap_has_all.md} (68%)
copy
docs/zh-CN/sql-reference/sql-functions/bitmap-functions/{bitmap_contains.md =>
bitmap_has_all.md} (68%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6912: [Feature] add functions of bitmap_and/or_count
morningman merged pull request #6912: URL: https://github.com/apache/incubator-doris/pull/6912 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Function] add functions of bitmap_and/or_count (#6912)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1ff3d70 [Function] add functions of bitmap_and/or_count (#6912)
1ff3d70 is described below
commit 1ff3d708cab1b9e839be002e0b870badf5e99ec5
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Mon Nov 1 14:00:07 2021 +0800
[Function] add functions of bitmap_and/or_count (#6912)
issue #6875
add bitmap_and_count/ bitmap_or_count
---
be/src/exprs/bitmap_function.cpp | 9
be/src/exprs/bitmap_function.h | 3 ++
be/test/exprs/bitmap_function_test.cpp | 52 ++
docs/.vuepress/sidebar/en.js | 2 +
docs/.vuepress/sidebar/zh-CN.js| 2 +
.../bitmap-functions/bitmap_and_count.md | 63 ++
.../bitmap-functions/bitmap_or_count.md| 63 ++
.../bitmap-functions/bitmap_and_count.md | 63 ++
.../bitmap-functions/bitmap_or_count.md| 63 ++
gensrc/script/doris_builtins_functions.py | 8 ++-
10 files changed, 327 insertions(+), 1 deletion(-)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index f257664..fe70a48 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -489,6 +489,15 @@ StringVal BitmapFunctions::bitmap_and(FunctionContext*
ctx, const StringVal& lhs
}
return serialize(ctx, &bitmap);
}
+BigIntVal BitmapFunctions::bitmap_and_count(FunctionContext* ctx, const
StringVal& lhs,
+const StringVal& rhs) {
+return bitmap_count(ctx, bitmap_and(ctx, lhs, rhs));
+}
+
+BigIntVal BitmapFunctions::bitmap_or_count(FunctionContext* ctx, const
StringVal& lhs,
+ const StringVal& rhs) {
+return bitmap_count(ctx, bitmap_or(ctx, lhs, rhs));
+}
StringVal BitmapFunctions::bitmap_xor(FunctionContext* ctx, const StringVal&
lhs,
const StringVal& rhs) {
diff --git a/be/src/exprs/bitmap_function.h b/be/src/exprs/bitmap_function.h
index 9c4f155..9143a51 100644
--- a/be/src/exprs/bitmap_function.h
+++ b/be/src/exprs/bitmap_function.h
@@ -59,6 +59,9 @@ public:
const StringVal& dst);
static BigIntVal bitmap_min(FunctionContext* ctx, const StringVal& str);
+static BigIntVal bitmap_and_count(FunctionContext* ctx, const StringVal&
lhs, const StringVal& rhs);
+static BigIntVal bitmap_or_count(FunctionContext* ctx, const StringVal&
lhs, const StringVal& rhs);
+
static StringVal bitmap_serialize(FunctionContext* ctx, const StringVal&
src);
static StringVal to_bitmap(FunctionContext* ctx, const StringVal& src);
static StringVal bitmap_hash(FunctionContext* ctx, const StringVal& src);
diff --git a/be/test/exprs/bitmap_function_test.cpp
b/be/test/exprs/bitmap_function_test.cpp
index f734e9f..686d2df 100644
--- a/be/test/exprs/bitmap_function_test.cpp
+++ b/be/test/exprs/bitmap_function_test.cpp
@@ -347,6 +347,58 @@ TEST_F(BitmapFunctionsTest, bitmap_and) {
ASSERT_EQ(expected, result);
}
+TEST_F(BitmapFunctionsTest, bitmap_and_count) {
+BitmapValue bitmap1({0, 1, 2});
+BitmapValue bitmap2;
+StringVal bitmap_src1 = convert_bitmap_to_string(ctx, bitmap1);
+StringVal bitmap_src2 = convert_bitmap_to_string(ctx, bitmap2);
+BigIntVal result = BitmapFunctions::bitmap_and_count(ctx, bitmap_src1,
bitmap_src2);
+ASSERT_EQ(BigIntVal(0), result);
+
+result = BitmapFunctions::bitmap_and_count(ctx, bitmap_src1,
StringVal::null());
+ASSERT_EQ(BigIntVal(0), result);
+
+bitmap1 = BitmapValue({0, 1, 2,std::numeric_limits::min()});
+bitmap2 = BitmapValue({0, 1, 2,std::numeric_limits::max()});
+bitmap_src1 = convert_bitmap_to_string(ctx, bitmap1);
+bitmap_src2 = convert_bitmap_to_string(ctx, bitmap2);
+result = BitmapFunctions::bitmap_and_count(ctx, bitmap_src1, bitmap_src2);
+ASSERT_EQ(BigIntVal(3), result);
+
+bitmap1 = BitmapValue({1, 2, 3});
+bitmap2 = BitmapValue({3, 4, 5});
+bitmap_src1 = convert_bitmap_to_string(ctx, bitmap1);
+bitmap_src2 = convert_bitmap_to_string(ctx, bitmap2);
+result = BitmapFunctions::bitmap_and_count(ctx, bitmap_src1, bitmap_src2);
+ASSERT_EQ(BigIntVal(1), result);
+}
+
+TEST_F(BitmapFunctionsTest, bitmap_or_count) {
+BitmapValue bitmap1({0, 1, 2});
+BitmapValue bitmap2;
+StringVal bitmap_src1 = convert_bitmap_to_string(ctx, bitmap1);
+StringVal bitmap_src2 = convert_bitmap_to_string(ctx, bitmap2);
+BigIntVal result = BitmapFunctions::bitmap_or_count(ctx, bitmap_src1,
bitmap_src2);
+AS
