[GitHub] [incubator-doris] kangpinghuang commented on issue #2233: fix schema change validate failure #2230
kangpinghuang commented on issue #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233#issuecomment-555384058 related issue # 2233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new pull request #2233: fix schema change validate failure #2230
kangpinghuang opened a new pull request #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233 fix schema change failure because of row number check failed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin opened a new pull request #2234: [gutil] Fix the line breaks to '\n'
lingbin opened a new pull request #2234: [gutil] Fix the line breaks to '\n' URL: https://github.com/apache/incubator-doris/pull/2234 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (23078a5 -> 4984be9)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 23078a5 Avoid Chinese garbled for streamLoad java sample (#2229) add 4984be9 Persist sqlmode in load metadata and add sqlmode to forward master request (#2216) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/catalog/View.java | 4 ++- .../java/org/apache/doris/common/FeConstants.java | 2 +- .../org/apache/doris/common/FeMetaVersion.java | 2 ++ .../apache/doris/load/loadv2/BrokerLoadJob.java| 36 +- .../doris/load/routineload/RoutineLoadJob.java | 35 - .../java/org/apache/doris/qe/ConnectProcessor.java | 1 + .../java/org/apache/doris/qe/MasterOpExecutor.java | 1 + .../java/org/apache/doris/qe/SqlModeHelper.java| 3 ++ .../java/org/apache/doris/qe/StmtExecutor.java | 2 +- fe/src/main/jflex/sql_scanner.flex | 31 +++ gensrc/thrift/FrontendService.thrift | 1 + 11 files changed, 113 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2216: Persist sqlmode in load metadata and add sqlmode to forward master request
morningman merged pull request #2216: Persist sqlmode in load metadata and add sqlmode to forward master request URL: https://github.com/apache/incubator-doris/pull/2216 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken opened a new pull request #2235: Support setting properties for storage_root_path
vagetablechicken opened a new pull request #2235: Support setting properties for storage_root_path URL: https://github.com/apache/incubator-doris/pull/2235 ref https://github.com/apache/incubator-doris/issues/2195 We can specify the properties of storage_root_path by setting ':', seperate by ',' e.g. `storage_root_path = /home/disk1/palo,medium:ssd,capacity:50` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2216: Persist sqlmode in load metadata and add sqlmode to forward master request
imay commented on a change in pull request #2216: Persist sqlmode in load
metadata and add sqlmode to forward master request
URL: https://github.com/apache/incubator-doris/pull/2216#discussion_r347803664
##
File path: gensrc/thrift/FrontendService.thrift
##
@@ -413,6 +413,7 @@ struct TMasterOpRequest {
8: optional string user_ip
9: optional string time_zone
10: optional i64 stmt_id
+11: required i64 sqlMode
Review comment:
New added field should be optional.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new pull request #2236: Change sqlmode 'required' to 'optional' in forward master request,
xy720 opened a new pull request #2236: Change sqlmode 'required' to 'optional' in forward master request, URL: https://github.com/apache/incubator-doris/pull/2236 Fix the previous pull request. This commit will change sqlMode param's modifier from 'required' to 'optional' in forward master request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Change sqlmode 'required' to 'optional' in forward master request, (#2236)
This is an automated email from the ASF dual-hosted git repository.
zhaoc pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new db8819d Change sqlmode 'required' to 'optional' in forward master
request, (#2236)
db8819d is described below
commit db8819d365a4bcb64d273a59dadc2ff47b6c6242
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Tue Nov 19 17:32:37 2019 +0800
Change sqlmode 'required' to 'optional' in forward master request, (#2236)
---
fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java | 4 +++-
gensrc/thrift/FrontendService.thrift | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
b/fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index 6362336..c2ea972 100644
--- a/fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -321,7 +321,6 @@ public class ConnectProcessor {
ctx.setQualifiedUser(request.user);
ctx.setCatalog(Catalog.getInstance());
ctx.getState().reset();
-ctx.getSessionVariable().setSqlMode(request.sqlMode);
if (request.isSetCluster()) {
ctx.setCluster(request.cluster);
}
@@ -343,6 +342,9 @@ public class ConnectProcessor {
if (request.isSetStmt_id()) {
ctx.setForwardedStmtId(request.getStmt_id());
}
+if (request.isSetSqlMode()) {
+ctx.getSessionVariable().setSqlMode(request.sqlMode);
+}
ctx.setThreadLocalInfo();
diff --git a/gensrc/thrift/FrontendService.thrift
b/gensrc/thrift/FrontendService.thrift
index 7fd6839..f42339c 100644
--- a/gensrc/thrift/FrontendService.thrift
+++ b/gensrc/thrift/FrontendService.thrift
@@ -413,7 +413,7 @@ struct TMasterOpRequest {
8: optional string user_ip
9: optional string time_zone
10: optional i64 stmt_id
-11: required i64 sqlMode
+11: optional i64 sqlMode
}
struct TColumnDefinition {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2236: Change sqlmode 'required' to 'optional' in forward master request,
imay merged pull request #2236: Change sqlmode 'required' to 'optional' in forward master request, URL: https://github.com/apache/incubator-doris/pull/2236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (db8819d -> 42a4fff)
This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from db8819d Change sqlmode 'required' to 'optional' in forward master request, (#2236) add 42a4fff Replace boost canonicalize (#2209) No new revisions were added by this update. Summary of changes: be/src/http/download_action.cpp | 31 --- be/src/olap/snapshot_manager.cpp | 32 +--- be/src/tools/meta_tool.cpp | 17 - be/src/util/file_utils.cpp | 4 be/src/util/file_utils.h | 8 5 files changed, 45 insertions(+), 47 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2209: Change replace boost canonicalize
imay merged pull request #2209: Change replace boost canonicalize URL: https://github.com/apache/incubator-doris/pull/2209 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2223: Improve to_bitmap parse int performance
imay commented on a change in pull request #2223: Improve to_bitmap parse int
performance
URL: https://github.com/apache/incubator-doris/pull/2223#discussion_r347828967
##
File path: be/test/util/string_parser_test.cpp
##
@@ -209,13 +224,59 @@ TEST(StringToInt, Limit) {
test_int_value("2147483647", 2147483647,
StringParser::PARSE_SUCCESS);
test_int_value("-2147483648", -2147483648,
StringParser::PARSE_SUCCESS);
test_int_value(
-"9223372036854775807",
-std::numeric_limits::max(),
-StringParser::PARSE_SUCCESS);
+"9223372036854775807",
+std::numeric_limits::max(),
+StringParser::PARSE_SUCCESS);
test_int_value(
-"-9223372036854775808",
-std::numeric_limits::min(),
-StringParser::PARSE_SUCCESS);
+"-9223372036854775808",
+std::numeric_limits::min(),
+StringParser::PARSE_SUCCESS);
+}
+
+TEST(StringToUnsignedInt, Basic) {
+test_unsigned_int_value("123", 123, StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("123", 123, StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("123", 123, StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("123", 123, StringParser::PARSE_SUCCESS);
+
+test_unsigned_int_value("123", 123, StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("12345", 12345,
StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("12345678", 12345678,
StringParser::PARSE_SUCCESS);
+test_unsigned_int_value("12345678901234", 12345678901234,
StringParser::PARSE_SUCCESS);
+
+test_unsigned_int_value("-10", 0, StringParser::PARSE_FAILURE);
+test_unsigned_int_value("-10", 0, StringParser::PARSE_FAILURE);
+test_unsigned_int_value("-10", 0, StringParser::PARSE_FAILURE);
+test_unsigned_int_value("-10", 0, StringParser::PARSE_FAILURE);
+
+test_unsigned_int_value("+1", 0, StringParser::PARSE_FAILURE);
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang closed pull request #2233: fix schema change validate failure #2230
kangpinghuang closed pull request #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Improve to_bitmap parse int performance (#2223)
This is an automated email from the ASF dual-hosted git repository.
zhaoc pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 14769b0 Improve to_bitmap parse int performance (#2223)
14769b0 is described below
commit 14769b0beb7671626da011611d2c73ffe841b831
Author: kangkaisen
AuthorDate: Tue Nov 19 18:00:19 2019 +0800
Improve to_bitmap parse int performance (#2223)
---
be/src/exprs/bitmap_function.cpp | 24 ---
be/src/util/string_parser.hpp | 68 +++
be/test/exprs/bitmap_function_test.cpp | 4 +-
be/test/util/CMakeLists.txt| 1 +
be/test/util/string_parser_test.cpp| 73 +++---
run-ut.sh | 1 +
6 files changed, 146 insertions(+), 25 deletions(-)
diff --git a/be/src/exprs/bitmap_function.cpp b/be/src/exprs/bitmap_function.cpp
index 107362a..2f1942a 100644
--- a/be/src/exprs/bitmap_function.cpp
+++ b/be/src/exprs/bitmap_function.cpp
@@ -19,6 +19,7 @@
#include "exprs/anyval_util.h"
#include "util/bitmap.h"
+#include "util/string_parser.hpp"
namespace doris {
void BitmapFunctions::init() {
@@ -79,27 +80,16 @@ BigIntVal BitmapFunctions::bitmap_count(FunctionContext*
ctx, const StringVal& s
StringVal BitmapFunctions::to_bitmap(doris_udf::FunctionContext* ctx, const
doris_udf::StringVal& src) {
std::unique_ptr bitmap {new RoaringBitmap()};
if (!src.is_null) {
-std::string tmp_str = std::string(reinterpret_cast(src.ptr),
src.len) ;
-unsigned long uint32_value = 0;
-try {
-uint32_value = std::stoul(tmp_str);
-// the std::stoul result type is unsigned long, not uint32_t. so
we need check it
-if(UNLIKELY(uint32_value > std::numeric_limits::max())) {
-throw std::out_of_range("");
-}
-} catch (std::invalid_argument& e) {
+StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+uint32_t int_value =
StringParser::string_to_unsigned_int(reinterpret_cast(src.ptr),
src.len, &parse_result);
+if (UNLIKELY(parse_result != StringParser::PARSE_SUCCESS)) {
std::stringstream error_msg;
-error_msg << "The to_bitmap function argument: " << tmp_str << "
type isn't integer family";
-ctx->set_error(error_msg.str().c_str());
-return StringVal::null();
-} catch (std::out_of_range& e) {
-std::stringstream error_msg;
-error_msg << "The to_bitmap function argument: " << tmp_str << "
exceed unsigned integer max value "
- << std::numeric_limits::max();
+error_msg << "The to_bitmap function argument: " <<
std::string(reinterpret_cast(src.ptr), src.len)
+<< " type isn't integer family or exceed unsigned integer max
value 4294967295";
ctx->set_error(error_msg.str().c_str());
return StringVal::null();
}
-bitmap->update(uint32_value);
+bitmap->update(int_value);
}
std::string buf;
buf.resize(bitmap->size());
diff --git a/be/src/util/string_parser.hpp b/be/src/util/string_parser.hpp
index 3a0aa73..0354343 100644
--- a/be/src/util/string_parser.hpp
+++ b/be/src/util/string_parser.hpp
@@ -86,6 +86,20 @@ public:
return string_to_int_internal(s + i, len - i, result);
}
+// This is considerably faster than glibc's implementation.
+// In the case of overflow, the max/min value for the data type will be
returned.
+// Assumes s represents a decimal number.
+template
+static inline T string_to_unsigned_int(const char* s, int len,
ParseResult* result) {
+T ans = string_to_unsigned_int_internal(s, len, result);
+if (LIKELY(*result == PARSE_SUCCESS)){
+return ans;
+}
+
+int i = skip_leading_whitespace(s, len);
+return string_to_unsigned_int_internal(s + i, len - i, result);
+}
+
// Convert a string s representing a number in given base into a decimal
number.
template
static inline T string_to_int(const char* s, int len, int base,
ParseResult* result) {
@@ -159,6 +173,13 @@ private:
template
static inline T string_to_int_internal(const char* s, int len,
ParseResult* result);
+// This is considerably faster than glibc's implementation.
+// In the case of overflow, the max/min value for the data type will be
returned.
+// Assumes s represents a decimal number.
+// Return PARSE_FAILURE on leading whitespace. Trailing whitespace is
allowed.
+template
+static inline T string_to_unsigned_int_internal(const char* s, int len,
ParseResult* result);
+
// Convert a string s representing a number in given base into a decimal
number.
// Return PARSE_FAILURE on leading whitespace. Trailing whitespace
[GitHub] [incubator-doris] imay merged pull request #2223: Improve to_bitmap parse int performance
imay merged pull request #2223: Improve to_bitmap parse int performance URL: https://github.com/apache/incubator-doris/pull/2223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new pull request #2233: fix schema change validate failure #2230
kangpinghuang opened a new pull request #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233 fix schema change failure because of row number check failed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347832159
##
File path: be/src/olap/options.h
##
@@ -26,11 +26,14 @@
namespace doris {
struct StorePath {
-StorePath() : capacity_bytes(-1) { }
+StorePath() : capacity_bytes(-1), storage_medium(TStorageMedium::HDD){ }
StorePath(const std::string& path_, int64_t capacity_bytes_)
: path(path_), capacity_bytes(capacity_bytes_) { }
Review comment:
missing medium's default value?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347833986
##
File path: be/src/olap/options.cpp
##
@@ -47,18 +47,63 @@ OLAPStatus parse_conf_store_paths(
return OLAP_ERR_INPUT_PARAMETER_ERROR;
}
-// parse root path capacity
+// parse root path capacity and storage medium
+std::string capacity_str, medium_str;
+
+boost::filesystem::path boost_path = tmp_vec[0];
+std::string extension =
boost::filesystem::canonical(boost_path).extension().string();
+if (!extension.empty()) {
+medium_str = extension.substr(1);
+}
+
+for (int i = 1; i < tmp_vec.size(); i++) {
+// : or
+std::string property, value;
+std::size_t found = tmp_vec[i].find(':');
+if (found != std::string::npos) {
+property = boost::trim_copy(tmp_vec[i].substr(0, found));
+value = boost::trim_copy(tmp_vec[i].substr(found + 1));
+LOG(INFO) << "get setting: "<< property <<": "< only supports setting capacity
+property = "capacity";
+value = boost::trim_copy(tmp_vec[i]);
+}
+
+if (property == "capacity") {
Review comment:
better to use ignore case compare
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347831905
##
File path: be/src/olap/options.h
##
@@ -26,11 +26,14 @@
namespace doris {
struct StorePath {
-StorePath() : capacity_bytes(-1) { }
+StorePath() : capacity_bytes(-1), storage_medium(TStorageMedium::HDD){ }
Review comment:
```suggestion
StorePath() : capacity_bytes(-1), storage_medium(TStorageMedium::HDD) { }
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties for storage_root_path URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347833307 ## File path: be/src/olap/options.cpp ## @@ -47,18 +47,63 @@ OLAPStatus parse_conf_store_paths( return OLAP_ERR_INPUT_PARAMETER_ERROR; } -// parse root path capacity +// parse root path capacity and storage medium +std::string capacity_str, medium_str; Review comment: Can you add some unit test for this code? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347834348
##
File path: be/src/olap/options.cpp
##
@@ -47,18 +47,63 @@ OLAPStatus parse_conf_store_paths(
return OLAP_ERR_INPUT_PARAMETER_ERROR;
}
-// parse root path capacity
+// parse root path capacity and storage medium
+std::string capacity_str, medium_str;
+
+boost::filesystem::path boost_path = tmp_vec[0];
+std::string extension =
boost::filesystem::canonical(boost_path).extension().string();
+if (!extension.empty()) {
+medium_str = extension.substr(1);
+}
+
+for (int i = 1; i < tmp_vec.size(); i++) {
+// : or
+std::string property, value;
+std::size_t found = tmp_vec[i].find(':');
+if (found != std::string::npos) {
+property = boost::trim_copy(tmp_vec[i].substr(0, found));
+value = boost::trim_copy(tmp_vec[i].substr(found + 1));
+LOG(INFO) << "get setting: "<< property <<": "<
[GitHub] [incubator-doris] vagetablechicken commented on issue #2235: Support setting properties for storage_root_path
vagetablechicken commented on issue #2235: Support setting properties for
storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#issuecomment-555438703
property medium has a higher priority, so I decide to set storage_medium in
c'tor of DataDir, change
'_init_extension_and_capacity()' to '_init_capacity()'.
https://github.com/apache/incubator-doris/blob/14769b0beb7671626da011611d2c73ffe841b831/be/src/olap/data_dir.h#L132
But in meta_tool,
https://github.com/apache/incubator-doris/blob/14769b0beb7671626da011611d2c73ffe841b831/be/src/tools/meta_tool.cpp#L143
var 'data_dir' will not be set the _storage_medium, cause I remove the
_init_extension part.
I don't want to redefine TStorageMedium enum, e.g. {UNSET, HDD,SSD}.
How about
```
DataDir(const std::string& path,
int64_t capacity_bytes = -1,
std::string medium_property = "",
TabletManager* tablet_manager = nullptr,
TxnManager* txn_manager = nullptr);
```
Thus,
1. medium_property != "", set _storage_medium
1. medium_property == "", use extension
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on a change in pull request #2235: Support setting properties for storage_root_path
vagetablechicken commented on a change in pull request #2235: Support setting
properties for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347844509
##
File path: be/src/olap/options.h
##
@@ -26,11 +26,14 @@
namespace doris {
struct StorePath {
-StorePath() : capacity_bytes(-1) { }
+StorePath() : capacity_bytes(-1), storage_medium(TStorageMedium::HDD){ }
Review comment:
How do you format the code? I can't find .clang-format, is it a predefined
style?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2235: Support setting properties for storage_root_path
imay commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347849961
##
File path: be/src/olap/options.h
##
@@ -26,11 +26,14 @@
namespace doris {
struct StorePath {
-StorePath() : capacity_bytes(-1) { }
+StorePath() : capacity_bytes(-1), storage_medium(TStorageMedium::HDD){ }
Review comment:
The code in Doris comes from many different projects. Generally, the style
of each file is consistent, if you add code to a file, you should keep style
with it. If it is a new file, Baidu's code style is used. It's about the same
style as Google's. Except lowercase and underscore as the function name, and
use 4-byte indentation and so on.
clang-format is a good suggestion, we should introduce it to our project to
make developers easier.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #2235: Support setting properties for storage_root_path
imay commented on issue #2235: Support setting properties for storage_root_path URL: https://github.com/apache/incubator-doris/pull/2235#issuecomment-555449216 @vagetablechicken 1. I think meta tool should not care about the medium of data path. 2. Even if meta tool care about medium, it is better to make medium and capacity parse in one place. You can make config parse a function, which can be called by both options and meta dir. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (14769b0 -> 03a5359)
This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 14769b0 Improve to_bitmap parse int performance (#2223) add 03a5359 Fix the line breaks to '\n' (#2234) No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2234: [gutil] Fix the line breaks to '\n'
imay merged pull request #2234: [gutil] Fix the line breaks to '\n' URL: https://github.com/apache/incubator-doris/pull/2234 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 Pure DocValue optimization for doris-on-es This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin opened a new pull request #2238: Add gutil split and strip tool
lingbin opened a new pull request #2238: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/pull/2238 With these two tools, we can very easily perform splitting and trimming operations on strings. The subsequent PR will use them to replace the existing `boost::split()` and `boost::trim()` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347861228
##
File path: be/src/olap/rowset/segment_v2/rle_page.h
##
@@ -86,14 +86,12 @@ class RlePageBuilder : public PageBuilder {
Status add(const uint8_t* vals, size_t* count) override {
DCHECK(!_finished);
-DCHECK_EQ(reinterpret_cast(vals) & (alignof(CppType) - 1),
0)
-<< "Pointer passed to Add() must be naturally-aligned";
-
-const CppType* new_vals = reinterpret_cast(vals);
+auto new_vals = reinterpret_cast(vals);
for (int i = 0; i < *count; ++i) {
-_rle_encoder->Put(new_vals[i]);
+CppType value;
+memcpy(&value, &new_vals[i], SIZE_OF_TYPE);
Review comment:
Better to add comment about why we use memcpy here.
And I think should add a TODO here to make input vals aligned later for
better performace.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347872264
##
File path: be/src/olap/rowset/segment_v2/encoding_info.cpp
##
@@ -50,7 +55,7 @@ struct TypeEncodingTraits {
};
template
-struct TypeEncodingTraits {
+struct TypeEncodingTraits {
Review comment:
BINARY_PLAIN_ENCODING and PLAIN_ENCODING is not same.
For HLL or Bitmap type, BINARY_PLAIN_ENCODING is used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347862996
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_writer.h
##
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "gutil/macros.h"
+
+namespace doris {
+
+class TypeInfo;
+class WritableFile;
+
+namespace segment_v2 {
+
+class BitmapIndexWriter {
+public:
+static Status create(const TypeInfo* typeinfo,
std::unique_ptr* res);
+
+BitmapIndexWriter() = default;
+virtual ~BitmapIndexWriter() = default;
+
+virtual void add_values(const void* values, size_t count) = 0;
Review comment:
void is enough? or change it to Status later?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347861808
##
File path: gensrc/proto/segment_v2.proto
##
@@ -48,12 +48,12 @@ message MetadataPairPB {
enum EncodingTypePB {
UNKNOWN_ENCODING = 0;
DEFAULT_ENCODING = 1;
-PLAIN_ENCODING = 2; // for non-binary type
+PLAIN_ENCODING = 2;
PREFIX_ENCODING = 3;
RLE = 4;
DICT_ENCODING = 5;
BIT_SHUFFLE = 6;
-BINARY_PLAIN_ENCODING = 7;
Review comment:
Is it OK to remove BINARY_PLAIN_ENCODING?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2235: Support setting properties for storage_root_path
lingbin commented on a change in pull request #2235: Support setting properties
for storage_root_path
URL: https://github.com/apache/incubator-doris/pull/2235#discussion_r347880981
##
File path: be/src/olap/data_dir.cpp
##
@@ -55,12 +55,13 @@ namespace doris {
static const char* const kMtabPath = "/etc/mtab";
static const char* const kTestFilePath = "/.testfile";
-DataDir::DataDir(const std::string& path, int64_t capacity_bytes,
+DataDir::DataDir(const std::string& path, int64_t
capacity_bytes,TStorageMedium::type storage_medium,
Review comment:
a space after the comma
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347888331
##
File path: be/src/olap/rowset/segment_v2/rle_page.h
##
@@ -86,14 +86,12 @@ class RlePageBuilder : public PageBuilder {
Status add(const uint8_t* vals, size_t* count) override {
DCHECK(!_finished);
-DCHECK_EQ(reinterpret_cast(vals) & (alignof(CppType) - 1),
0)
-<< "Pointer passed to Add() must be naturally-aligned";
-
-const CppType* new_vals = reinterpret_cast(vals);
+auto new_vals = reinterpret_cast(vals);
for (int i = 0; i < *count; ++i) {
-_rle_encoder->Put(new_vals[i]);
+CppType value;
+memcpy(&value, &new_vals[i], SIZE_OF_TYPE);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347888331
##
File path: be/src/olap/rowset/segment_v2/rle_page.h
##
@@ -86,14 +86,12 @@ class RlePageBuilder : public PageBuilder {
Status add(const uint8_t* vals, size_t* count) override {
DCHECK(!_finished);
-DCHECK_EQ(reinterpret_cast(vals) & (alignof(CppType) - 1),
0)
-<< "Pointer passed to Add() must be naturally-aligned";
-
-const CppType* new_vals = reinterpret_cast(vals);
+auto new_vals = reinterpret_cast(vals);
for (int i = 0; i < *count; ++i) {
-_rle_encoder->Put(new_vals[i]);
+CppType value;
+memcpy(&value, &new_vals[i], SIZE_OF_TYPE);
Review comment:
ok, I'll add the TODO to PageBuilder
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang edited a comment on issue #2233: fix schema change validate failure #2230
kangpinghuang edited a comment on issue #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233#issuecomment-555384058 related issue #2233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang edited a comment on issue #2233: fix schema change validate failure #2230
kangpinghuang edited a comment on issue #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233#issuecomment-555384058 related issue #2230 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347894233
##
File path: gensrc/proto/segment_v2.proto
##
@@ -48,12 +48,12 @@ message MetadataPairPB {
enum EncodingTypePB {
UNKNOWN_ENCODING = 0;
DEFAULT_ENCODING = 1;
-PLAIN_ENCODING = 2; // for non-binary type
+PLAIN_ENCODING = 2;
PREFIX_ENCODING = 3;
RLE = 4;
DICT_ENCODING = 5;
BIT_SHUFFLE = 6;
-BINARY_PLAIN_ENCODING = 7;
Review comment:
Yes, we could reuse PLAIN_ENCODING for BinaryPlainPage, see
https://github.com/apache/incubator-doris/pull/2050/files#diff-7c9224046521dc23a17299f3b1f4a849R60
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347895644
##
File path: be/src/olap/rowset/segment_v2/bitmap_index_writer.h
##
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "gutil/macros.h"
+
+namespace doris {
+
+class TypeInfo;
+class WritableFile;
+
+namespace segment_v2 {
+
+class BitmapIndexWriter {
+public:
+static Status create(const TypeInfo* typeinfo,
std::unique_ptr* res);
+
+BitmapIndexWriter() = default;
+virtual ~BitmapIndexWriter() = default;
+
+virtual void add_values(const void* values, size_t count) = 0;
Review comment:
I can't think of any error condition for now. And sure we can change it to
Status later when it's necessary.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin opened a new issue #2239: Add gutil split and strip tool
lingbin opened a new issue #2239: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/issues/2239 With these two tools, we can very easily perform splitting and trimming operations on strings. In these tools, StringPiece is used to split a string, which can avoid string copying, resulting in high performance than the boost library. we can also replace the existing boost::split() and boost::trim(). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on issue #2238: Add gutil split and strip tool
lingbin commented on issue #2238: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/pull/2238#issuecomment-555487504 #2239 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin closed pull request #2238: Add gutil split and strip tool
lingbin closed pull request #2238: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/pull/2238 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin opened a new pull request #2238: Add gutil split and strip tool
lingbin opened a new pull request #2238: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/pull/2238 With these two tools, we can very easily perform splitting and trimming operations on strings. The subsequent PR will use them to replace the existing `boost::split()` and `boost::trim()` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347898217
##
File path: be/src/olap/rowset/segment_v2/encoding_info.cpp
##
@@ -50,7 +55,7 @@ struct TypeEncodingTraits {
};
template
-struct TypeEncodingTraits {
+struct TypeEncodingTraits {
Review comment:
In fact there is no need to use different EncodingTypePB to instantiate
different PageBuilder for different data type. Partial template specification
is used here to create BinaryPlainPageBuilder for type using Slice as CppType.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new issue #2240: Fix some bugs about load label
morningman opened a new issue #2240: Fix some bugs about load label URL: https://github.com/apache/incubator-doris/issues/2240 1. `dbIdToTxnLabels` in `GlobalTransactionMgr` should be consistent with `idToTransactionState`, not only contains running or finished transactions' labels. 2. callback id should be removed when replaying transaction abort or visible edit log This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2240: Fix some bugs about load label
morningman commented on issue #2240: Fix some bugs about load label URL: https://github.com/apache/incubator-doris/issues/2240#issuecomment-00016 There still remaining a bug that load job's state is not consistent with transaction status. It looks like the job' state is LOADING but transactions state is COMMITTED of VISIBLE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2241: Fix some bugs about load label
morningman opened a new pull request #2241: Fix some bugs about load label URL: https://github.com/apache/incubator-doris/pull/2241 1. `dbIdToTxnLabels` in `GlobalTransactionMgr` should be consistent with `idToTransactionState`, not only contains running or finished transactions' labels. 2. callback id should be removed when replaying transaction abort or visible edit log. 3. `LabelAlreadyUsed` exception should be thrown before adding load job. Otherwise, there will be lots of CANCELLED load jobs when reason "label already used". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347915541
##
File path: gensrc/proto/segment_v2.proto
##
@@ -48,12 +48,12 @@ message MetadataPairPB {
enum EncodingTypePB {
UNKNOWN_ENCODING = 0;
DEFAULT_ENCODING = 1;
-PLAIN_ENCODING = 2; // for non-binary type
+PLAIN_ENCODING = 2;
PREFIX_ENCODING = 3;
RLE = 4;
DICT_ENCODING = 5;
BIT_SHUFFLE = 6;
-BINARY_PLAIN_ENCODING = 7;
Review comment:
OK, I see
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347915326
##
File path: be/src/olap/rowset/segment_v2/encoding_info.cpp
##
@@ -50,7 +55,7 @@ struct TypeEncodingTraits {
};
template
-struct TypeEncodingTraits {
+struct TypeEncodingTraits {
Review comment:
OK, I see
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2050: segment_v2: Support bitmap index build
imay commented on a change in pull request #2050: segment_v2: Support bitmap
index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r347915326
##
File path: be/src/olap/rowset/segment_v2/encoding_info.cpp
##
@@ -50,7 +55,7 @@ struct TypeEncodingTraits {
};
template
-struct TypeEncodingTraits {
+struct TypeEncodingTraits {
Review comment:
OK, I see
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error
caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error URL: https://github.com/apache/incubator-doris/issues/2232#issuecomment-02890 it seems that only show create table statement from no range partition table can produce replace keyword, so fix show create table is also work well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #2242: Is that will be better create table with column default enable to be null?
caiconghui opened a new issue #2242: Is that will be better create table with column default enable to be null? URL: https://github.com/apache/incubator-doris/issues/2242 on production enviroment, create table usually with so many nullable columns, i think except key column, other column can be nullable for default? or not nullable column for default is for special purpose? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #2232: When create table with unique key using replace type for non key column encounter error
imay commented on issue #2232: When create table with unique key using replace type for non key column encounter error URL: https://github.com/apache/incubator-doris/issues/2232#issuecomment-09884 @caiconghui I have see your PR for this patch. This is one way to make thing work. And is it easy to let show create table return column defination without REPLACE? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #2242: Is that will be better create table with column default enable to be null?
imay commented on issue #2242: Is that will be better create table with column default enable to be null? URL: https://github.com/apache/incubator-doris/issues/2242#issuecomment-14193 Yes, I aggre with you. Column is created nullable is easy for user, but some impact for performance. I think we can make a column nullable as default. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error
caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error URL: https://github.com/apache/incubator-doris/issues/2232#issuecomment-21952 @imay yes, but for unique key table there is nothing wrong for create table with replace keyword. After deep thinking, I think the real bug is create table statement not the show create table statement, because the create table is nothing wrong, so we should fix create table statement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #2232: When create table with unique key using replace type for non key column encounter error
imay commented on issue #2232: When create table with unique key using replace type for non key column encounter error URL: https://github.com/apache/incubator-doris/issues/2232#issuecomment-26405 @caiconghui Actually, value column of unique column has no aggregate method. REPLACE is unnecessary, there is no other choice for it. We just use REPLACE method internally to implement unique key, which should not be exposed to Doris user. So I think it is better to fix show create table's result if it is easy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error
caiconghui commented on issue #2232: When create table with unique key using replace type for non key column encounter error URL: https://github.com/apache/incubator-doris/issues/2232#issuecomment-31438 @imay ok, I see. I will fix it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #2240: Fix some bugs about load label
imay commented on issue #2240: Fix some bugs about load label URL: https://github.com/apache/incubator-doris/issues/2240#issuecomment-31495 @morningman Can you add more explain about why old version code will cause problem? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new issue #2243: self host null is not in electable fe service group FE. Exit now.
morningman opened a new issue #2243: self host null is not in electable fe service group FE. Exit now. URL: https://github.com/apache/incubator-doris/issues/2243 **Describe the bug** When using DeployManager, FE failed to restart with following error log: `self host null is not in electable fe service group FE. Exit now.` This is because DeployManager is started before catalog is ready, and it failed to get master ip address from not-ready catalog This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2244: Fix bug that DeployManager should start working after catalog is ready.
morningman opened a new pull request #2244: Fix bug that DeployManager should start working after catalog is ready. URL: https://github.com/apache/incubator-doris/pull/2244 Otherwise, it can not get master ip/port from not-ready catalog. ISSUE #2243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2233: fix schema change validate failure #2230
imay merged pull request #2233: fix schema change validate failure #2230 URL: https://github.com/apache/incubator-doris/pull/2233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (03a5359 -> c9db1a5)
This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 03a5359 Fix the line breaks to '\n' (#2234) add c9db1a5 Fix schema change validate failure #2230 (#2233) No new revisions were added by this update. Summary of changes: be/src/olap/rowset/beta_rowset_reader.cpp | 12 ++-- be/src/olap/rowset/beta_rowset_reader.h | 1 + 2 files changed, 7 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2240: Fix some bugs about load label
morningman commented on issue #2240: Fix some bugs about load label URL: https://github.com/apache/incubator-doris/issues/2240#issuecomment-73845 > @morningman > Can you add more explain about why old version code will cause problem? updated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #2050: segment_v2: Support bitmap index build
kangpinghuang commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r348018276
##
File path: be/src/olap/rowset/segment_v2/index_page.h
##
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "gutil/macros.h"
+#include "olap/rowset/segment_v2/page_pointer.h"
+#include "util/faststring.h"
+#include "util/slice.h"
+
+namespace doris {
+namespace segment_v2 {
+
+class IndexPageIterator; // forward decl.
+
+// IndexPage is the building block for IndexedColumn's ordinal index and value
index.
+// It is used to guide searching for a particular key to the data page
containing it.
+// We use the same general format for all index pages, regardless of the data
type and node type (leaf or internal)
+// IndexPage := IndexEntry^NumEntry, StartOffset(4)^NumEntry,
IndexPageFooterPB, IndexPageFooterPBSize(4)
+// IndexEntry := IndexKey, PagePointer
+// IndexKey := KeyLength(vint32), KeyData(KeyLength bytes)
+// PagePointer := PageOffset(vint64), PageSize(vint32)
+//
+// IndexPageFooterPB records NumEntry and type (leaf/internal) of the index
page.
+// For leaf, IndexKey records the first/smallest key of the data page
PagePointer points to.
+// For internal, IndexKey records the first/smallest key of the next-level
index page PagePointer points to.
+//
+// All keys are treated as binary string and compared with memcpy. Keys of
other data type are encoded first by
+// KeyCoder, e.g., ordinal index's original key type is uint32_t but is
encoded to binary string.
+class IndexPageBuilder {
+public:
+explicit IndexPageBuilder(size_t index_page_size, bool is_leaf)
+: _index_page_size(index_page_size), _is_leaf(is_leaf) {
+}
+
+void add(const Slice& key, const PagePointer& ptr);
+
+bool is_full() const;
+
+size_t count() const { return _entry_offsets.size(); }
+
+Slice finish();
+
+// Return the key of the first entry in this index block.
+// The pointed-to data is only valid until the next call to this builder.
+Status get_first_key(Slice* key) const;
+
+void reset() {
+_finished = false;
+_buffer.clear();
+_entry_offsets.clear();
+}
+
+private:
+DISALLOW_COPY_AND_ASSIGN(IndexPageBuilder);
+const size_t _index_page_size;
+const bool _is_leaf;
+// is the builder currently between finish() and reset()?
+bool _finished = false;
+faststring _buffer;
+std::vector _entry_offsets;
+};
+
+class IndexPageReader {
Review comment:
Maybe IndexPage is a better name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #2050: segment_v2: Support bitmap index build
kangpinghuang commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r348025528
##
File path: gensrc/proto/segment_v2.proto
##
@@ -164,3 +166,51 @@ message SegmentFooterPB {
optional PagePointerPB short_key_index_page = 9;
}
+message IndexPageFooterPB {
+ required int32 num_entries = 1;
Review comment:
I think use optional to replace required is better for future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #2050: segment_v2: Support bitmap index build
kangpinghuang commented on issue #2050: segment_v2: Support bitmap index build URL: https://github.com/apache/incubator-doris/pull/2050#issuecomment-84339 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang closed issue #1195: Optimize meta tool for be_refactor
kangpinghuang closed issue #1195: Optimize meta tool for be_refactor URL: https://github.com/apache/incubator-doris/issues/1195 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2244: Fix bug that DeployManager should start working after catalog is ready.
morningman merged pull request #2244: Fix bug that DeployManager should start working after catalog is ready. URL: https://github.com/apache/incubator-doris/pull/2244 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Fix bug that DeployManager should start working after catalog is ready. (#2244)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9b5eeae Fix bug that DeployManager should start working after catalog
is ready. (#2244)
9b5eeae is described below
commit 9b5eeaec19d0c4dbf8fbb83e31f58a9f187f73e5
Author: Mingyu Chen
AuthorDate: Wed Nov 20 09:49:09 2019 +0800
Fix bug that DeployManager should start working after catalog is ready.
(#2244)
Otherwise, it can not get master ip/port from not-ready catalog.
---
fe/src/main/java/org/apache/doris/deploy/DeployManager.java | 6 ++
fe/src/main/java/org/apache/doris/qe/Coordinator.java | 6 +++---
fe/src/main/java/org/apache/doris/qe/ResultReceiver.java | 4 ++--
fe/src/main/java/org/apache/doris/qe/SimpleScheduler.java | 2 +-
fe/src/test/java/org/apache/doris/qe/SimpleSchedulerTest.java | 6 +++---
5 files changed, 15 insertions(+), 9 deletions(-)
diff --git a/fe/src/main/java/org/apache/doris/deploy/DeployManager.java
b/fe/src/main/java/org/apache/doris/deploy/DeployManager.java
index 1db2a5d..4cf5ec3 100644
--- a/fe/src/main/java/org/apache/doris/deploy/DeployManager.java
+++ b/fe/src/main/java/org/apache/doris/deploy/DeployManager.java
@@ -323,6 +323,12 @@ public class DeployManager extends Daemon {
return;
}
+if (!Catalog.getCurrentCatalog().isReady()) {
+// this deploy manager thread is started before catalog is ready.
+// so we have to wait the catalog to be ready.
+return;
+}
+
// 0. init
if (!init()) {
return;
diff --git a/fe/src/main/java/org/apache/doris/qe/Coordinator.java
b/fe/src/main/java/org/apache/doris/qe/Coordinator.java
index 9d61032..15bfd95 100644
--- a/fe/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -499,7 +499,7 @@ public class Coordinator {
case TIMEOUT:
throw new UserException("query timeout. backend
id: " + pair.first.backend.getId());
case THRIFT_RPC_ERROR:
-
SimpleScheduler.updateBlacklistBackends(pair.first.backend.getId());
+
SimpleScheduler.addToBlacklist(pair.first.backend.getId());
throw new
RpcException(pair.first.backend.getHost(), "rpc failed");
default:
throw new UserException(errMsg);
@@ -1334,7 +1334,7 @@ public class Coordinator {
} catch (RpcException e) {
LOG.warn("cancel plan fragment get a exception,
address={}:{}", brpcAddress.getHostname(),
brpcAddress.getPort());
-
SimpleScheduler.updateBlacklistBackends(addressToBackendID.get(brpcAddress));
+
SimpleScheduler.addToBlacklist(addressToBackendID.get(brpcAddress));
}
this.hasCanceled = true;
@@ -1373,7 +1373,7 @@ public class Coordinator {
try {
return
BackendServiceProxy.getInstance().execPlanFragmentAsync(brpcAddress, rpcParams);
} catch (RpcException e) {
-SimpleScheduler.updateBlacklistBackends(backend.getId());
+SimpleScheduler.addToBlacklist(backend.getId());
throw e;
}
}
diff --git a/fe/src/main/java/org/apache/doris/qe/ResultReceiver.java
b/fe/src/main/java/org/apache/doris/qe/ResultReceiver.java
index c309414..857b76c 100644
--- a/fe/src/main/java/org/apache/doris/qe/ResultReceiver.java
+++ b/fe/src/main/java/org/apache/doris/qe/ResultReceiver.java
@@ -116,7 +116,7 @@ public class ResultReceiver {
} catch (RpcException e) {
LOG.warn("fetch result rpc exception, finstId={}", finstId, e);
status.setRpcStatus(e.getMessage());
-SimpleScheduler.updateBlacklistBackends(backendId);
+SimpleScheduler.addToBlacklist(backendId);
} catch (ExecutionException e) {
LOG.warn("fetch result execution exception, finstId={}", finstId,
e);
if (e.getMessage().contains("time out")) {
@@ -124,7 +124,7 @@ public class ResultReceiver {
status.setStatus(new Status(TStatusCode.TIMEOUT,
e.getMessage()));
} else {
status.setRpcStatus(e.getMessage());
-SimpleScheduler.updateBlacklistBackends(backendId);
+SimpleScheduler.addToBlacklist(backendId);
}
} catch (TimeoutException e) {
LOG.warn("fetch result timeout, finstId={}", finstId, e);
diff --git a/fe/src/main/java/org/apache/doris/qe/SimpleScheduler.java
b/fe/src/main/java/org/apache/doris/qe/SimpleScheduler.java
index 8db
[incubator-doris] branch master updated (9b5eeae -> 9be86a3)
This is an automated email from the ASF dual-hosted git repository.
zhaoc pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 9b5eeae Fix bug that DeployManager should start working after catalog
is ready. (#2244)
add 9be86a3 Add gutil split and strip tool (#2238)
No new revisions were added by this update.
Summary of changes:
be/src/gutil/CMakeLists.txt|2 +
.../alpha_rowset_meta.h => gutil/logging-inl.h}| 54 +-
be/src/gutil/strings/split.cc | 1091 ++
be/src/gutil/strings/split.h | 1205
be/src/gutil/strings/split_internal.h | 413 +++
be/src/gutil/strings/strip.cc | 384 +++
be/src/gutil/strings/strip.h | 272 +
7 files changed, 3397 insertions(+), 24 deletions(-)
copy be/src/{olap/rowset/alpha_rowset_meta.h => gutil/logging-inl.h} (56%)
create mode 100644 be/src/gutil/strings/split.cc
create mode 100644 be/src/gutil/strings/split.h
create mode 100644 be/src/gutil/strings/split_internal.h
create mode 100644 be/src/gutil/strings/strip.cc
create mode 100644 be/src/gutil/strings/strip.h
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2238: Add gutil split and strip tool
imay merged pull request #2238: Add gutil split and strip tool URL: https://github.com/apache/incubator-doris/pull/2238 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2231: Fix bug when create table using unique key and repalce aggragate type for non key column
imay merged pull request #2231: Fix bug when create table using unique key and repalce aggragate type for non key column URL: https://github.com/apache/incubator-doris/pull/2231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (9be86a3 -> 46005bf)
This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 9be86a3 Add gutil split and strip tool (#2238) add 46005bf Fix bug for show create table statement with unique key types (#2231) No new revisions were added by this update. Summary of changes: fe/src/main/java/org/apache/doris/analysis/CreateTableStmt.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new issue #2245: support online upgrade from alpha rowset to beta rowset
kangpinghuang opened a new issue #2245: support online upgrade from alpha rowset to beta rowset URL: https://github.com/apache/incubator-doris/issues/2245 In order to support dictionary encoding and bitmap index, we develop new file storage format: segment_v2(#1305 ), and add a new Rowset type: BetaRowset based on the new format. To support online upgrade from alpha rowset to beta rowset, there are several problems should be resolved: - add a mechanism for new created table to specify which storage type to use(alpha or beta) - add a mechanism to change the new loaded rowset's type to BetaRowset for old table - add a mechanism to convert AlphaRowsets in table to BetaRowset(one method is setting the dst rowset type of compaction) - get the rowset statistics of rowset of all table This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new issue #2246: specify storage format for newly created table
kangpinghuang opened a new issue #2246: specify storage format for newly created table URL: https://github.com/apache/incubator-doris/issues/2246 For issue #2245 , to solve problem No.1, we should add a method to specify the storage format for newly created table. here is a method: 1. add a property(namely storage_format) in create table/alter table add partition 2. add an optional field(namely storage_format) in TabletMeta of BE to record the preferred storage format. the following load,compaction, schema change, rollup will use the specified format. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new issue #2247: add a flag mechanism for FE to control all BEs' behaviour
kangpinghuang opened a new issue #2247: add a flag mechanism for FE to control all BEs' behaviour URL: https://github.com/apache/incubator-doris/issues/2247 **Is your feature request related to a problem? Please describe.** Now, Doris lack a mechinism for FE to control all BEs. To solve the issue #2245 's problem No.2, we need a ability like this to set all BE's default rowset type and compaction rowset type. This mechanism can also be used for other purposes. Here my basic design: 1. add a field(maybe use binary format for extensible) in heartbeat message between FE and BE 2. treat the flag field as bitmap, different bit stands for different function. 3. add a http request to set the flag in FE 4. BE can parse the flag to do the different functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #2247: add a flag mechanism for FE to control all BEs' behaviour
kangpinghuang commented on issue #2247: add a flag mechanism for FE to control all BEs' behaviour URL: https://github.com/apache/incubator-doris/issues/2247#issuecomment-555805219 There are two flag bits for Issue #2245, one for set the default rowset type to BETA; one for set the default compaction type to BETA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang edited a comment on issue #2247: add a flag mechanism for FE to control all BEs' behaviour
kangpinghuang edited a comment on issue #2247: add a flag mechanism for FE to control all BEs' behaviour URL: https://github.com/apache/incubator-doris/issues/2247#issuecomment-555805219 There are two flag bits for Issue #2245, - one for set the default rowset type to BETA; - one for set the default compaction type to BETA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset
kangpinghuang opened a new issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset URL: https://github.com/apache/incubator-doris/issues/2248 For issue #2245 , in the final phase, we should convert all AlphaRowsets to BetaRowset, and need a method to make sure to achieve this purpose. Here is a method. 1. add a convert thread in BE to do the convert job 2. make use of the flag mechanism of issue #2247, add a flag bit to indicate the convert thread to start 3. report all tablet's rowset type to FE to collect the process. BTW, we should control the convert process to make sure that it do not affect the online read and write performance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about load label
EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about
load label
URL: https://github.com/apache/incubator-doris/pull/2241#discussion_r348266634
##
File path: fe/src/main/java/org/apache/doris/load/loadv2/LoadManager.java
##
@@ -231,20 +225,20 @@ public void createLoadJobV1FromMultiStart(String
fullDbName, String label) throw
}
public void replayCreateLoadJob(LoadJob loadJob) {
-createLoadJob(loadJob);
+addLoadJobImpl(loadJob);
Review comment:
Why don't you add the callback of txn in here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about load label
EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about
load label
URL: https://github.com/apache/incubator-doris/pull/2241#discussion_r348266288
##
File path: fe/src/main/java/org/apache/doris/load/loadv2/LoadManager.java
##
@@ -135,23 +135,17 @@ public long
createLoadJobFromMiniLoad(TMiniLoadBeginRequest request) throws User
LoadJob loadJob = null;
writeLock();
try {
-checkLabelUsed(database.getId(), request.getLabel(),
request.getRequest_id());
+// checkLabelUsed(database.getId(), request.getLabel(),
request.getRequest_id());
loadJob = new MiniLoadJob(database.getId(), request);
-createLoadJob(loadJob);
+// call unprotectedExecute before adding load job. so that if job
is not started ok, no need to add
// Mini load job must be executed before release write lock.
// Otherwise, the duplicated request maybe get the transaction id
before transaction of mini load is begun.
loadJob.unprotectedExecute();
-} catch (DuplicatedRequestException e) {
Review comment:
The duplicated request should return the txnId to BE
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about load label
EmmyMiao87 commented on a change in pull request #2241: Fix some bugs about
load label
URL: https://github.com/apache/incubator-doris/pull/2241#discussion_r348266093
##
File path: fe/src/main/java/org/apache/doris/load/loadv2/LoadManager.java
##
@@ -135,23 +135,17 @@ public long
createLoadJobFromMiniLoad(TMiniLoadBeginRequest request) throws User
LoadJob loadJob = null;
writeLock();
try {
-checkLabelUsed(database.getId(), request.getLabel(),
request.getRequest_id());
+// checkLabelUsed(database.getId(), request.getLabel(),
request.getRequest_id());
loadJob = new MiniLoadJob(database.getId(), request);
-createLoadJob(loadJob);
+// call unprotectedExecute before adding load job. so that if job
is not started ok, no need to add
// Mini load job must be executed before release write lock.
// Otherwise, the duplicated request maybe get the transaction id
before transaction of mini load is begun.
loadJob.unprotectedExecute();
-} catch (DuplicatedRequestException e) {
-LOG.info(new LogBuilder(LogKey.LOAD_JOB,
e.getDuplicatedRequestId())
- .add("msg", "the duplicated request returns the
txn id "
- + "which was created by the same mini
load")
- .build());
-return
dbIdToLabelToLoadJobs.get(database.getId()).get(request.getLabel())
-.stream().filter(entity -> entity.getState() !=
JobState.CANCELLED).findFirst()
-.get().getTransactionId();
+addLoadJob(loadJob);
Review comment:
addLoadJob(loadJob);
loadJob.unprotectedExecute();
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset
kangpinghuang commented on issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset URL: https://github.com/apache/incubator-doris/issues/2248#issuecomment-555808636 Maybe we shoule export a api to convert one table's format from Alpha to Beta for users who want to use the new features in BetaRowsets. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang edited a comment on issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset
kangpinghuang edited a comment on issue #2248: add a mechanism to convert all AlphaRowsets in table to BetaRowset URL: https://github.com/apache/incubator-doris/issues/2248#issuecomment-555808636 Maybe we shoule export **a api to convert one table's format** from Alpha to Beta for users who want to use the new features in BetaRowsets. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new issue #2249: add a mechanism for FE to collect the table's format statistics
kangpinghuang opened a new issue #2249: add a mechanism for FE to collect the table's format statistics URL: https://github.com/apache/incubator-doris/issues/2249 For issue #2245 , in order to be aware of the convert process from AlphaRowset to BetaRowset, we need a mechanism to know the process of convert. here is my basic design: 1. when BE report to FE, add fields to indicate how many AlphaRowset and BetaRowset there are. 2. FE collect the total statistics from all BEs, and calcute the the total process of convert, and should also provide a api to get the convert process of single table. 3. http api or show proc can be the apis to get the statistics. like: curl -x GET http://fe_host:fe_http_port/api/format/convert/progress curl -x GET http://fe_host:fe_http_port/api/format/convert/progress/cluster/db/table_name show proc "/convert/process"; show proc "/convert/cluster/db/table_name" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #1389: Support bloom filter in new format segment
kangpinghuang commented on issue #1389: Support bloom filter in new format segment URL: https://github.com/apache/incubator-doris/issues/1389#issuecomment-555811201 I will solve this issue. Here is my basic design: 1. use BlockedBloomFilter to realize the bloom filter format, ref: https://github.com/apache/parquet-format/blob/master/BloomFilter.md 2. use the IndexPage to add ordinal index for bloom filter index to load the required bf page and boost the query performance This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wkhappy1 opened a new pull request #2250: fix export to hdfs bug
wkhappy1 opened a new pull request #2250: fix export to hdfs bug URL: https://github.com/apache/incubator-doris/pull/2250 can not export table to hdfs thanks to @imay help This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r348272133
##
File path: gensrc/proto/segment_v2.proto
##
@@ -164,3 +166,51 @@ message SegmentFooterPB {
optional PagePointerPB short_key_index_page = 9;
}
+message IndexPageFooterPB {
+ required int32 num_entries = 1;
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request #2050: segment_v2: Support bitmap index build
gaodayue commented on a change in pull request #2050: segment_v2: Support
bitmap index build
URL: https://github.com/apache/incubator-doris/pull/2050#discussion_r348275793
##
File path: be/src/olap/rowset/segment_v2/index_page.h
##
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "gutil/macros.h"
+#include "olap/rowset/segment_v2/page_pointer.h"
+#include "util/faststring.h"
+#include "util/slice.h"
+
+namespace doris {
+namespace segment_v2 {
+
+class IndexPageIterator; // forward decl.
+
+// IndexPage is the building block for IndexedColumn's ordinal index and value
index.
+// It is used to guide searching for a particular key to the data page
containing it.
+// We use the same general format for all index pages, regardless of the data
type and node type (leaf or internal)
+// IndexPage := IndexEntry^NumEntry, StartOffset(4)^NumEntry,
IndexPageFooterPB, IndexPageFooterPBSize(4)
+// IndexEntry := IndexKey, PagePointer
+// IndexKey := KeyLength(vint32), KeyData(KeyLength bytes)
+// PagePointer := PageOffset(vint64), PageSize(vint32)
+//
+// IndexPageFooterPB records NumEntry and type (leaf/internal) of the index
page.
+// For leaf, IndexKey records the first/smallest key of the data page
PagePointer points to.
+// For internal, IndexKey records the first/smallest key of the next-level
index page PagePointer points to.
+//
+// All keys are treated as binary string and compared with memcpy. Keys of
other data type are encoded first by
+// KeyCoder, e.g., ordinal index's original key type is uint32_t but is
encoded to binary string.
+class IndexPageBuilder {
+public:
+explicit IndexPageBuilder(size_t index_page_size, bool is_leaf)
+: _index_page_size(index_page_size), _is_leaf(is_leaf) {
+}
+
+void add(const Slice& key, const PagePointer& ptr);
+
+bool is_full() const;
+
+size_t count() const { return _entry_offsets.size(); }
+
+Slice finish();
+
+// Return the key of the first entry in this index block.
+// The pointed-to data is only valid until the next call to this builder.
+Status get_first_key(Slice* key) const;
+
+void reset() {
+_finished = false;
+_buffer.clear();
+_entry_offsets.clear();
+}
+
+private:
+DISALLOW_COPY_AND_ASSIGN(IndexPageBuilder);
+const size_t _index_page_size;
+const bool _is_leaf;
+// is the builder currently between finish() and reset()?
+bool _finished = false;
+faststring _buffer;
+std::vector _entry_offsets;
+};
+
+class IndexPageReader {
Review comment:
let's review read part in separate PR :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay closed pull request #2050: segment_v2: Support bitmap index build
imay closed pull request #2050: segment_v2: Support bitmap index build URL: https://github.com/apache/incubator-doris/pull/2050 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] gaodayue opened a new pull request #2050: segment_v2: Support bitmap index build
gaodayue opened a new pull request #2050: segment_v2: Support bitmap index build URL: https://github.com/apache/incubator-doris/pull/2050 This PR implements the build part of bitmap index support. It follows most of the design described in #1684 , but with the following differences and enhancements 1. Bitmap indexes are now written in the segment file for simplicity. Separate index file would be helpful when we're going to support `alter table add bitmap index` in the future though. 2. We switch to a generalized index page format for all data types rather than specialize for each one. Code simplicity and reusability is preferred here than optimal compression rate. 3. We introduce a new abstraction called `IndexedColumn` to unify the processing of the dictionary section and bitmap section of bitmap index. IndexedColumn is a column with an optional ordinal index and an optional value index. Ordinal index enables us to seek to a particular rowid within the column. Value index requires IndexedColumn to store ordered values and enables us to seek to a particular value. Therefore, the dictionary section can be represented by an IndexedColumn with value index and the bitmap section can be represented by an IndexedColumn with ordinal index. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new issue #2251: String | | String Syntax not supported in Doris
xy720 opened a new issue #2251: String | | String Syntax not supported in Doris URL: https://github.com/apache/incubator-doris/issues/2251 String | | String Syntax is a required test in tp-cds benchmark but it is not supported for Doris, we can rewrite is as concat (string, string). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new pull request #2252: Add pipes_as_concat_mode
xy720 opened a new pull request #2252: Add pipes_as_concat_mode
URL: https://github.com/apache/incubator-doris/pull/2252
ISSUE #2251
This commit will add a new sql mode named MODE_PIPES_AS_CONCAT:
Description:
1、If this mode is active, '||' will be handled different from the original
way ('||' and 'or' are seen as the same symbols in Doris) that it can be used
to concat two exps and returns a new string. For example, 'a' || 'b' = 'ab' and
1 || 0 = '10'.
2. User can active this mode by "SET sql_mode = PIPES_AS_CONCAT", and
deactive it by "SET sql_mode = '' ". This setting is only valid for the current
session.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new pull request #2253: fix npe for null stats
kangpinghuang opened a new pull request #2253: fix npe for null stats URL: https://github.com/apache/incubator-doris/pull/2253 fix npe for null stats when schema change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #2253: fix npe for null stats
chaoyli merged pull request #2253: fix npe for null stats URL: https://github.com/apache/incubator-doris/pull/2253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (46005bf -> aedccc4)
This is an automated email from the ASF dual-hosted git repository. lichaoyong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 46005bf Fix bug for show create table statement with unique key types (#2231) add aedccc4 fix npe for null stats (#2253) No new revisions were added by this update. Summary of changes: be/src/olap/rowset/beta_rowset_reader.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #2254: Fix bug for showing columns from non exist table doesn't prompt error
caiconghui opened a new pull request #2254: Fix bug for showing columns from non exist table doesn't prompt error URL: https://github.com/apache/incubator-doris/pull/2254 when we execute show columns statement for non exist table, the response from doris is nothing, which may make user confused This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #2255: Showing columns from non exist table doesn't prompt error
caiconghui opened a new issue #2255: Showing columns from non exist table doesn't prompt error URL: https://github.com/apache/incubator-doris/issues/2255 **Describe the bug** showing columns from non exist table doesn't prompt error **To Reproduce** Steps to reproduce the behavior: execute such statement for non exist table like: show columns from test.non_exist_table **Expected behavior** ERROR 1064 (HY000): Table default_cluster:test.non_exist_table no found **Screenshots** now, just no any hint to tell user that table doesn't exist. the result is Empty set (0.00 sec) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
