[GitHub] [doris] morningman commented on a diff in pull request #16036: [feature] Support mutable property for partition
morningman commented on code in PR #16036:
URL: https://github.com/apache/doris/pull/16036#discussion_r1103753667
##
fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionInfo.java:
##
@@ -76,13 +76,16 @@ public class PartitionInfo implements Writable {
// so we defer adding meta serialization until memory engine feature is
more complete.
protected Map idToTabletType;
+protected Map idToMutable;
Review Comment:
You can add this new properties in `DataProperty`
##
be/src/vec/sink/vtablet_sink.cpp:
##
@@ -1004,6 +1004,23 @@ Status VOlapTableSink::find_tablet(RuntimeState* state,
vectorized::Block* block
is_continue = true;
return status;
}
+if (!(*partition)->is_mutable) {
+RETURN_IF_ERROR(state->append_error_msg_to_file(
+[]() -> std::string { return ""; },
+[&]() -> std::string {
+fmt::memory_buffer buf;
+fmt::format_to(buf, "partition immutable for this tuple.
tuple={}",
Review Comment:
And if you call `append_error_msg_to_file`, it may return `stop_processing`
to true, and stop the load job.
But in your design, these rows should be `unselected rows`, and should not
stop the job
##
be/src/vec/sink/vtablet_sink.cpp:
##
@@ -1004,6 +1004,23 @@ Status VOlapTableSink::find_tablet(RuntimeState* state,
vectorized::Block* block
is_continue = true;
return status;
}
+if (!(*partition)->is_mutable) {
+RETURN_IF_ERROR(state->append_error_msg_to_file(
+[]() -> std::string { return ""; },
+[&]() -> std::string {
+fmt::memory_buffer buf;
+fmt::format_to(buf, "partition immutable for this tuple.
tuple={}",
Review Comment:
```suggestion
fmt::format_to(buf, "partition is immutable for this
tuple. tuple={}",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16036: [feature] Support mutable property for partition
morningman commented on PR #16036: URL: https://github.com/apache/doris/pull/16036#issuecomment-1426968706 And please add some regression test for this feature -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16643: [docs](docs)Fix Docker documentation description
github-actions[bot] commented on PR #16643: URL: https://github.com/apache/doris/pull/16643#issuecomment-1426970724 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16643: [docs](docs)Fix Docker documentation description
github-actions[bot] commented on PR #16643: URL: https://github.com/apache/doris/pull/16643#issuecomment-1426970731 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16643: [docs](docs)Fix Docker documentation description
yiguolei merged PR #16643: URL: https://github.com/apache/doris/pull/16643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [docs](docs)Fix Docker documentation description (#16643)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 0701ce1d71 [docs](docs)Fix Docker documentation description (#16643) 0701ce1d71 is described below commit 0701ce1d718608871212cf162fb142e502d6b466 Author: FreeOnePlus <54164178+freeonep...@users.noreply.github.com> AuthorDate: Sun Feb 12 16:37:51 2023 +0800 [docs](docs)Fix Docker documentation description (#16643) * change sh_checker_exclude * add broker Dockerfile and init_broker.sh * add docker docs * Adjust the field naming rules when creating tables * Fix Docker documentation description - Co-authored-by: Yijia Su --- .../install/construct-docker/construct-docker-image.md | 2 +- .../install/construct-docker/run-docker-cluster.md | 6 -- .../install/construct-docker/construct-docker-image.md | 2 +- .../install/construct-docker/run-docker-cluster.md | 18 ++ 4 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/en/docs/install/construct-docker/construct-docker-image.md b/docs/en/docs/install/construct-docker/construct-docker-image.md index d98d190a1b..4d270d3801 100644 --- a/docs/en/docs/install/construct-docker/construct-docker-image.md +++ b/docs/en/docs/install/construct-docker/construct-docker-image.md @@ -67,7 +67,7 @@ In the Dockerfile script for compiling the Docker Image, there are two methods t 1. Execute the download command via wget / curl when compiling, and then start the docker build process. 2. Download the binary package to the compilation directory in advance, and then load it into the docker build process through the ADD or COPY command. -Method 1 can produce a smaller Docker image, but if the docker build process fails, the download operation might be repeated and result in longer build time; Method 2 is more suitable for less-than-ideal network environments. Method 2 will produce a Docker image that is slightly larger than that from Method 1. +Method 1 can produce a smaller Docker image, but if the docker build process fails, the download operation might be repeated and result in longer build time; Method 2 is more suitable for less-than-ideal network environments. **The examples below are based on Method 2. If you prefer to go for Method 1, you may modify the steps accordingly.** diff --git a/docs/en/docs/install/construct-docker/run-docker-cluster.md b/docs/en/docs/install/construct-docker/run-docker-cluster.md index a6ca26aae9..36cdcb1b94 100644 --- a/docs/en/docs/install/construct-docker/run-docker-cluster.md +++ b/docs/en/docs/install/construct-docker/run-docker-cluster.md @@ -81,6 +81,7 @@ From the version of `Apache Doris 1.2.1 Docker Image`, the interface list of eac | FE\BE\BROKER| FE_SERVERS | FE node main information | fe1:172.20.80.2:9010,fe2:172.20.80.3:9010,fe3:172.20.80.4:9010 | | FE | FE_ID | FE node ID | 1 | | BE | BE_ADDR | BE node main information | 172.20.80.5:9050 | +| BE | NODE_ROLE | BE node type | computation | | BROKER | BROKER_ADDR | Main information of BROKER node | 172.20.80.6:8000 | Note that the above interface must fill in the information, otherwise the process cannot be started. @@ -91,6 +92,8 @@ Note that the above interface must fill in the information, otherwise the proces > > BE_ADDR interface rule is: `BE_HOST:BE_HEARTBEAT_SERVICE_PORT` > +> The NODE_ROLE interface rule is: `computation` or empty, where empty or other values indicate that the node type is `mix` type +> > BROKER_ADDR interface rule is: `BROKER_HOST:BROKER_IPC_PORT` ### Script Template @@ -211,5 +214,4 @@ Then `exit` exits and creates the Doris Docker cluster. ## Unfinished business -1. The Computer Node interface has not yet been adapted, and the next version will adapt -2. Compose Demo List +1. Compose Demo List diff --git a/docs/zh-CN/docs/install/construct-docker/construct-docker-image.md b/docs/zh-CN/docs/install/construct-docker/construct-docker-image.md index 178412375e..d2e87bdd14 100644 --- a/docs/zh-CN/docs/install/construct-docker/construct-docker-image.md +++ b/docs/zh-CN/docs/install/construct-docker/construct-docker-image.md @@ -68,7 +68,7 @@ Dockerfile 脚本编写需要注意以下几点： 1. 通过 wget / curl 在编译时执行下载命令，随后完成 docker build 制作过程 2. 提前下载二进制包至编译目录，然后通过 ADD 或者 COPY 命令加载至 docker build 过程中 -使用前者会让 Docker Image Size 更小，但是如果构建失败的话可能下载操作会重复进行，导致构建时间过长，而后者更适用于网络环境不是很好的构建环境。后者构建的镜像要略大于前者，但是不会大很多。 +使用前者会让 Docker Image Size 更小，但是如果构建失败的话可能下载操作会重复进行，导致构建时间过长，而后者更适用于网络环境不是很好的构建环境。 **综上，本文档的示例以第二种方式为准，若有第一种诉求，可根据自己需求定制修改即可。** diff --git a/docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md b/docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md index 963f91a94f..20484acdd8 100644 --- a/docs/zh-CN/docs/install/construct-docker/run-d
[GitHub] [doris] morningman commented on a diff in pull request #14063: [Feature](ipv6)Support IPV6
morningman commented on code in PR #14063:
URL: https://github.com/apache/doris/pull/14063#discussion_r1103756592
##
fe/fe-core/src/main/java/org/apache/doris/httpv2/controller/BaseController.java:
##
@@ -296,6 +296,6 @@ protected long checkLongParam(String strParam) {
}
protected String getCurrentFrontendURL() {
-return "http://"; + FrontendOptions.getLocalHostAddress() + ":" +
Config.http_port;
+return "http://"; + FrontendOptions.getHostname() + ":" +
Config.http_port;
Review Comment:
What is diff between `getLocalHostAddress` and `getHostname`?
##
be/src/util/network_util.cpp:
##
@@ -39,22 +39,38 @@
namespace doris {
InetAddress::InetAddress(struct sockaddr* addr) {
-this->addr = *(struct sockaddr_in*)addr;
+this->addr = *addr;
}
-bool InetAddress::is_address_v4() const {
-return addr.sin_family == AF_INET;
+bool InetAddress::is_loopback() {
+if (addr.sa_family == AF_INET) {
+in_addr_t s_addr = ((struct sockaddr_in*)&addr)->sin_addr.s_addr;
+return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+} else if (addr.sa_family == AF_INET6) {
+struct in6_addr in6_addr = ((struct sockaddr_in6*)&addr)->sin6_addr;
+return IN6_IS_ADDR_LOOPBACK(&in6_addr);
+} else {
+LOG(WARNING) << "unknow address";
Review Comment:
```suggestion
LOG(WARNING) << "unknow address: " << addr.sa_family;
```
##
be/src/util/network_util.cpp:
##
@@ -39,22 +39,38 @@
namespace doris {
InetAddress::InetAddress(struct sockaddr* addr) {
-this->addr = *(struct sockaddr_in*)addr;
+this->addr = *addr;
}
-bool InetAddress::is_address_v4() const {
-return addr.sin_family == AF_INET;
+bool InetAddress::is_loopback() {
+if (addr.sa_family == AF_INET) {
+in_addr_t s_addr = ((struct sockaddr_in*)&addr)->sin_addr.s_addr;
+return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+} else if (addr.sa_family == AF_INET6) {
+struct in6_addr in6_addr = ((struct sockaddr_in6*)&addr)->sin6_addr;
+return IN6_IS_ADDR_LOOPBACK(&in6_addr);
+} else {
+LOG(WARNING) << "unknow address";
+return false;
+}
}
-bool InetAddress::is_loopback_v4() {
-in_addr_t s_addr = addr.sin_addr.s_addr;
-return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+std::string InetAddress::get_host_address() {
+if (addr.sa_family == AF_INET) {
+char addr_buf[INET_ADDRSTRLEN];
+inet_ntop(AF_INET, &(((struct sockaddr_in*)&addr)->sin_addr),
addr_buf, INET_ADDRSTRLEN);
+return std::string(addr_buf);
+} else if (addr.sa_family == AF_INET6) {
+char addr_buf[INET6_ADDRSTRLEN];
+inet_ntop(AF_INET6, &(((struct sockaddr_in6*)&addr)->sin6_addr),
addr_buf,
+ INET6_ADDRSTRLEN);
+return std::string(addr_buf);
+} else {
+return std::string {"unknown address"};
Review Comment:
```suggestion
return std::string {"unknown address: " + addr.sa_family};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16499: [enhance](community): polish PULL_REQUEST_TEMPLATE.md
yiguolei merged PR #16499: URL: https://github.com/apache/doris/pull/16499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](community): polish PULL_REQUEST_TEMPLATE.md (#16499)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 4e814a7bbc [enhance](community): polish PULL_REQUEST_TEMPLATE.md (#16499) 4e814a7bbc is described below commit 4e814a7bbc45c0c10889fcd7fe7d49f52dfd3db9 Author: jakevin AuthorDate: Sun Feb 12 16:39:02 2023 +0800 [enhance](community): polish PULL_REQUEST_TEMPLATE.md (#16499) --- .github/PULL_REQUEST_TEMPLATE.md | 23 +-- 1 file changed, 5 insertions(+), 18 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index c6604f632a..4a3ddae5f1 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -8,24 +8,11 @@ Describe your changes. ## Checklist(Required) -1. Does it affect the original behavior: -- [ ] Yes -- [ ] No -- [ ] I don't know -2. Has unit tests been added: -- [ ] Yes -- [ ] No -- [ ] No Need -3. Has document been added or modified: -- [ ] Yes -- [ ] No -- [ ] No Need -4. Does it need to update dependencies: -- [ ] Yes -- [ ] No -5. Are there any changes that cannot be rolled back: -- [ ] Yes (If Yes, please explain WHY) -- [ ] No +* [ ] Does it affect the original behavior +* [ ] Has unit tests been added +* [ ] Has document been added or modified +* [ ] Does it need to update dependencies +* [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
morningman commented on code in PR #16073:
URL: https://github.com/apache/doris/pull/16073#discussion_r1103758624
##
gensrc/thrift/FrontendService.thrift:
##
@@ -483,6 +483,7 @@ struct TLoadTxnBeginRequest {
10: optional i64 timeout
11: optional Types.TUniqueId request_id
12: optional string auth_code_uuid
+13: optional string token
Review Comment:
I think we don't need to handle this compatibility, just keep only one field
is ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16641: [Bug](Cooldown) fix load balance causing no cooldown replica
dataroaring merged PR #16641: URL: https://github.com/apache/doris/pull/16641 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](Cooldown) fix load balance causing no cooldown replica (#16641)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 6a8fc35b78 [Bug](Cooldown) fix load balance causing no cooldown
replica (#16641)
6a8fc35b78 is described below
commit 6a8fc35b78a1ea66777c151a195f3e404b97028b
Author: AlexYue
AuthorDate: Sun Feb 12 16:47:38 2023 +0800
[Bug](Cooldown) fix load balance causing no cooldown replica (#16641)
---
be/src/olap/tablet.cpp | 3 ++-
be/src/olap/tablet_meta.cpp| 1 +
be/src/olap/task/engine_clone_task.cpp | 13 -
gensrc/proto/olap_file.proto | 3 ++-
4 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index ab095be7d5..7ed8d14b43 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -1769,7 +1769,8 @@ Status Tablet::_write_cooldown_meta(io::RemoteFileSystem*
fs, RowsetMeta* new_rs
Status Tablet::_follow_cooldowned_data(io::RemoteFileSystem* fs, int64_t
cooldown_replica_id) {
LOG(INFO) << "try to follow cooldowned data. tablet_id=" << tablet_id()
- << " cooldown_replica_id=" << cooldown_replica_id;
+ << " cooldown_replica_id=" << cooldown_replica_id
+ << " local replica=" << replica_id();
TabletMetaPB cooldown_meta_pb;
RETURN_IF_ERROR(_read_cooldown_meta(fs, cooldown_replica_id,
&cooldown_meta_pb));
DCHECK(cooldown_meta_pb.rs_metas_size() > 0);
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index 61bb4bcf56..7562efa5b2 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -247,6 +247,7 @@ TabletMeta::TabletMeta(const TabletMeta& b)
: _table_id(b._table_id),
_partition_id(b._partition_id),
_tablet_id(b._tablet_id),
+ _replica_id(b._replica_id),
_schema_hash(b._schema_hash),
_shard_id(b._shard_id),
_creation_time(b._creation_time),
diff --git a/be/src/olap/task/engine_clone_task.cpp
b/be/src/olap/task/engine_clone_task.cpp
index b77373ec6b..80afdb456f 100644
--- a/be/src/olap/task/engine_clone_task.cpp
+++ b/be/src/olap/task/engine_clone_task.cpp
@@ -97,7 +97,7 @@ Status EngineCloneTask::_do_clone() {
// completed. Or remote be will just return header not the rowset
files. clone will failed.
if (missed_versions.empty()) {
LOG(INFO) << "missed version size = 0, skip clone and return
success. tablet_id="
- << _clone_req.tablet_id;
+ << _clone_req.tablet_id << " req replica=" <<
_clone_req.replica_id;
_set_tablet_info(is_new_tablet);
return Status::OK();
}
@@ -105,7 +105,8 @@ Status EngineCloneTask::_do_clone() {
LOG(INFO) << "clone to existed tablet. missed_versions_size=" <<
missed_versions.size()
<< ", allow_incremental_clone=" << allow_incremental_clone
<< ", signature=" << _signature << ", tablet_id=" <<
_clone_req.tablet_id
- << ", committed_version=" << _clone_req.committed_version;
+ << ", committed_version=" << _clone_req.committed_version
+ << ", req replica=" << _clone_req.replica_id;
// try to download missing version from src backend.
// if tablet on src backend does not contains missing version, it will
download all versions,
@@ -119,7 +120,8 @@ Status EngineCloneTask::_do_clone() {
} else {
LOG(INFO) << "clone tablet not exist, begin clone a new tablet from
remote be. "
<< "signature=" << _signature << ", tablet_id=" <<
_clone_req.tablet_id
- << ", committed_version=" << _clone_req.committed_version;
+ << ", committed_version=" << _clone_req.committed_version
+ << ", req replica=" << _clone_req.replica_id;
// create a new tablet in this be
// Get local disk from olap
string local_shard_root_path;
@@ -194,7 +196,7 @@ Status EngineCloneTask::_set_tablet_info(bool
is_new_tablet) {
}
LOG(INFO) << "clone get tablet info success. tablet_id:" <<
_clone_req.tablet_id
<< ", schema_hash:" << _clone_req.schema_hash << ", signature:"
<< _signature
- << ", version:" << tablet_info.version;
+ << ", replica id:" << _clone_req.replica_id << ", version:" <<
tablet_info.version;
_tablet_infos->push_back(tablet_info);
return Status::OK();
}
@@ -537,7 +539,8 @@ Status EngineCloneTask::_finish_incremental_clone(Tablet*
tablet,
const TabletMeta&
cloned_tablet_meta,
int64_t committed_version) {
LOG(INFO) << "begin to finish incremental clone. tablet=" <<
[GitHub] [doris] morningman merged pull request #16622: [improvement](Load) Make broker load support the properties of trim_double_quotes and skip_lines
morningman merged PR #16622: URL: https://github.com/apache/doris/pull/16622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](Load) Make broker load support the properties of trim_double_quotes and skip_lines (#16622)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 78a958467f [improvement](Load) Make broker load support the properties
of trim_double_quotes and skip_lines (#16622)
78a958467f is described below
commit 78a958467fbb4cfd491071bca22df09cf581b595
Author: huangzhaowei
AuthorDate: Sun Feb 12 16:52:59 2023 +0800
[improvement](Load) Make broker load support the properties of
trim_double_quotes and skip_lines (#16622)
`trim_double_quotes` and `skip_lines` were supported in stream load.
So make it support broker load too.
---
.../Data-Manipulation-Statements/Load/BROKER-LOAD.md | 18 ++
.../Data-Manipulation-Statements/Load/STREAM-LOAD.md | 2 ++
.../Data-Manipulation-Statements/Load/BROKER-LOAD.md | 18 ++
.../Data-Manipulation-Statements/Load/STREAM-LOAD.md | 1 +
.../org/apache/doris/analysis/DataDescription.java | 7 +++
.../main/java/org/apache/doris/analysis/LoadStmt.java | 13 +
6 files changed, 59 insertions(+)
diff --git
a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
index 4aa1a1aff5..6665b87c83 100644
---
a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
+++
b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
@@ -477,6 +477,24 @@ WITH BROKER broker_name
)
```
+12. Load CSV date and trim double quotes and skip first 5 lines
+
+```SQL
+LOAD LABEL example_db.label12
+(
+DATA INFILE("cosn://my_bucket/input/file.csv")
+INTO TABLE `my_table`
+(k1, k2, k3)
+PROPERTIES("trim_double_quotes" = "true", "skip_lines" = "5")
+)
+WITH BROKER "broker_name"
+(
+"fs.cosn.userinfo.secretId" = "xxx",
+"fs.cosn.userinfo.secretKey" = "",
+"fs.cosn.bucket.endpoint_suffix" = "cos.x.myqcloud.com"
+)
+```
+
### Keywords
BROKER, LOAD
diff --git
a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
index 897b6f5116..48c3d0284d 100644
---
a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
+++
b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
@@ -182,6 +182,8 @@ ERRORS:
25. trim_double_quotes: Boolean type, The default value is false. True means
that the outermost double quotes of each field in the csv file are trimmed.
+26. skip_lines: Integer type, the default
value is 0. It will skip some lines in the head of csv file. It will be
disabled when format is `csv_with_names` or `csv_with_names_and_types`.
+
### Example
1. Import the data in the local file 'testData' into the table 'testTbl' in
the database 'testDb', and use Label for deduplication. Specify a timeout of
100 seconds
diff --git
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
index 83838b00a3..03f73e5578 100644
---
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
+++
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/BROKER-LOAD.md
@@ -475,6 +475,24 @@ WITH BROKER broker_name
)
```
+12. 导入CSV数据时去掉双引号, 并跳过前5行。
+
+```SQL
+LOAD LABEL example_db.label12
+(
+DATA INFILE("cosn://my_bucket/input/file.csv")
+INTO TABLE `my_table`
+(k1, k2, k3)
+PROPERTIES("trim_double_quotes" = "true", "skip_lines" = "5")
+)
+WITH BROKER "broker_name"
+(
+"fs.cosn.userinfo.secretId" = "xxx",
+"fs.cosn.userinfo.secretKey" = "",
+"fs.cosn.bucket.endpoint_suffix" = "cos.x.myqcloud.com"
+)
+```
+
### Keywords
BROKER, LOAD
diff --git
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
index 444509b5c9..94ff15167c 100644
---
a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
+++
b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/STREAM-LOAD.md
@@ -179,6 +179,7 @@ ERRORS:
25. trim_double_quotes: 布尔类型，默认值为 false，为 true 时表示裁剪掉 csv 文件每个字段最外层的双引号。
+26. skip_lines: 整数类型, 默认值为0,
含义为跳过csv文件的前几行. 当设置format设置为 `csv_with_names` 或、`csv_with_names_and_types` 时,
该参数会失效.
### Example
1. 将本地文件'testData'中的数据导入到数据库'testDb'中'testTbl'的表，使用Label用于去重。指定超时时间为 100 秒
diff --git
a/fe/fe-core
[GitHub] [doris] dataroaring merged pull request #16343: [Enhancement](Stmt) Set insert_into timeout session variable separately
dataroaring merged PR #16343: URL: https://github.com/apache/doris/pull/16343 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring closed issue #16301: [Enhancement] insert into timeout easily
dataroaring closed issue #16301: [Enhancement] insert into timeout easily URL: https://github.com/apache/doris/issues/16301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](Stmt) Set insert_into timeout session variable separately (#16343)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cf739e7496 [Enhancement](Stmt) Set insert_into timeout session
variable separately (#16343)
cf739e7496 is described below
commit cf739e74962c544a84208cf9c3121c219ddb392f
Author: 奕冷 <82279870+tangsiyang2...@users.noreply.github.com>
AuthorDate: Sun Feb 12 16:56:10 2023 +0800
[Enhancement](Stmt) Set insert_into timeout session variable separately
(#16343)
---
be/src/exprs/runtime_filter.cpp| 4 +-
be/src/runtime/fragment_mgr.cpp| 4 +-
be/src/runtime/result_buffer_mgr.cpp | 4 +-
be/src/runtime/result_buffer_mgr.h | 2 +-
be/src/runtime/runtime_state.h | 2 +
be/src/vec/sink/vdata_stream_sender.cpp| 2 +-
be/src/vec/sink/vresult_file_sink.cpp | 2 +-
be/src/vec/sink/vresult_sink.cpp | 2 +-
be/src/vec/sink/vtablet_sink.cpp | 6 +--
docs/en/docs/advanced/variables.md | 7 ++-
.../import/import-way/insert-into-manual.md| 4 +-
.../Manipulation/INSERT.md | 2 +-
.../SET-VARIABLE.md| 1 +
docs/zh-CN/docs/advanced/variables.md | 6 ++-
.../import/import-way/insert-into-manual.md| 4 +-
.../Manipulation/INSERT.md | 2 +-
.../SET-VARIABLE.md| 1 +
.../java/org/apache/doris/analysis/InsertStmt.java | 2 +-
.../java/org/apache/doris/qe/ConnectContext.java | 55 --
.../java/org/apache/doris/qe/ConnectProcessor.java | 2 +
.../main/java/org/apache/doris/qe/Coordinator.java | 7 +--
.../java/org/apache/doris/qe/SessionVariable.java | 15 ++
.../java/org/apache/doris/qe/StmtExecutor.java | 9 +++-
.../org/apache/doris/qe/ConnectContextTest.java| 8 +++-
.../org/apache/doris/qe/SessionVariablesTest.java | 2 +
.../java/org/apache/doris/qe/VariableMgrTest.java | 3 ++
gensrc/thrift/PaloInternalService.thrift | 4 ++
.../suites/datev2/ssb_sf0.1_p1/load.groovy | 4 +-
.../suites/datev2/ssb_sf1_p2/load.groovy | 4 +-
regression-test/suites/ssb_sf0.1_p1/load.groovy| 4 +-
regression-test/suites/ssb_sf100_p2/load.groovy| 4 +-
regression-test/suites/ssb_sf1_p2/load.groovy | 4 +-
tools/ssb-tools/bin/load-ssb-data.sh | 7 +--
33 files changed, 129 insertions(+), 60 deletions(-)
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index c8db62c3b6..7ec1f5a21c 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -1159,7 +1159,7 @@ bool IRuntimeFilter::await() {
DCHECK(is_consumer());
// bitmap filter is precise filter and only filter once, so it must be
applied.
int64_t wait_times_ms = _wrapper->get_real_type() ==
RuntimeFilterType::BITMAP_FILTER
-? _state->query_options().query_timeout
+? _state->execution_timeout()
: _state->runtime_filter_wait_time_ms();
if (_state->enable_pipeline_exec()) {
auto expected = _rf_state_atomic.load(std::memory_order_acquire);
@@ -1209,7 +1209,7 @@ bool IRuntimeFilter::is_ready_or_timeout() {
auto cur_state = _rf_state_atomic.load(std::memory_order_acquire);
// bitmap filter is precise filter and only filter once, so it must be
applied.
int64_t wait_times_ms = _wrapper->get_real_type() ==
RuntimeFilterType::BITMAP_FILTER
-? _state->query_options().query_timeout
+? _state->execution_timeout()
: _state->runtime_filter_wait_time_ms();
int64_t ms_since_registration = MonotonicMillis() - registration_time_;
if (!_state->enable_pipeline_exec()) {
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index 654c252d58..c18f3e9866 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -192,7 +192,7 @@ FragmentExecState::FragmentExecState(const TUniqueId&
query_id,
Status FragmentExecState::prepare(const TExecPlanFragmentParams& params) {
if (params.__isset.query_options) {
-_timeout_second = params.query_options.query_timeout;
+_timeout_second = params.query_options.execution_timeout;
}
if (_fragments_ctx == nullptr) {
@@ -648,7 +648,7 @@ Status FragmentMgr::exec_plan_fragment(const
TExecPlanFragmentParams& params, Fi
fragments_ctx->get_shared_hash_table_controller()->set_pipeline_engine_enabled(
pipeline_engine_enabled);
-fragments_ctx->timeout_second = params.query_
[GitHub] [doris] ByteYue opened a new pull request, #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
ByteYue opened a new pull request, #16644: URL: https://github.com/apache/doris/pull/16644 # Proposed changes Issue Number: close #xxx When doing clone task, the logic would early exit if the missed version is empty. But the tablet meta is not set with the newly replica id inside the clone request. It would result inconsistent cooldown replica id when doing cooldown. ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] SaintBacchus opened a new issue, #16645: [Bug] Malformed packet error will query on es table
SaintBacchus opened a new issue, #16645: URL: https://github.com/apache/doris/issues/16645 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? https://user-images.githubusercontent.com/7404824/218301650-3a4de965-deea-46ee-960f-2439285ad55d.png";> When using a `select * limit 10` sql to query es table, I got these issue. ### What You Expected? Get right result ### How to Reproduce? I create a es table with the new version of DataTimeV2 and DecimalV3, and these may cuase these problem since BE had not impl these two type. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
github-actions[bot] commented on PR #16644: URL: https://github.com/apache/doris/pull/16644#issuecomment-1426979268 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
github-actions[bot] commented on PR #16644: URL: https://github.com/apache/doris/pull/16644#issuecomment-1426985470 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
github-actions[bot] commented on PR #16644: URL: https://github.com/apache/doris/pull/16644#issuecomment-1426985477 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16640: [Feature](array-function) Support array functions for nested type dec…
github-actions[bot] commented on PR #16640: URL: https://github.com/apache/doris/pull/16640#issuecomment-1426990378 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
github-actions[bot] commented on PR #16644: URL: https://github.com/apache/doris/pull/16644#issuecomment-1426991620 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on issue #16392: Doris Roadmap 2023
wangbo commented on issue #16392: URL: https://github.com/apache/doris/issues/16392#issuecomment-1426991990 How about link related PR/issue to the item in the list? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16640: [Feature](array-function) Support array functions for nested type dec…
github-actions[bot] commented on PR #16640: URL: https://github.com/apache/doris/pull/16640#issuecomment-1426994874 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
morningman commented on code in PR #16073:
URL: https://github.com/apache/doris/pull/16073#discussion_r1103793147
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/TokenManager.java:
##
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.load.loadv2;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ClientPool;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.thrift.FrontendService;
+import org.apache.doris.thrift.TMySqlLoadAcquireTokenResult;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.common.collect.EvictingQueue;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TException;
+import org.apache.thrift.transport.TTransportException;
+
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TokenManager {
+private static final Logger LOG = LogManager.getLogger(TokenManager.class);
+
+private final int thriftTimeoutMs = 300 * 1000;
+private final EvictingQueue tokenQueue;
+private final ScheduledExecutorService tokenGenerator;
+
+public TokenManager() {
+this.tokenQueue = EvictingQueue.create(Config.token_queue_size);
+// init one token to avoid async issue.
+this.tokenQueue.offer(generateNewToken());
+this.tokenGenerator = Executors.newScheduledThreadPool(1);
+this.tokenGenerator.scheduleAtFixedRate(() -> {
+tokenQueue.offer(generateNewToken());
+}, 0, Config.token_generate_period_hour, TimeUnit.HOURS);
+}
+
+private String generateNewToken() {
+return UUID.randomUUID().toString();
+}
+
+// this method only will be called in master node, since stream load only
send message to master.
+public boolean checkAuthToken(String token) {
+return tokenQueue.contains(token);
+}
+
+public String acquireToken() {
Review Comment:
Better throw exception when acquiring token failed, instead of returning
`null`.
It is every error prone and I saw some place which not handle `null` value.
##
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/TokenManager.java:
##
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.load.loadv2;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ClientPool;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.thrift.FrontendService;
+import org.apache.doris.thrift.TMySqlLoadAcquireTokenResult;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.common.collect.EvictingQueue;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TException;
+import org.apache.thrift.transport.TTransportException;
+
+import java.util.UUID;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TokenManager {
+private static final Logger LOG = LogManager.getLogger(TokenManager.class);
+
+private final int thriftTimeoutMs
[GitHub] [doris] caiconghui opened a new pull request, #16646: [fix](metric) fix be core when set enable_system_metrics to true in be
caiconghui opened a new pull request, #16646: URL: https://github.com/apache/doris/pull/16646 # Proposed changes Issue Number: close #xxx ## Problem summary when enable_system_metrics is false, we should not use system_metrics any more ## Checklist(Required) * [] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16646: [fix](metric) Fix be core when set enable_system_metrics to true in be
github-actions[bot] commented on PR #16646: URL: https://github.com/apache/doris/pull/16646#issuecomment-1427025018 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16646: [fix](metric) Fix be core when set enable_system_metrics to true in be
github-actions[bot] commented on PR #16646: URL: https://github.com/apache/doris/pull/16646#issuecomment-1427025028 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16646: [fix](metric) Fix be core when set enable_system_metrics to true in be
github-actions[bot] commented on PR #16646: URL: https://github.com/apache/doris/pull/16646#issuecomment-1427025033 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427027661 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427027668 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
github-actions[bot] commented on PR #16073: URL: https://github.com/apache/doris/pull/16073#issuecomment-1427029647 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16569: [Optimization](index) Optimization for no need to read raw data for index column that only in where clause
github-actions[bot] commented on PR #16569: URL: https://github.com/apache/doris/pull/16569#issuecomment-1427040894 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14063: [Feature](ipv6)Support IPV6
github-actions[bot] commented on PR #14063: URL: https://github.com/apache/doris/pull/14063#issuecomment-1427051793 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #14063: [Feature](ipv6)Support IPV6
github-actions[bot] commented on code in PR #14063:
URL: https://github.com/apache/doris/pull/14063#discussion_r1103816731
##
be/src/util/network_util.cpp:
##
@@ -39,22 +39,38 @@
namespace doris {
InetAddress::InetAddress(struct sockaddr* addr) {
-this->addr = *(struct sockaddr_in*)addr;
+this->addr = *(struct sockaddr_storage*)addr;
}
-bool InetAddress::is_address_v4() const {
-return addr.sin_family == AF_INET;
+bool InetAddress::is_loopback() {
+if (addr.ss_family == AF_INET) {
+in_addr_t s_addr = ((struct sockaddr_in*)&addr)->sin_addr.s_addr;
+return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+} else if (addr.ss_family == AF_INET6) {
+struct in6_addr in6_addr = ((struct sockaddr_in6*)&addr)->sin6_addr;
+return IN6_IS_ADDR_LOOPBACK(&in6_addr);
+} else {
+LOG(WARNING) << "unknow address: " << addr.ss_family;;
+return false;
+}
}
-bool InetAddress::is_loopback_v4() {
-in_addr_t s_addr = addr.sin_addr.s_addr;
-return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+std::string InetAddress::get_host_address() {
+if (addr.ss_family == AF_INET) {
+char addr_buf[INET_ADDRSTRLEN];
+inet_ntop(AF_INET, &(((struct sockaddr_in*)&addr)->sin_addr),
addr_buf, INET_ADDRSTRLEN);
+return std::string(addr_buf);
+} else if (addr.ss_family == AF_INET6) {
+char addr_buf[INET6_ADDRSTRLEN];
+inet_ntop(AF_INET6, &(((struct sockaddr_in6*)&addr)->sin6_addr),
addr_buf,
+ INET6_ADDRSTRLEN);
+return std::string(addr_buf);
+} else {
+return std::string {"unknown address: " + addr.ss_family};
Review Comment:
warning: adding 'sa_family_t' (aka 'unsigned short') to a string does not
append to the string [clang-diagnostic-string-plus-int]
```cpp
return std::string {"unknown address: " + addr.ss_family};
^
```
**be/src/util/network_util.cpp:68:** use array indexing to silence this
warning
```cpp
return std::string {"unknown address: " + addr.ss_family};
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caiconghui merged pull request #16646: [fix](metric) Fix be core when set enable_system_metrics to false in be
caiconghui merged PR #16646: URL: https://github.com/apache/doris/pull/16646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](metric) Fix be core when set enable_system_metrics to false in be (#16646)
This is an automated email from the ASF dual-hosted git repository.
caiconghui pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1de4e312cc [fix](metric) Fix be core when set enable_system_metrics to
false in be (#16646)
1de4e312cc is described below
commit 1de4e312cc7bcff70b5df08b99d185873e2e2c3b
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Sun Feb 12 23:01:41 2023 +0800
[fix](metric) Fix be core when set enable_system_metrics to false in be
(#16646)
when enable_system_metrics is false, we should not use system_metrics any
more
Co-authored-by: caiconghui1
---
be/src/common/configbase.cpp | 5 ++---
be/src/common/daemon.cpp | 4 +++-
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/be/src/common/configbase.cpp b/be/src/common/configbase.cpp
index bbce71e15f..8f9ca0e6ac 100644
--- a/be/src/common/configbase.cpp
+++ b/be/src/common/configbase.cpp
@@ -439,9 +439,8 @@ Status set_fuzzy_config(const std::string& field, const
std::string& value) {
void set_fuzzy_configs() {
// random value true or false
-Status s =
-set_fuzzy_config("disable_storage_page_cache", ((rand() % 2) == 0)
? "true" : "false");
-LOG(INFO) << s.to_string();
+set_fuzzy_config("disable_storage_page_cache", ((rand() % 2) == 0) ?
"true" : "false");
+set_fuzzy_config("enable_system_metrics", ((rand() % 2) == 0) ? "true" :
"false");
// random value from 8 to 48
// s = set_fuzzy_config("doris_scanner_thread_pool_thread_num",
std::to_string((rand() % 41) + 8));
// LOG(INFO) << s.to_string();
diff --git a/be/src/common/daemon.cpp b/be/src/common/daemon.cpp
index deadffc423..e393faa70f 100644
--- a/be/src/common/daemon.cpp
+++ b/be/src/common/daemon.cpp
@@ -181,7 +181,9 @@ void Daemon::memory_maintenance_thread() {
// Refresh allocator memory metrics.
#if !defined(ADDRESS_SANITIZER) && !defined(LEAK_SANITIZER) &&
!defined(THREAD_SANITIZER)
doris::MemInfo::refresh_allocator_mem();
-DorisMetrics::instance()->system_metrics()->update_allocator_metrics();
+if (config::enable_system_metrics) {
+
DorisMetrics::instance()->system_metrics()->update_allocator_metrics();
+}
#endif
doris::MemInfo::refresh_proc_mem_no_allocator_cache();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #14063: [Feature](ipv6)Support IPV6
github-actions[bot] commented on code in PR #14063:
URL: https://github.com/apache/doris/pull/14063#discussion_r1103818977
##
be/src/util/network_util.cpp:
##
@@ -39,22 +39,38 @@
namespace doris {
InetAddress::InetAddress(struct sockaddr* addr) {
-this->addr = *(struct sockaddr_in*)addr;
+this->addr = *(struct sockaddr_storage*)addr;
}
-bool InetAddress::is_address_v4() const {
-return addr.sin_family == AF_INET;
+bool InetAddress::is_loopback() {
+if (addr.ss_family == AF_INET) {
+in_addr_t s_addr = ((struct sockaddr_in*)&addr)->sin_addr.s_addr;
+return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+} else if (addr.ss_family == AF_INET6) {
+struct in6_addr in6_addr = ((struct sockaddr_in6*)&addr)->sin6_addr;
+return IN6_IS_ADDR_LOOPBACK(&in6_addr);
+} else {
+LOG(WARNING) << "unknown address family: " << addr.ss_family;
+return false;
+}
}
-bool InetAddress::is_loopback_v4() {
-in_addr_t s_addr = addr.sin_addr.s_addr;
-return (ntohl(s_addr) & 0xFF00) == 0x7F00;
+std::string InetAddress::get_host_address() {
+if (addr.ss_family == AF_INET) {
+char addr_buf[INET_ADDRSTRLEN];
+inet_ntop(AF_INET, &(((struct sockaddr_in*)&addr)->sin_addr),
addr_buf, INET_ADDRSTRLEN);
+return std::string(addr_buf);
+} else if (addr.ss_family == AF_INET6) {
+char addr_buf[INET6_ADDRSTRLEN];
+inet_ntop(AF_INET6, &(((struct sockaddr_in6*)&addr)->sin6_addr),
addr_buf,
+ INET6_ADDRSTRLEN);
+return std::string(addr_buf);
+} else {
+return std::string {"unknown address family: " + addr.ss_family};
Review Comment:
warning: adding 'sa_family_t' (aka 'unsigned short') to a string does not
append to the string [clang-diagnostic-string-plus-int]
```cpp
return std::string {"unknown address family: " + addr.ss_family};
^
```
**be/src/util/network_util.cpp:68:** use array indexing to silence this
warning
```cpp
return std::string {"unknown address family: " + addr.ss_family};
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16647: [fix](information_schema) catch and skip exception when getting schema from FE catalog
morningman opened a new pull request, #16647: URL: https://github.com/apache/doris/pull/16647 # Proposed changes Issue Number: close #xxx ## Problem summary When querying `information_schema` database, BE will call FE RPC to get schema info such as db name list, table name list, etc. But some external catalog when failed to get these info because of wrong connection info. We should catch these kind of exception and skip it, so that it can continue to get schema info of other catalogs. Otherwise, the whole query on `information_schema` will fail, even if user just want to get info of internal catalog. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427074590 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #16648: [improvement](dynamic-partition) add storage_medium property for dynamic partition
morningman opened a new pull request, #16648: URL: https://github.com/apache/doris/pull/16648 # Proposed changes Issue Number: close #xxx ## Problem summary Add new property "dynamic_partition.storage_medium" for creating dynamic partition table. Sometimes user may set all BE data dir with `SSD` suffix. And he can not create dynamic partition table because by default, the dynamic partition's storage medium is HDD. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16647: [fix](information_schema) catch and skip exception when getting schema from FE catalog
hello-stephen commented on PR #16647: URL: https://github.com/apache/doris/pull/16647#issuecomment-1427082673 TeamCity pipeline, clickbench performance test result: the sum of best hot time: 35.9 seconds stream load tsv: 478 seconds loaded 74807831229 Bytes, about 149 MB/s stream load json: 41 seconds loaded 2358488459 Bytes, about 54 MB/s stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230212170612_clickbench_pr_94815.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427086898 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on a diff in pull request #16641: [Bug](Cooldown) fix load balance causing no cooldown replica
platoneko commented on code in PR #16641:
URL: https://github.com/apache/doris/pull/16641#discussion_r1103844641
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -537,7 +539,8 @@ Status EngineCloneTask::_finish_incremental_clone(Tablet*
tablet,
const TabletMeta&
cloned_tablet_meta,
int64_t committed_version) {
LOG(INFO) << "begin to finish incremental clone. tablet=" <<
tablet->full_name()
- << ", committed_version=" << committed_version;
+ << ", committed_version=" << committed_version
+ << ", cloned_tablet_replica_id=" <<
cloned_tablet_meta.tablet_id();
Review Comment:
`tablet_id()` or `replica_id()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] platoneko commented on a diff in pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
platoneko commented on code in PR #16644:
URL: https://github.com/apache/doris/pull/16644#discussion_r1103845854
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -98,6 +98,12 @@ Status EngineCloneTask::_do_clone() {
if (missed_versions.empty()) {
LOG(INFO) << "missed version size = 0, skip clone and return
success. tablet_id="
<< _clone_req.tablet_id << " req replica=" <<
_clone_req.replica_id;
+// update replica id to meet cooldown replica
Review Comment:
Seems only `if (_clone_req.replica_id != tablet->replica_id())` should reset
replica id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427092956 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] closed pull request #10802: [enhancement][protocol] support proxy protocol
github-actions[bot] closed pull request #10802: [enhancement][protocol] support proxy protocol URL: https://github.com/apache/doris/pull/10802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 merged pull request #16477: [enhance](Nereids): refactor JoinReorder code.
starocean999 merged PR #16477: URL: https://github.com/apache/doris/pull/16477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16477: [enhance](Nereids): refactor JoinReorder code.
github-actions[bot] commented on PR #16477: URL: https://github.com/apache/doris/pull/16477#issuecomment-1427192249 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16477: [enhance](Nereids): refactor JoinReorder code.
github-actions[bot] commented on PR #16477: URL: https://github.com/apache/doris/pull/16477#issuecomment-1427192258 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhance](Nereids): refactor JoinReorder code. (#16477)
This is an automated email from the ASF dual-hosted git repository.
starocean999 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 80c1a99ef6 [enhance](Nereids): refactor JoinReorder code. (#16477)
80c1a99ef6 is described below
commit 80c1a99ef617a5ea9169ba01dc4c3848a49d82e0
Author: jakevin
AuthorDate: Mon Feb 13 09:08:58 2023 +0800
[enhance](Nereids): refactor JoinReorder code. (#16477)
* [enhance](Nereids): refactor JoinReorder code.
* apply nullable
* checkstyle
* set enableDPHypOptimizer default false
---
.../exploration/join/InnerJoinLAsscomProject.java | 102 +-
...oinReorderCommon.java => JoinReorderUtils.java} | 42 +-
.../exploration/join/OuterJoinLAsscomProject.java | 148 ++---
.../join/SemiJoinLogicalJoinTransposeProject.java | 12 +-
.../join/SemiJoinSemiJoinTransposeProject.java | 19 +--
.../org/apache/doris/nereids/util/JoinUtils.java | 18 +--
.../org/apache/doris/nereids/util/PlanUtils.java | 15 ---
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../join/InnerJoinLAsscomProjectTest.java | 4 +-
.../join/OuterJoinLAsscomProjectTest.java | 1 +
.../apache/doris/nereids/util/PlanUtilsTest.java | 19 ---
11 files changed, 132 insertions(+), 250 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
index ca8cb1a130..8a3fe670b0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/InnerJoinLAsscomProject.java
@@ -32,7 +32,6 @@ import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.nereids.util.JoinUtils;
-import org.apache.doris.nereids.util.PlanUtils;
import com.google.common.base.Preconditions;
@@ -64,7 +63,7 @@ public class InnerJoinLAsscomProject extends
OneExplorationRuleFactory {
return innerLogicalJoin(logicalProject(innerLogicalJoin()), group())
.when(topJoin -> InnerJoinLAsscom.checkReorder(topJoin,
topJoin.left().child()))
.whenNot(join -> join.hasJoinHint() ||
join.left().child().hasJoinHint())
-.when(join -> JoinReorderCommon.checkProject(join.left()))
+.when(join -> JoinReorderUtils.checkProject(join.left()))
.then(topJoin -> {
/* ** init ** */
@@ -74,41 +73,30 @@ public class InnerJoinLAsscomProject extends
OneExplorationRuleFactory {
GroupPlan b = bottomJoin.right();
GroupPlan c = topJoin.right();
Set cOutputSet = c.getOutputSet();
-Set bOutputExprIdSet = b.getOutputExprIdSet();
Set cOutputExprIdSet = c.getOutputExprIdSet();
/* ** Split projects ** */
-Map> projectExprsMap =
projects.stream()
-.collect(Collectors.partitioningBy(projectExpr -> {
-Set usedExprIds = projectExpr
-
.>collect(SlotReference.class::isInstance)
-.stream()
-.map(SlotReference::getExprId)
-.collect(Collectors.toSet());
-return
bOutputExprIdSet.containsAll(usedExprIds);
-}));
-List newLeftProjects =
projectExprsMap.get(Boolean.FALSE);
-List newRightProjects =
projectExprsMap.get(Boolean.TRUE);
-
-Set bExprIdSet =
getExprIdSetForB(bottomJoin.right(), newRightProjects);
+Map> map =
JoinReorderUtils.splitProjection(projects, b);
+List newLeftProjects = map.get(false);
+List newRightProjects = map.get(true);
+Set bExprIdSet =
JoinReorderUtils.combineProjectAndChildExprId(b, newRightProjects);
/* ** split HashConjuncts ** */
-Map> splitHashJoinConjuncts =
splitConjunctsWithAlias(
+Map> splitHashConjuncts =
splitConjunctsWithAlias(
topJoin.getHashJoinConjuncts(),
bottomJoin.getHashJoinConjuncts(), bExprIdSet);
-List newTopHashJoinConjuncts =
splitHashJoinConjuncts.get(true);
-List newBottomHashJoinConjuncts =
splitHashJoinConjuncts.get(fals
[GitHub] [doris] ByteYue commented on a diff in pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
ByteYue commented on code in PR #16644:
URL: https://github.com/apache/doris/pull/16644#discussion_r1103915390
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -98,6 +98,12 @@ Status EngineCloneTask::_do_clone() {
if (missed_versions.empty()) {
LOG(INFO) << "missed version size = 0, skip clone and return
success. tablet_id="
<< _clone_req.tablet_id << " req replica=" <<
_clone_req.replica_id;
+// update replica id to meet cooldown replica
Review Comment:
I thought each clone task had one new replica id. It seems
VERSION_INCOMPLETE task would has the same replica id. I'll go fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ByteYue commented on a diff in pull request #16641: [Bug](Cooldown) fix load balance causing no cooldown replica
ByteYue commented on code in PR #16641:
URL: https://github.com/apache/doris/pull/16641#discussion_r1103917449
##
be/src/olap/task/engine_clone_task.cpp:
##
@@ -537,7 +539,8 @@ Status EngineCloneTask::_finish_incremental_clone(Tablet*
tablet,
const TabletMeta&
cloned_tablet_meta,
int64_t committed_version) {
LOG(INFO) << "begin to finish incremental clone. tablet=" <<
tablet->full_name()
- << ", committed_version=" << committed_version;
+ << ", committed_version=" << committed_version
+ << ", cloned_tablet_replica_id=" <<
cloned_tablet_meta.tablet_id();
Review Comment:
fix it in #16644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #16617: ZhangShunjieNet interface
morningman commented on PR #16617: URL: https://github.com/apache/doris/pull/16617#issuecomment-1427198926 Please amend your PR title and add describe in your PR comment. And I suggest to wait #14063 merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16644: [Bug](cooldown) set new replica id when early exit in doing clone when no missed versions
github-actions[bot] commented on PR #16644: URL: https://github.com/apache/doris/pull/16644#issuecomment-1427198978 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg commented on pull request #16571: [Enhancement](HttpServer) Add http interface authentication
yangzhg commented on PR #16571: URL: https://github.com/apache/doris/pull/16571#issuecomment-1427210636 After supporting authentication and https, can the internal access to these ports in doris still work normally? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
morningman commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103927268
##
be/src/service/internal_service.cpp:
##
@@ -286,125 +334,134 @@ Status PInternalServiceImpl::_exec_plan_fragment(const
std::string& ser_request,
}
}
-void
PInternalServiceImpl::cancel_plan_fragment(google::protobuf::RpcController*
cntl_base,
+void
PInternalServiceImpl::cancel_plan_fragment(google::protobuf::RpcController*
controller,
const
PCancelPlanFragmentRequest* request,
PCancelPlanFragmentResult*
result,
google::protobuf::Closure*
done) {
-auto span = telemetry::start_rpc_server_span("exec_plan_fragment_start",
cntl_base);
-auto scope = OpentelemetryScope {span};
-brpc::ClosureGuard closure_guard(done);
-TUniqueId tid;
-tid.__set_hi(request->finst_id().hi());
-tid.__set_lo(request->finst_id().lo());
-
-Status st = Status::OK();
-if (request->has_cancel_reason()) {
-LOG(INFO) << "cancel fragment, fragment_instance_id=" << print_id(tid)
- << ", reason: " << request->cancel_reason();
-_exec_env->fragment_mgr()->cancel(tid, request->cancel_reason());
-} else {
-LOG(INFO) << "cancel fragment, fragment_instance_id=" << print_id(tid);
-_exec_env->fragment_mgr()->cancel(tid);
-}
-
-// TODO: the logic seems useless, cancel only return Status::OK. remove it
-st.to_protobuf(result->mutable_status());
+DorisMetrics::instance()->cancel_plan_fragment->increment(1);
+_light_work_pool.offer([this, controller, request, result, done]() {
+auto span =
telemetry::start_rpc_server_span("exec_plan_fragment_start", controller);
+auto scope = OpentelemetryScope {span};
+brpc::ClosureGuard closure_guard(done);
+TUniqueId tid;
+tid.__set_hi(request->finst_id().hi());
+tid.__set_lo(request->finst_id().lo());
+
+Status st = Status::OK();
+if (request->has_cancel_reason()) {
+LOG(INFO) << "cancel fragment, fragment_instance_id=" <<
print_id(tid)
+ << ", reason: " << request->cancel_reason();
+_exec_env->fragment_mgr()->cancel(tid, request->cancel_reason());
+} else {
+LOG(INFO) << "cancel fragment, fragment_instance_id=" <<
print_id(tid);
+_exec_env->fragment_mgr()->cancel(tid);
+}
+// TODO: the logic seems useless, cancel only return Status::OK.
remove it
+st.to_protobuf(result->mutable_status());
+});
}
-void PInternalServiceImpl::fetch_data(google::protobuf::RpcController*
cntl_base,
+void PInternalServiceImpl::fetch_data(google::protobuf::RpcController*
controller,
const PFetchDataRequest* request,
PFetchDataResult* result,
google::protobuf::Closure* done) {
-brpc::Controller* cntl = static_cast(cntl_base);
-GetResultBatchCtx* ctx = new GetResultBatchCtx(cntl, result, done);
-_exec_env->result_mgr()->fetch_data(request->finst_id(), ctx);
+DorisMetrics::instance()->fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, result, done]() {
+brpc::Controller* cntl = static_cast(controller);
+GetResultBatchCtx* ctx = new GetResultBatchCtx(cntl, result, done);
+_exec_env->result_mgr()->fetch_data(request->finst_id(), ctx);
+});
}
void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController*
controller,
const PFetchTableSchemaRequest*
request,
PFetchTableSchemaResult* result,
google::protobuf::Closure* done)
{
-VLOG_RPC << "fetch table schema";
-brpc::ClosureGuard closure_guard(done);
-TFileScanRange file_scan_range;
-Status st = Status::OK();
-{
-const uint8_t* buf = (const
uint8_t*)(request->file_scan_range().data());
-uint32_t len = request->file_scan_range().size();
-st = deserialize_thrift_msg(buf, &len, false, &file_scan_range);
+DorisMetrics::instance()->fetch_table_schema->increment(1);
+_light_work_pool.offer([request, result, done]() {
Review Comment:
This is a heavy work, it may read the remote file
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-
[GitHub] [doris] morningman commented on pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
morningman commented on PR #16639: URL: https://github.com/apache/doris/pull/16639#issuecomment-1427216658 Please also update this document: https://doris.apache.org/zh-CN/docs/dev/admin-manual/maint-monitor/monitor-metrics/metrics -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
morningman merged PR #16073: URL: https://github.com/apache/doris/pull/16073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (80c1a99ef6 -> f41a2055d3)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 80c1a99ef6 [enhance](Nereids): refactor JoinReorder code. (#16477)
add f41a2055d3 [feature](Load)Remove user/password in properties for mysql
load to avoid double auth. (#16073)
No new revisions were added by this update.
Summary of changes:
be/src/common/utils.h | 8 +-
be/src/http/utils.cpp | 23 ++--
be/src/runtime/fragment_mgr.cpp| 2 +-
.../main/java/org/apache/doris/common/Config.java | 13 ++
.../org/apache/doris/analysis/DataDescription.java | 10 +-
.../java/org/apache/doris/analysis/LoadStmt.java | 8 ++
.../org/apache/doris/load/loadv2/LoadManager.java | 16 +--
.../apache/doris/load/loadv2/MysqlLoadManager.java | 86 +++--
.../org/apache/doris/load/loadv2/TokenManager.java | 134 +
.../doris/load/sync/canal/CanalSyncChannel.java| 5 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 14 +--
.../apache/doris/service/FrontendServiceImpl.java | 59 +
.../apache/doris/transaction/TransactionState.java | 12 --
.../apache/doris/analysis/DataDescriptionTest.java | 4 +-
.../loadv2/TokenManagerTest.java} | 34 --
gensrc/thrift/FrontendService.thrift | 17 ++-
gensrc/thrift/PaloInternalService.thrift | 2 +-
.../load_p0/mysql_load/test_mysql_load.groovy | 10 +-
.../mysql_load/test_mysql_load_skip_lines.groovy | 12 +-
19 files changed, 323 insertions(+), 146 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/TokenManager.java
copy fe/fe-core/src/test/java/org/apache/doris/{bdb/BDBToolOptionsTest.java =>
load/loadv2/TokenManagerTest.java} (54%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
github-actions[bot] commented on PR #16073: URL: https://github.com/apache/doris/pull/16073#issuecomment-1427219077 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16073: [feature](Load)Remove user/password in properties for mysql load to avoid double auth.
github-actions[bot] commented on PR #16073: URL: https://github.com/apache/doris/pull/16073#issuecomment-1427219036 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed pull request #16583: [Feature](ES): Es support sort node pushdown limit and date type
stalary closed pull request #16583: [Feature](ES): Es support sort node pushdown limit and date type URL: https://github.com/apache/doris/pull/16583 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on pull request #16633: [Improvement](ES)Supprt datav2 and datetimev2 for es query
stalary commented on PR #16633: URL: https://github.com/apache/doris/pull/16633#issuecomment-1427223082 Can you add some regression-test case? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #16533: [Enhance](ComputeNode) ES Scan node support to be scheduled to compute node.
morningman commented on code in PR #16533:
URL: https://github.com/apache/doris/pull/16533#discussion_r1103931880
##
fe/fe-core/src/main/java/org/apache/doris/system/BeSelectionPolicy.java:
##
@@ -49,6 +49,8 @@ public class BeSelectionPolicy {
public boolean preferComputeNode = false;
public int candidateNum = Integer.MAX_VALUE;
+public List preLocations = new ArrayList<>();
Review Comment:
better name it `preferLocations`, `pre` always means `previous` or `prefix`
##
fe/fe-core/src/main/java/org/apache/doris/planner/EsScanNode.java:
##
@@ -231,39 +212,23 @@ private List getShardLocations()
throws UserException {
LOG.debug("partition prune finished, unpartitioned index [{}], " +
"partitioned index [{}]",
String.join(",", unPartitionedIndices), String.join(",",
partitionedIndices));
}
-int size = backendList.size();
-int beIndex = random.nextInt(size);
List result = Lists.newArrayList();
for (EsShardPartitions indexState : selectedIndex) {
for (List shardRouting :
indexState.getShardRoutings().values()) {
// get backends
-Set colocatedBes = Sets.newHashSet();
-int numBe = Math.min(3, size);
List shardAllocations = new ArrayList<>();
+List preLocations = new ArrayList<>();
for (EsShardRouting item : shardRouting) {
shardAllocations.add(item.getHttpAddress());
+preLocations.add(item.getHttpAddress().getHostname());
}
-Collections.shuffle(shardAllocations, random);
-for (TNetworkAddress address : shardAllocations) {
-colocatedBes.addAll(backendMap.get(address.getHostname()));
-}
-boolean usingRandomBackend = colocatedBes.size() == 0;
-List candidateBeList = Lists.newArrayList();
-if (usingRandomBackend) {
-for (int i = 0; i < numBe; ++i) {
-candidateBeList.add(backendList.get(beIndex++ % size));
-}
-} else {
-candidateBeList.addAll(colocatedBes);
-Collections.shuffle(candidateBeList);
-}
-
-// Locations
+BackendPolicy backendPolicy = new BackendPolicy();
Review Comment:
The `BackendPolicy` is `preferComputeNode` by default, which may be
confusing for other developer.
I think we should rename the `BackendPolicy` or explicitly set
`preferComputeNode()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
dataroaring commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103931956
##
be/src/util/doris_metrics.cpp:
##
@@ -197,6 +230,37 @@ DorisMetrics::DorisMetrics() :
_metric_registry(_s_registry_name) {
_server_metric_entity = _metric_registry.register_entity("server");
INT_COUNTER_METRIC_REGISTER(_server_metric_entity,
fragment_requests_total);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, transmit_data);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, transmit_data_by_http);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, exec_plan_fragment);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity,
exec_plan_fragment_prepare);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity,
exec_plan_fragment_start);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, cancel_plan_fragment);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, fetch_data);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, fetch_table_schema);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, tablet_writer_open);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity,
tablet_writer_add_block);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity,
tablet_writer_add_block_by_http);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, tablet_writer_cancel);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, get_info);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, update_cache);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, fetch_cache);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, clear_cache);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, merge_filter);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, apply_filter);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, transmit_block);
+INT_COUNTER_METRIC_REGISTER(_server_metric_entity, transmit_block_by_http);
Review Comment:
actually, there are these metrics in bvar in brpc already.
##
be/src/common/config.h:
##
@@ -35,8 +35,10 @@ CONF_Int32(be_port, "9060");
// port for brpc
CONF_Int32(brpc_port, "8060");
-// the number of bthreads for brpc, the default value is set to -1, which
means the number of bthreads is #cpu-cores
-CONF_Int32(brpc_num_threads, "-1");
+// the number of bthreads for brpc, the default value is set to 32
+// brpc only for network service send or accept request
+// no more process any logic
+CONF_Int32(brpc_num_threads, "32");
Review Comment:
We do not need to change default value here, e.g. if a user runs server on a
machine with 128 cores, it would break his/her config used.
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
+[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
+brpc::ClosureGuard guard(done);
+Status st = _tablet_fetch_data(request, response);
+st.to_protobuf(response->mutable_status());
+});
}
void PInternalServiceImpl::get_info(google::protobuf::RpcController*
controller,
const PProxyRequest* request,
PProxyResult* response,
google::protobuf::Closure* done) {
-brpc::ClosureGuard closure_guard(done);
-// PProxyRequest is defined in gensrc/proto/internal_service.proto
-// Currently it supports 2 kinds of requests:
-// 1. get all kafka partition ids for given topic
-// 2. get all kafka partition offsets for given topic and timestamp.
-if (request->has_kafka_meta_request()) {
-const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
-if (!kafka_request.partition_id_for_latest_offsets().empty()) {
-// get latest offsets for specified partition ids
-std::vector partition_offsets;
-Status st = _exec_env->routine_load_task_executor()
-->get_kafka_latest_offsets_for_partitions(
-request->kafka_meta_request(),
&partition_offsets);
-if (st.ok()) {
-PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
-for (const auto& entry : partition_offsets) {
-
[GitHub] [doris] morningman commented on a diff in pull request #16483: [Doc](Load)Add mysql load document
morningman commented on code in PR #16483:
URL: https://github.com/apache/doris/pull/16483#discussion_r1101221897
##
docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md:
##
@@ -0,0 +1,119 @@
+---
+{
+"title": "MySql load",
+"language": "zh-CN"
+}
+---
+
+
+
+
Review Comment:
这个要包裹文字，不是直接写。
参考：docs/zh-CN/docs/lakehouse/multi-catalog/multi-catalog.md
##
docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md:
##
@@ -0,0 +1,119 @@
+---
+{
+"title": "MySql load",
+"language": "zh-CN"
+}
+---
+
+
+
+
+
+# MySql load
+
+MySql load 是SQL交互的导入方式，用户通过SQL命令将客户端本地或者服务端本地的数据导入Doris 中。
Review Comment:
该语句兼容 MySQL 标准 [LOAD
DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html)
语法，方便用户导入本地数据，并降低学习成本。
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "MYSQL-LOAD",
+"language": "zh-CN"
+}
+---
+
+
+
+
+
+## MYSQL-LOAD
+
+### Name
+
+MYSQL LOAD
Review Comment:
```suggestion
MYSQL LOAD
```
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "MYSQL-LOAD",
+"language": "zh-CN"
+}
+---
+
+
+
+
+
+## MYSQL-LOAD
+
+### Name
+
+MYSQL LOAD
+
+### Description
+
+mysql-load: 使用MySql客户端导入本地数据
+
+```
+LOAD DATA
+[LOCAL]
+INFILE 'file_name'
+INTO TABLE tbl_name
+[PARTITION (partition_name [, partition_name] ...)]
+[COLUMNS TERMINATED BY 'string']
+[LINES TERMINATED BY 'string']
+[IGNORE number {LINES | ROWS}]
+[(col_name_or_user_var [, col_name_or_user_var] ...)]
+[SET (col_name={expr | DEFAULT} [, col_name={expr | DEFAULT}] ...)]
+[PROPERTIES (key1 = value1 [, key2=value2]) ]
+```
+
+该语句用于向指定的 table 导入数据，与普通Load区别是，这种导入方式是同步导入。
+
+这种导入方式仍然能够保证一批导入任务的原子性，要么全部数据导入成功，要么全部失败。
+
+1. MySQL Load以语法`LOAD DATA`开头, 无须指定LABEL
+2. 指定`LOCAL`表示读取客户端文件.不指定表示读取FE服务端本地文件
+3. `INFILE`内填写本地文件路径, 可以是相对路径, 也可以是绝对路径.目前只支持单个文件, 不支持多个文件
+4. `INTO TABLE`的表名可以指定数据库名, 如案例所示. 也可以省略, 则会使用当前用户所在的数据库.
+5. `PARTITION`语法支持指定分区导入
+6. `COLUMNS TERMINATED BY`指定列分隔符
+7. `LINES TERMINATED BY`指定行分隔符
+8. `IGNORE num LINES`用户跳过CSV的表头, 可以跳过任意行数. 该语法也可以用`IGNORE num ROWS`代替
+9. 列映射语法,
具体参数详见[导入的数据转换](../../../data-operate/import/import-way/mysql-load-manual.md)
的列映射章节
+10. `PROPERTIES`参数配置, 详见下文
+
+### PROPERTIES
+
+1. max_filter_ratio：最大容忍可过滤（数据不规范等原因）的数据比例。默认零容忍。
Review Comment:
```suggestion
1. `max_filter_ratio`：最大容忍可过滤（数据不规范等原因）的数据比例。默认零容忍。
```
Same for other properties
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "MYSQL-LOAD",
+"language": "zh-CN"
+}
+---
+
+
+
+
+
+## MYSQL-LOAD
+
+### Name
+
+MYSQL LOAD
+
+### Description
+
+mysql-load: 使用MySql客户端导入本地数据
+
+```
+LOAD DATA
+[LOCAL]
+INFILE 'file_name'
+INTO TABLE tbl_name
+[PARTITION (partition_name [, partition_name] ...)]
+[COLUMNS TERMINATED BY 'string']
+[LINES TERMINATED BY 'string']
+[IGNORE number {LINES | ROWS}]
+[(col_name_or_user_var [, col_name_or_user_var] ...)]
+[SET (col_name={expr | DEFAULT} [, col_name={expr | DEFAULT}] ...)]
+[PROPERTIES (key1 = value1 [, key2=value2]) ]
+```
+
+该语句用于向指定的 table 导入数据，与普通Load区别是，这种导入方式是同步导入。
+
+这种导入方式仍然能够保证一批导入任务的原子性，要么全部数据导入成功，要么全部失败。
+
+1. MySQL Load以语法`LOAD DATA`开头, 无须指定LABEL
+2. 指定`LOCAL`表示读取客户端文件.不指定表示读取FE服务端本地文件
+3. `INFILE`内填写本地文件路径, 可以是相对路径, 也可以是绝对路径.目前只支持单个文件, 不支持多个文件
+4. `INTO TABLE`的表名可以指定数据库名, 如案例所示. 也可以省略, 则会使用当前用户所在的数据库.
+5. `PARTITION`语法支持指定分区导入
+6. `COLUMNS TERMINATED BY`指定列分隔符
+7. `LINES TERMINATED BY`指定行分隔符
+8. `IGNORE num LINES`用户跳过CSV的表头, 可以跳过任意行数. 该语法也可以用`IGNORE num ROWS`代替
+9. 列映射语法,
具体参数详见[导入的数据转换](../../../data-operate/import/import-way/mysql-load-manual.md)
的列映射章节
+10. `PROPERTIES`参数配置, 详见下文
+
+### PROPERTIES
+
+1. max_filter_ratio：最大容忍可过滤（数据不规范等原因）的数据比例。默认零容忍。
+
+2. timeout: 指定导入的超时时间。单位秒。默认是 600 秒。可设置范围为 1 秒 ~ 259200 秒。
+
+3. strict_mode: 用户指定此次导入是否开启严格模式，默认为关闭。
+
+4. timezone: 指定本次导入所使用的时区。默认为东八区。该参数会影响所有导入涉及的和时区有关的函数结果。
+
+5. exec_mem_limit: 导入内存限制。默认为 2GB。单位为字节。
+
+### Example
+
+1. 将客户端本地文件'testData'中的数据导入到数据库'testDb'中'testTbl'的表。指定超时时间为 100 秒
+
+```sql
+LOAD DATA LOCAL
+INFILE 'testData'
+INTO TABLE testDb.testTbl
+PROPERTIES ("timeout"="100")
+```
+
+2. 将服务端本地文件'/root/testData'中的数据导入到数据库'testDb'中'testTbl'的表。指定超时时间为 100 秒
+
+```sql
Review Comment:
Use same indent level for all example
##
docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md:
##
@@ -0,0 +1,169 @@
+---
+{
+"title": "MYSQL-LOAD",
+"language": "zh-CN"
+}
+---
+
+
+
+
+
+## MYSQL-LOAD
+
+### Name
+
+MYSQL LOAD
+
+### Description
+
+mysql-load: 使用MySql客户端导入本地数据
+
+```
+LOAD DATA
+[LOCAL]
+INFILE 'file_name'
+INTO TABLE tbl_name
+[PARTITION (partition_name [, partition_name] ...)]
+[COLUMNS TERMINATED BY 'string']
+[L
[GitHub] [doris] yixiutt commented on a diff in pull request #16605: [bugfix](column_reader) index_page should not be pre-decoded
yixiutt commented on code in PR #16605:
URL: https://github.com/apache/doris/pull/16605#discussion_r1103935888
##
be/src/olap/rowset/segment_v2/column_reader.cpp:
##
@@ -210,6 +210,10 @@ Status ColumnReader::read_page(const
ColumnIteratorOptions& iter_opts, const Pag
opts.type = iter_opts.type;
opts.encoding_info = _encoding_info;
opts.io_ctx = iter_opts.io_ctx;
+// index page should not pre decode
+if (iter_opts.type == INDEX_PAGE) {
Review Comment:
it's a inner bug and logic here is two mistakes leads to running correnct,
we already have case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt commented on pull request #16605: [bugfix](column_reader) index_page should not be pre-decoded
yixiutt commented on PR #16605:
URL: https://github.com/apache/doris/pull/16605#issuecomment-1427231123
> Add a DCHECK for the type variable to make sure we used the right decoder?
>
> ```
> if constexpr (USED_IN_DICT_ENCODING) {
> auto type = decode_fixed32_le((const uint8_t*)&data.data[0]);
> if (static_cast(type) !=
EncodingTypePB::DICT_ENCODING) {
> return Status::OK();
> }
> size_of_dict_header = BINARY_DICT_PAGE_HEADER_SIZE;
> data.remove_prefix(4);
> }
> ```
only USED_IN_DICT_ENCODING have a type, other encoder may encode offset or
nums in the first bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
chenlinzhong commented on code in PR #16639: URL: https://github.com/apache/doris/pull/16639#discussion_r1103937266 ## be/src/common/config.h: ## @@ -35,8 +35,10 @@ CONF_Int32(be_port, "9060"); // port for brpc CONF_Int32(brpc_port, "8060"); -// the number of bthreads for brpc, the default value is set to -1, which means the number of bthreads is #cpu-cores -CONF_Int32(brpc_num_threads, "-1"); +// the number of bthreads for brpc, the default value is set to 32 +// brpc only for network service send or accept request +// no more process any logic +CONF_Int32(brpc_num_threads, "32"); Review Comment: brpc now only for network i/o, i think no need to configure so many threads -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] AshinGau opened a new pull request, #16649: [doc](filecache) add file cache document
AshinGau opened a new pull request, #16649: URL: https://github.com/apache/doris/pull/16649 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [x] Does it affect the original behavior * [x] Has unit tests been added * [-] Has document been added or modified * [x] Does it need to update dependencies * [x] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16610: [fix](compaction) if there is a rowset version hole, do not compaction
github-actions[bot] commented on PR #16610: URL: https://github.com/apache/doris/pull/16610#issuecomment-1427233760 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
chenlinzhong commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103938276
##
be/src/service/internal_service.cpp:
##
@@ -1097,9 +1208,8 @@ void
PInternalServiceImpl::multiget_data(google::protobuf::RpcController* contro
const PMultiGetRequest* request,
PMultiGetResponse* response,
google::protobuf::Closure* done) {
-// Submit task to seperate ThreadPool for avoiding block bthread working
pthread
-ThreadPool* task_pool =
StorageEngine::instance()->get_bg_multiget_threadpool();
-Status submit_st = task_pool->submit_func([request, response, done,
this]() {
+DorisMetrics::instance()->multiget_data->increment(1);
+_light_work_pool.offer([request, response, done, this]() {
Review Comment:
ok
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
+[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
+brpc::ClosureGuard guard(done);
+Status st = _tablet_fetch_data(request, response);
+st.to_protobuf(response->mutable_status());
+});
}
void PInternalServiceImpl::get_info(google::protobuf::RpcController*
controller,
const PProxyRequest* request,
PProxyResult* response,
google::protobuf::Closure* done) {
-brpc::ClosureGuard closure_guard(done);
-// PProxyRequest is defined in gensrc/proto/internal_service.proto
-// Currently it supports 2 kinds of requests:
-// 1. get all kafka partition ids for given topic
-// 2. get all kafka partition offsets for given topic and timestamp.
-if (request->has_kafka_meta_request()) {
-const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
-if (!kafka_request.partition_id_for_latest_offsets().empty()) {
-// get latest offsets for specified partition ids
-std::vector partition_offsets;
-Status st = _exec_env->routine_load_task_executor()
-->get_kafka_latest_offsets_for_partitions(
-request->kafka_meta_request(),
&partition_offsets);
-if (st.ok()) {
-PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
-for (const auto& entry : partition_offsets) {
-PIntegerPair* res = part_offsets->add_offset_times();
-res->set_key(entry.key());
-res->set_val(entry.val());
+DorisMetrics::instance()->get_info->increment(1);
+_light_work_pool.offer([this, request, response, done]() {
+brpc::ClosureGuard closure_guard(done);
+// PProxyRequest is defined in gensrc/proto/internal_service.proto
+// Currently it supports 2 kinds of requests:
+// 1. get all kafka partition ids for given topic
+// 2. get all kafka partition offsets for given topic and timestamp.
+if (request->has_kafka_meta_request()) {
+const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
+if (!kafka_request.partition_id_for_latest_offsets().empty()) {
+// get latest offsets for specified partition ids
+std::vector partition_offsets;
+Status st = _exec_env->routine_load_task_executor()
+->get_kafka_latest_offsets_for_partitions(
+request->kafka_meta_request(),
&partition_offsets);
+if (st.ok()) {
+PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
+for (const auto& entry : partition_offsets) {
+PIntegerPair* res = part_offsets->add_offset_times();
+res->set_key(entry.key());
+res->set_val(entry.val());
+}
}
-}
-st.to_protobuf(response->mutable_status());
-return;
-} else if (!kafka_request.offset_times().empty()) {
-// if offset
[GitHub] [doris] chenlinzhong commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
chenlinzhong commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103938347
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
+[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
+brpc::ClosureGuard guard(done);
+Status st = _tablet_fetch_data(request, response);
+st.to_protobuf(response->mutable_status());
+});
}
void PInternalServiceImpl::get_info(google::protobuf::RpcController*
controller,
const PProxyRequest* request,
PProxyResult* response,
google::protobuf::Closure* done) {
-brpc::ClosureGuard closure_guard(done);
-// PProxyRequest is defined in gensrc/proto/internal_service.proto
-// Currently it supports 2 kinds of requests:
-// 1. get all kafka partition ids for given topic
-// 2. get all kafka partition offsets for given topic and timestamp.
-if (request->has_kafka_meta_request()) {
-const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
-if (!kafka_request.partition_id_for_latest_offsets().empty()) {
-// get latest offsets for specified partition ids
-std::vector partition_offsets;
-Status st = _exec_env->routine_load_task_executor()
-->get_kafka_latest_offsets_for_partitions(
-request->kafka_meta_request(),
&partition_offsets);
-if (st.ok()) {
-PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
-for (const auto& entry : partition_offsets) {
-PIntegerPair* res = part_offsets->add_offset_times();
-res->set_key(entry.key());
-res->set_val(entry.val());
+DorisMetrics::instance()->get_info->increment(1);
+_light_work_pool.offer([this, request, response, done]() {
Review Comment:
ok
##
be/src/service/internal_service.cpp:
##
@@ -286,125 +334,134 @@ Status PInternalServiceImpl::_exec_plan_fragment(const
std::string& ser_request,
}
}
-void
PInternalServiceImpl::cancel_plan_fragment(google::protobuf::RpcController*
cntl_base,
+void
PInternalServiceImpl::cancel_plan_fragment(google::protobuf::RpcController*
controller,
const
PCancelPlanFragmentRequest* request,
PCancelPlanFragmentResult*
result,
google::protobuf::Closure*
done) {
-auto span = telemetry::start_rpc_server_span("exec_plan_fragment_start",
cntl_base);
-auto scope = OpentelemetryScope {span};
-brpc::ClosureGuard closure_guard(done);
-TUniqueId tid;
-tid.__set_hi(request->finst_id().hi());
-tid.__set_lo(request->finst_id().lo());
-
-Status st = Status::OK();
-if (request->has_cancel_reason()) {
-LOG(INFO) << "cancel fragment, fragment_instance_id=" << print_id(tid)
- << ", reason: " << request->cancel_reason();
-_exec_env->fragment_mgr()->cancel(tid, request->cancel_reason());
-} else {
-LOG(INFO) << "cancel fragment, fragment_instance_id=" << print_id(tid);
-_exec_env->fragment_mgr()->cancel(tid);
-}
-
-// TODO: the logic seems useless, cancel only return Status::OK. remove it
-st.to_protobuf(result->mutable_status());
+DorisMetrics::instance()->cancel_plan_fragment->increment(1);
+_light_work_pool.offer([this, controller, request, result, done]() {
+auto span =
telemetry::start_rpc_server_span("exec_plan_fragment_start", controller);
+auto scope = OpentelemetryScope {span};
+brpc::ClosureGuard closure_guard(done);
+TUniqueId tid;
+tid.__set_hi(request->finst_id().hi());
+tid.__set_lo(request->finst_id().lo());
+
+Status st = Status::OK();
+if (request->has_cancel_reason()) {
+LOG(INFO) << "cancel fragment, fragment_instance_id=" <<
print_id(tid)
+ << ", reason: " << request->cancel_reason();
+_exec_env->fragment_mgr()->cancel(tid, request->cancel_reason());
+
[GitHub] [doris] SaintBacchus commented on a diff in pull request #16533: [Enhance](ComputeNode) ES Scan node support to be scheduled to compute node.
SaintBacchus commented on code in PR #16533:
URL: https://github.com/apache/doris/pull/16533#discussion_r1103938399
##
fe/fe-core/src/main/java/org/apache/doris/system/BeSelectionPolicy.java:
##
@@ -49,6 +49,8 @@ public class BeSelectionPolicy {
public boolean preferComputeNode = false;
public int candidateNum = Integer.MAX_VALUE;
+public List preLocations = new ArrayList<>();
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
eldenmoon commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103940279
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
Review Comment:
please Do not put this RPC interface into thread pool, it's very light
weight and can't be mixed with other rpc interface, since it's latency sensitive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #16615: [Feature-WIP](inverted index) step 1 for supporting range predicate pushing down to inverted index
dataroaring merged PR #16615: URL: https://github.com/apache/doris/pull/16615 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee opened a new pull request, #16650: [Opt](Hash) Deduce virtual function call is null at in single nullable column
HappenLee opened a new pull request, #16650: URL: https://github.com/apache/doris/pull/16650 # Proposed changes Remove unless virtual function call in `HashMethodSingleLowNullableColumn` Before： Query group by in clickbench 0.82s After： Query group by in clickbench 0.76s ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yongjinhou commented on pull request #16571: [Enhancement](HttpServer) Add http interface authentication
yongjinhou commented on PR #16571: URL: https://github.com/apache/doris/pull/16571#issuecomment-1427239627 > maybe add some test case is more better. Currently, there are no unit tests for http interface, I'll add one later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (f41a2055d3 -> 91c4d1cade)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f41a2055d3 [feature](Load)Remove user/password in properties for mysql load to avoid double auth. (#16073) add 91c4d1cade [Feature-WIP](inverted index) step 1 for supporting range predicate pushing down to inverted index (#16615) No new revisions were added by this update. Summary of changes: be/src/olap/block_column_predicate.cpp | 7 + be/src/olap/block_column_predicate.h | 10 ++ be/src/olap/rowset/segment_v2/segment_iterator.cpp | 166 +++-- be/src/olap/rowset/segment_v2/segment_iterator.h | 10 +- 4 files changed, 148 insertions(+), 45 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14063: [Feature](ipv6)Support IPV6
github-actions[bot] commented on PR #14063: URL: https://github.com/apache/doris/pull/14063#issuecomment-1427240495 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16650: [Opt](Hash) Deduce virtual function call is null at in single nullable column
github-actions[bot] commented on PR #16650: URL: https://github.com/apache/doris/pull/16650#issuecomment-1427242086 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yixiutt commented on a diff in pull request #16607: [fix](engine_clone_task) disable compact when tablet version discontinuous to avoid engine clone task failure
yixiutt commented on code in PR #16607: URL: https://github.com/apache/doris/pull/16607#discussion_r1103944601 ## be/src/common/config.h: ## @@ -270,6 +270,10 @@ CONF_Bool(enable_low_cardinality_optimize, "true"); CONF_mBool(enable_compaction_checksum, "false"); // whether disable automatic compaction task CONF_mBool(disable_auto_compaction, "false"); +// whether disable compact when there is a hole in tablet versions +CONF_mBool(disable_compact_tablet_version_discontinuous, "true"); +// when version incontinuous, if tablet versions exceed `this ratio * max_tablet_version_num`, allow tablet compaction +CONF_mDouble(allow_compact_tablet_discontinuous_max_rowset_num_ratio, "0.7"); Review Comment: may be smaller? 0.4 and 200 versions is big enough -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LemonLiTree opened a new pull request, #16651: [typo](docs) fix doc
LemonLiTree opened a new pull request, #16651: URL: https://github.com/apache/doris/pull/16651 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16651: [typo](docs) fix doc
github-actions[bot] commented on PR #16651: URL: https://github.com/apache/doris/pull/16651#issuecomment-1427250572 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16651: [typo](docs) fix doc
github-actions[bot] commented on PR #16651: URL: https://github.com/apache/doris/pull/16651#issuecomment-1427250521 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #16652: [improvement](lock raii) use raii to lock and unlock
yiguolei opened a new pull request, #16652: URL: https://github.com/apache/doris/pull/16652 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) * [ ] Does it affect the original behavior * [ ] Has unit tests been added * [ ] Has document been added or modified * [ ] Does it need to update dependencies * [ ] Is this PR support rollback (If NO, please explain WHY) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16652: [improvement](lock raii) use raii to lock and unlock
github-actions[bot] commented on PR #16652: URL: https://github.com/apache/doris/pull/16652#issuecomment-1427262424 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16588: [feature](cooldown) Auto delete unused remote files
github-actions[bot] commented on PR #16588: URL: https://github.com/apache/doris/pull/16588#issuecomment-1427264127 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #16650: [Opt](Hash) Deduce virtual function call is null at in single nullable column
BiteThet commented on code in PR #16650:
URL: https://github.com/apache/doris/pull/16650#discussion_r1103953424
##
be/src/vec/common/columns_hashing.h:
##
@@ -227,11 +227,11 @@ struct HashMethodSingleLowNullableColumn : public
SingleColumnMethod {
HashMethodSingleLowNullableColumn(const ColumnRawPtrs&
key_columns_nullable,
const Sizes& key_sizes, const
HashMethodContextPtr& context)
: Base(get_nested_column(key_columns_nullable[0]), key_sizes,
context),
- key_columns(key_columns_nullable) {}
+ key_column(reinterpret_cast(key_columns_nullable[0])) {}
Review Comment:
Maybe it is better to use `assert_cast` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16612: [fix](nereids)make slot binding compatible to original planner
github-actions[bot] commented on PR #16612: URL: https://github.com/apache/doris/pull/16612#issuecomment-1427267419 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16612: [fix](nereids)make slot binding compatible to original planner
github-actions[bot] commented on PR #16612: URL: https://github.com/apache/doris/pull/16612#issuecomment-1427267380 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
chenlinzhong commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103954749
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
+[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
+brpc::ClosureGuard guard(done);
+Status st = _tablet_fetch_data(request, response);
+st.to_protobuf(response->mutable_status());
+});
}
void PInternalServiceImpl::get_info(google::protobuf::RpcController*
controller,
const PProxyRequest* request,
PProxyResult* response,
google::protobuf::Closure* done) {
-brpc::ClosureGuard closure_guard(done);
-// PProxyRequest is defined in gensrc/proto/internal_service.proto
-// Currently it supports 2 kinds of requests:
-// 1. get all kafka partition ids for given topic
-// 2. get all kafka partition offsets for given topic and timestamp.
-if (request->has_kafka_meta_request()) {
-const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
-if (!kafka_request.partition_id_for_latest_offsets().empty()) {
-// get latest offsets for specified partition ids
-std::vector partition_offsets;
-Status st = _exec_env->routine_load_task_executor()
-->get_kafka_latest_offsets_for_partitions(
-request->kafka_meta_request(),
&partition_offsets);
-if (st.ok()) {
-PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
-for (const auto& entry : partition_offsets) {
-PIntegerPair* res = part_offsets->add_offset_times();
-res->set_key(entry.key());
-res->set_val(entry.val());
+DorisMetrics::instance()->get_info->increment(1);
+_light_work_pool.offer([this, request, response, done]() {
+brpc::ClosureGuard closure_guard(done);
+// PProxyRequest is defined in gensrc/proto/internal_service.proto
+// Currently it supports 2 kinds of requests:
+// 1. get all kafka partition ids for given topic
+// 2. get all kafka partition offsets for given topic and timestamp.
+if (request->has_kafka_meta_request()) {
+const PKafkaMetaProxyRequest& kafka_request =
request->kafka_meta_request();
+if (!kafka_request.partition_id_for_latest_offsets().empty()) {
+// get latest offsets for specified partition ids
+std::vector partition_offsets;
+Status st = _exec_env->routine_load_task_executor()
+->get_kafka_latest_offsets_for_partitions(
+request->kafka_meta_request(),
&partition_offsets);
+if (st.ok()) {
+PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
+for (const auto& entry : partition_offsets) {
+PIntegerPair* res = part_offsets->add_offset_times();
+res->set_key(entry.key());
+res->set_val(entry.val());
+}
}
-}
-st.to_protobuf(response->mutable_status());
-return;
-} else if (!kafka_request.offset_times().empty()) {
-// if offset_times() has elements, which means this request is to
get offset by timestamp.
-std::vector partition_offsets;
-Status st =
-
_exec_env->routine_load_task_executor()->get_kafka_partition_offsets_for_times(
-request->kafka_meta_request(), &partition_offsets);
-if (st.ok()) {
-PKafkaPartitionOffsets* part_offsets =
response->mutable_partition_offsets();
-for (const auto& entry : partition_offsets) {
-PIntegerPair* res = part_offsets->add_offset_times();
-res->set_key(entry.key());
-res->set_val(entry.val());
+st.to_protobuf(response->mutable_status());
+return;
+} els
[GitHub] [doris] github-actions[bot] commented on pull request #16652: [improvement](lock raii) use raii to lock and unlock
github-actions[bot] commented on PR #16652: URL: https://github.com/apache/doris/pull/16652#issuecomment-1427270720 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16652: [improvement](lock raii) use raii to lock and unlock
github-actions[bot] commented on PR #16652: URL: https://github.com/apache/doris/pull/16652#issuecomment-1427270750 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16612: [fix](nereids)make slot binding compatible to original planner
morrySnow merged PR #16612: URL: https://github.com/apache/doris/pull/16612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (91c4d1cade -> 46dd887ae2)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 91c4d1cade [Feature-WIP](inverted index) step 1 for supporting range predicate pushing down to inverted index (#16615) add 46dd887ae2 [fix](nereids) make slot binding compatible to original planner (#16612) No new revisions were added by this update. Summary of changes: .../nereids/rules/analysis/BindExpression.java | 32 --- .../doris/nereids/rules/analysis/SlotBinder.java | 48 +++--- .../suites/nereids_syntax_p0/bind_priority.groovy | 5 +++ 3 files changed, 56 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] chenlinzhong commented on a diff in pull request #16639: [Improvement](brpc) replace brpc bthread with pthread
chenlinzhong commented on code in PR #16639:
URL: https://github.com/apache/doris/pull/16639#discussion_r1103956391
##
be/src/service/internal_service.cpp:
##
@@ -423,200 +480,233 @@ void
PInternalServiceImpl::tablet_fetch_data(google::protobuf::RpcController* co
const PTabletKeyLookupRequest*
request,
PTabletKeyLookupResponse*
response,
google::protobuf::Closure* done) {
-[[maybe_unused]] brpc::Controller* cntl =
static_cast(controller);
-brpc::ClosureGuard guard(done);
-Status st = _tablet_fetch_data(request, response);
-st.to_protobuf(response->mutable_status());
+DorisMetrics::instance()->tablet_fetch_data->increment(1);
+_heavy_work_pool.offer([this, controller, request, response, done]() {
Review Comment:
before this pr ,this rpc interface also mixed with each other
i don't think put into thread pool will effect latency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
