[GitHub] [incubator-doris] morningman merged pull request #8174: [Enhancement][Vectorized] support es node predicate peel
morningman merged pull request #8174: URL: https://github.com/apache/incubator-doris/pull/8174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8173: [Enhancement][Vectorized] support es node predicate peel
morningman closed issue #8173: URL: https://github.com/apache/incubator-doris/issues/8173 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [improvement][vectorized] support es node predicate peel (#8174)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 668188b [improvement][vectorized] support es node predicate peel
(#8174)
668188b is described below
commit 668188b91f5c2a9d27ad257fc409f1df11aa0c17
Author: Pxl <952130...@qq.com>
AuthorDate: Sat Feb 26 17:02:54 2022 +0800
[improvement][vectorized] support es node predicate peel (#8174)
---
be/src/exec/es/es_predicate.cpp| 2 +-
be/src/exec/es/es_predicate.h | 4 +-
be/src/exec/es/es_query_builder.cpp| 121 ++---
be/src/exec/es/es_query_builder.h | 1 +
be/src/exec/es_http_scan_node.cpp | 24 +++-
be/src/exec/es_http_scan_node.h| 10 +-
be/src/exec/olap_scan_node.cpp | 58 ++
be/src/exec/olap_scan_node.h | 14 +--
be/src/exec/scan_node.cpp | 28 +
be/src/exec/scan_node.h| 9 +-
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/exec/ves_http_scanner.cpp | 8 +-
be/src/vec/exec/ves_http_scanner.h | 24 ++--
be/src/vec/functions/function_convert_tz.h | 2 +
.../function_date_or_datetime_computation.h| 2 +
.../function_date_or_datetime_to_something.h | 2 +
.../function_date_or_datetime_to_string.h | 2 +
.../functions/function_datetime_string_to_string.h | 2 +
.../{function_hash.h => function_fake.cpp} | 22 +---
...{function_multi_same_args.h => function_fake.h} | 36 +++---
be/src/vec/functions/function_grouping.h | 10 +-
be/src/vec/functions/function_hash.h | 1 +
be/src/vec/functions/function_ifnull.h | 49 -
be/src/vec/functions/function_multi_same_args.h| 4 +-
be/src/vec/functions/function_rpc.h| 2 +
be/src/vec/functions/function_string_to_string.h | 2 +
be/src/vec/functions/simple_function_factory.h | 3 +
be/src/vec/utils/util.hpp | 26 -
gensrc/script/doris_builtins_functions.py | 2 +-
29 files changed, 244 insertions(+), 227 deletions(-)
diff --git a/be/src/exec/es/es_predicate.cpp b/be/src/exec/es/es_predicate.cpp
index 3b8be66..7dab9ca 100644
--- a/be/src/exec/es/es_predicate.cpp
+++ b/be/src/exec/es/es_predicate.cpp
@@ -181,7 +181,7 @@ Status EsPredicate::build_disjuncts_list() {
}
// make sure to build by build_disjuncts_list
-const std::vector& EsPredicate::get_predicate_list() {
+const std::vector& EsPredicate::get_predicate_list() const {
return _disjuncts;
}
diff --git a/be/src/exec/es/es_predicate.h b/be/src/exec/es/es_predicate.h
index 825bc5e..826af0a 100644
--- a/be/src/exec/es/es_predicate.h
+++ b/be/src/exec/es/es_predicate.h
@@ -141,10 +141,10 @@ class EsPredicate {
public:
EsPredicate(ExprContext* context, const TupleDescriptor* tuple_desc,
ObjectPool* pool);
~EsPredicate();
-const std::vector& get_predicate_list();
+const std::vector& get_predicate_list() const;
Status build_disjuncts_list();
// public for tests
-EsPredicate(const std::vector& all_predicates) { _disjuncts
= all_predicates; };
+EsPredicate(const std::vector& all_predicates) { _disjuncts
= all_predicates; }
Status get_es_query_status() { return _es_query_status; }
diff --git a/be/src/exec/es/es_query_builder.cpp
b/be/src/exec/es/es_query_builder.cpp
index 5f8dcfd..1f9488e 100644
--- a/be/src/exec/es/es_query_builder.cpp
+++ b/be/src/exec/es/es_query_builder.cpp
@@ -54,19 +54,19 @@ TermQueryBuilder::TermQueryBuilder(const std::string&
field, const std::string&
TermQueryBuilder::TermQueryBuilder(const ExtBinaryPredicate& binary_predicate)
: _field(binary_predicate.col.name), _match_none(false) {
-if (binary_predicate.col.type.type == PrimitiveType::TYPE_BOOLEAN) {
-int val = atoi(binary_predicate.value.to_string().c_str());
-if (val == 1) {
-_term = std::string("true");
-} else if (val == 0){
-_term = std::string("false");
-} else {
-// keep semantic consistent with mysql
-_match_none = true;
-}
+if (binary_predicate.col.type.type == PrimitiveType::TYPE_BOOLEAN) {
+int val = atoi(binary_predicate.value.to_string().c_str());
+if (val == 1) {
+_term = std::string("true");
+} else if (val == 0) {
+_term = std::string("false");
} else {
-_term = binary_predicate.value.to_string();
+// keep semantic consistent with mysql
+_match_none = true;
}
+} else {
+_term = bin
[GitHub] [incubator-doris] morningman closed issue #8178: [Feature] Support query Iceberg table
morningman closed issue #8178: URL: https://github.com/apache/incubator-doris/issues/8178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [feature](iceberg) Step3: Support query iceberg external table (#8179)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 87b96cf [feature](iceberg) Step3: Support query iceberg external
table (#8179)
87b96cf is described below
commit 87b96cfcd6c3cdecb7441f89e208b1f77177182c
Author: qiye
AuthorDate: Sat Feb 26 17:04:11 2022 +0800
[feature](iceberg) Step3: Support query iceberg external table (#8179)
1. Add Iceberg scan node
2. Add Iceberg/Hive table type in thrift
3. Support querying Iceberg tables of format types `parquet` and `orc`
---
be/src/runtime/descriptors.cpp | 28
be/src/runtime/descriptors.h | 18 +++
fe/fe-core/pom.xml | 8 ++
.../java/org/apache/doris/catalog/HiveTable.java | 2 +-
.../org/apache/doris/catalog/IcebergTable.java | 159 ++---
.../org/apache/doris/load/BrokerFileGroup.java | 11 ++
.../java/org/apache/doris/load/LoadChecker.java| 2 +-
.../org/apache/doris/planner/BrokerScanNode.java | 28 ++--
.../apache/doris/planner/DistributedPlanner.java | 2 +-
.../org/apache/doris/planner/HiveScanNode.java | 2 +-
.../org/apache/doris/planner/IcebergScanNode.java | 109 ++
.../apache/doris/planner/SingleNodePlanner.java| 4 +
.../java/org/apache/doris/qe/AuditLogBuilder.java | 5 +-
.../apache/doris/catalog/CreateTableLikeTest.java | 7 +-
.../org/apache/doris/plugin/PluginMgrTest.java | 3 +-
fe/pom.xml | 12 ++
gensrc/thrift/Types.thrift | 4 +-
17 files changed, 362 insertions(+), 42 deletions(-)
diff --git a/be/src/runtime/descriptors.cpp b/be/src/runtime/descriptors.cpp
index 162545a..4225a75 100644
--- a/be/src/runtime/descriptors.cpp
+++ b/be/src/runtime/descriptors.cpp
@@ -147,6 +147,28 @@ std::string BrokerTableDescriptor::debug_string() const {
return out.str();
}
+HiveTableDescriptor::HiveTableDescriptor(const TTableDescriptor& tdesc)
+: TableDescriptor(tdesc) {}
+
+HiveTableDescriptor::~HiveTableDescriptor() {}
+
+std::string HiveTableDescriptor::debug_string() const {
+std::stringstream out;
+out << "HiveTable(" << TableDescriptor::debug_string() << ")";
+return out.str();
+}
+
+IcebergTableDescriptor::IcebergTableDescriptor(const TTableDescriptor& tdesc)
+: TableDescriptor(tdesc) {}
+
+IcebergTableDescriptor::~IcebergTableDescriptor() {}
+
+std::string IcebergTableDescriptor::debug_string() const {
+std::stringstream out;
+out << "IcebergTable(" << TableDescriptor::debug_string() << ")";
+return out.str();
+}
+
EsTableDescriptor::EsTableDescriptor(const TTableDescriptor& tdesc) :
TableDescriptor(tdesc) {}
EsTableDescriptor::~EsTableDescriptor() {}
@@ -523,6 +545,12 @@ Status DescriptorTbl::create(ObjectPool* pool, const
TDescriptorTable& thrift_tb
case TTableType::ES_TABLE:
desc = pool->add(new EsTableDescriptor(tdesc));
break;
+case TTableType::HIVE_TABLE:
+desc = pool->add(new HiveTableDescriptor(tdesc));
+break;
+case TTableType::ICEBERG_TABLE:
+desc = pool->add(new IcebergTableDescriptor(tdesc));
+break;
default:
DCHECK(false) << "invalid table type: " << tdesc.tableType;
}
diff --git a/be/src/runtime/descriptors.h b/be/src/runtime/descriptors.h
index 0bdc22b..cdbc3d7 100644
--- a/be/src/runtime/descriptors.h
+++ b/be/src/runtime/descriptors.h
@@ -190,6 +190,24 @@ public:
private:
};
+class HiveTableDescriptor : public TableDescriptor {
+public:
+HiveTableDescriptor(const TTableDescriptor& tdesc);
+virtual ~HiveTableDescriptor();
+virtual std::string debug_string() const;
+
+private:
+};
+
+class IcebergTableDescriptor : public TableDescriptor {
+public:
+IcebergTableDescriptor(const TTableDescriptor& tdesc);
+virtual ~IcebergTableDescriptor();
+virtual std::string debug_string() const;
+
+private:
+};
+
class EsTableDescriptor : public TableDescriptor {
public:
EsTableDescriptor(const TTableDescriptor& tdesc);
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 9b11108..a71545c 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -599,6 +599,14 @@ under the License.
iceberg-hive-metastore
provided
+
+
+
+org.apache.avro
+avro
+provided
+
+
palo-fe
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
index 236d266..ff4adc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris
[GitHub] [incubator-doris] morningman merged pull request #8179: [feature-wip](iceberg) Step3: Support query iceberg external table
morningman merged pull request #8179: URL: https://github.com/apache/incubator-doris/pull/8179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8195: [ifx](routine-load) fix show routine load task error
morningman merged pull request #8195: URL: https://github.com/apache/incubator-doris/pull/8195 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #8194: [Bug] [routine load] the tnxStatus of "show routine load task" result are not correct
morningman closed issue #8194: URL: https://github.com/apache/incubator-doris/issues/8194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](routine-load) fix show routine load task error (#8195)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 385ccf7 [fix](routine-load) fix show routine load task error (#8195)
385ccf7 is described below
commit 385ccf7c8a49e3364c4177ae46676ab08ddecb99
Author: Henry2SS <45096548+henry...@users.noreply.github.com>
AuthorDate: Sat Feb 26 17:04:39 2022 +0800
[fix](routine-load) fix show routine load task error (#8195)
---
.../java/org/apache/doris/load/routineload/RoutineLoadJob.java | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index 0026b80..94523a1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -31,6 +31,7 @@ import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Table;
+import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeMetaVersion;
@@ -1310,7 +1311,14 @@ public abstract class RoutineLoadJob extends
AbstractTxnStateChangeCallback impl
public List> getTasksShowInfo() {
List> rows = Lists.newArrayList();
-routineLoadTaskInfoList.stream().forEach(entity ->
rows.add(entity.getTaskShowInfo()));
+routineLoadTaskInfoList.stream().forEach(entity -> {
+try {
+
entity.setTxnStatus(Catalog.getCurrentCatalog().getGlobalTransactionMgr().getDatabaseTransactionMgr(dbId).getTransactionState(entity.getTxnId()).getTransactionStatus());
+rows.add(entity.getTaskShowInfo());
+} catch (AnalysisException e) {
+LOG.warn("failed to setTxnStatus db: {}, txnId: {}, err: {}",
dbId, entity.getTxnId(), e.getMessage());
+}
+});
return rows;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8239: [docs][fix][fix some confusing doc content]
morningman merged pull request #8239: URL: https://github.com/apache/incubator-doris/pull/8239 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [typo](doc)fix some confusing doc content (#8239)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 30fbf1d [typo](doc)fix some confusing doc content (#8239)
30fbf1d is described below
commit 30fbf1da6819410f67a37968eb96226beecd4162
Author: daikon <1059907...@qq.com>
AuthorDate: Sat Feb 26 17:05:08 2022 +0800
[typo](doc)fix some confusing doc content (#8239)
---
docs/en/extending-doris/hive-of-doris.md | 2 +-
docs/en/extending-doris/iceberg-of-doris.md| 2 +-
docs/en/extending-doris/odbc-of-doris.md | 2 +-
docs/en/installing/install-deploy.md | 14 +++---
docs/zh-CN/extending-doris/hive-of-doris.md| 2 +-
docs/zh-CN/extending-doris/iceberg-of-doris.md | 2 +-
docs/zh-CN/extending-doris/odbc-of-doris.md| 2 +-
docs/zh-CN/installing/install-deploy.md| 6 +++---
8 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/docs/en/extending-doris/hive-of-doris.md
b/docs/en/extending-doris/hive-of-doris.md
index 3d1ccac..04b9d19 100644
--- a/docs/en/extending-doris/hive-of-doris.md
+++ b/docs/en/extending-doris/hive-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "Hive of Doris",
+"title": "Doris On Hive",
"language": "en"
}
---
diff --git a/docs/en/extending-doris/iceberg-of-doris.md
b/docs/en/extending-doris/iceberg-of-doris.md
index 387b00c..aa83042 100644
--- a/docs/en/extending-doris/iceberg-of-doris.md
+++ b/docs/en/extending-doris/iceberg-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "Iceberg of Doris",
+"title": "Doris On Iceberg",
"language": "en"
}
---
diff --git a/docs/en/extending-doris/odbc-of-doris.md
b/docs/en/extending-doris/odbc-of-doris.md
index 2e95c4c..fe8915c 100644
--- a/docs/en/extending-doris/odbc-of-doris.md
+++ b/docs/en/extending-doris/odbc-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "ODBC of Doris",
+"title": "Doris On ODBC",
"language": "en"
}
---
diff --git a/docs/en/installing/install-deploy.md
b/docs/en/installing/install-deploy.md
index 3d5b1ba..6c4b2a3 100644
--- a/docs/en/installing/install-deploy.md
+++ b/docs/en/installing/install-deploy.md
@@ -115,15 +115,15 @@ Doris instances communicate directly over the network.
The following table shows
| Instance Name | Port Name | Default Port | Communication Direction |
Description|
| ---|---|---|---|---|
-| BE | be_port | 9060 | FE - > BE | BE for receiving requests from FE|
+| BE | be_port | 9060 | FE --> BE | BE for receiving requests from FE|
| BE | webserver\_port | 8040 | BE <--> BE | BE|
-| BE | heartbeat\_service_port | 9050 | FE - > BE | the heart beat service
port (thrift) on BE, used to receive heartbeat from FE|
-| BE | brpc\_port* | 8060 | FE < - > BE, BE < - > BE | BE for communication
between BEs|
-| FE | http_port* | 8030 | FE < - > FE, HTTP server port on user | FE|
-| FE | rpc_port | 9020 | BE - > FE, FE < - > FE | thrift server port on FE,
the configuration of each fe needs to be consistent|
-| FE | query_port | 9030 | user | FE|
+| BE | heartbeat\_service_port | 9050 | FE --> BE | the heart beat service
port (thrift) on BE, used to receive heartbeat from FE|
+| BE | brpc\_port | 8060 | FE <--> BE, BE <--> BE | BE for communication
between BEs|
+| FE | http_port | 8030 | FE <--> FE, user <--> FE | HTTP server port on FE |
+| FE | rpc_port | 9020 | BE --> FE, FE <--> FE | thrift server port on FE, the
configuration of each fe needs to be consistent|
+| FE | query_port | 9030 | user <--> FE | FE|
| FE | edit\_log_port | 9010 | FE <--> FE | FE|
-| Broker | broker ipc_port | 8000 | FE - > Broker, BE - > Broker | Broker for
receiving requests|
+| Broker | broker ipc_port | 8000 | FE --> Broker, BE --> Broker | Broker for
receiving requests|
> Note:
>
diff --git a/docs/zh-CN/extending-doris/hive-of-doris.md
b/docs/zh-CN/extending-doris/hive-of-doris.md
index 545b810..b1da9ee 100644
--- a/docs/zh-CN/extending-doris/hive-of-doris.md
+++ b/docs/zh-CN/extending-doris/hive-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "Hive of Doris",
+"title": "Doris On Hive",
"language": "zh-CN"
}
---
diff --git a/docs/zh-CN/extending-doris/iceberg-of-doris.md
b/docs/zh-CN/extending-doris/iceberg-of-doris.md
index cf3450c..8fd873a 100644
--- a/docs/zh-CN/extending-doris/iceberg-of-doris.md
+++ b/docs/zh-CN/extending-doris/iceberg-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "Iceberg of Doris",
+"title": "Doris On Iceberg",
"language": "zh-CN"
}
---
diff --git a/docs/zh-CN/extending-doris/odbc-of-doris.md
b/docs/zh-CN/extending-doris/odbc-of-doris.md
index 95bc663..951f9d8 100644
--- a/docs/zh-CN/extending-doris/odbc-of-doris.md
+++ b/docs/zh-CN/extending-doris/odbc-of-doris.md
@@ -1,6 +1,6 @@
---
{
-"title": "ODBC of Doris",
+"title": "Doris On ODBC",
"language": "zh-CN"
}
---
diff --git a/d
[GitHub] [incubator-doris] morningman merged pull request #8241: [fix](olap) add check statement to protect get_dict_word_info() from crash
morningman merged pull request #8241: URL: https://github.com/apache/incubator-doris/pull/8241 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](olap) add check statement to protect get_dict_word_info() from crash (#8241)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 351fbbc [fix](olap) add check statement to protect
get_dict_word_info() from crash (#8241)
351fbbc is described below
commit 351fbbcba58514d9de4b4de761340c2772c82216
Author: zuochunwei
AuthorDate: Sat Feb 26 17:05:58 2022 +0800
[fix](olap) add check statement to protect get_dict_word_info() from crash
(#8241)
check _num_elems firstly at the beginning of get_dict_word_info for safe
---
be/src/olap/rowset/segment_v2/binary_plain_page.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h
b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 2060bd5..061884f 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -269,6 +269,8 @@ public:
}
void get_dict_word_info(StringRef* dict_word_info) {
+if (_num_elems <= 0) [[unlikely]] return;
+
char* data_begin = (char*)&_data[0];
char* offset_ptr = (char*)&_data[_offsets_pos];
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 05/06: [fix](routine-load) fix show routine load task error (#8195)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit bc174fc9a6fdfebbc8aa354b879f4827009de13b
Author: Henry2SS <45096548+henry...@users.noreply.github.com>
AuthorDate: Sat Feb 26 17:04:39 2022 +0800
[fix](routine-load) fix show routine load task error (#8195)
---
.../java/org/apache/doris/load/routineload/RoutineLoadJob.java | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index 2549ce3..be61e99 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -31,6 +31,7 @@ import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Table;
+import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeMetaVersion;
@@ -1299,7 +1300,14 @@ public abstract class RoutineLoadJob extends
AbstractTxnStateChangeCallback impl
public List> getTasksShowInfo() {
List> rows = Lists.newArrayList();
-routineLoadTaskInfoList.stream().forEach(entity ->
rows.add(entity.getTaskShowInfo()));
+routineLoadTaskInfoList.stream().forEach(entity -> {
+try {
+
entity.setTxnStatus(Catalog.getCurrentCatalog().getGlobalTransactionMgr().getDatabaseTransactionMgr(dbId).getTransactionState(entity.getTxnId()).getTransactionStatus());
+rows.add(entity.getTaskShowInfo());
+} catch (AnalysisException e) {
+LOG.warn("failed to setTxnStatus db: {}, txnId: {}, err: {}",
dbId, entity.getTxnId(), e.getMessage());
+}
+});
return rows;
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 02/06: [Bug] Double/Float % 0 should be NULL (#8230)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit fc7437683d88a4c069ee7958ad9c57be172f4575
Author: HappenLee
AuthorDate: Fri Feb 25 11:03:42 2022 +0800
[Bug] Double/Float % 0 should be NULL (#8230)
Co-authored-by: lihaopeng
---
be/src/exprs/arithmetic_expr.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/exprs/arithmetic_expr.cpp b/be/src/exprs/arithmetic_expr.cpp
index 06546c2..dc4399b 100644
--- a/be/src/exprs/arithmetic_expr.cpp
+++ b/be/src/exprs/arithmetic_expr.cpp
@@ -113,7 +113,7 @@ FloatVal ModExpr::get_float_val(ExprContext* context,
TupleRow* row) {
return FloatVal::null();
}
FloatVal v2 = _children[1]->get_float_val(context, row);
-if (v2.is_null) {
+if (v2.is_null || v2.val == 0) {
return FloatVal::null();
}
return FloatVal(fmod(v1.val, v2.val));
@@ -125,7 +125,7 @@ DoubleVal ModExpr::get_double_val(ExprContext* context,
TupleRow* row) {
return DoubleVal::null();
}
DoubleVal v2 = _children[1]->get_double_val(context, row);
-if (v2.is_null) {
+if (v2.is_null || v2.val == 0) {
return DoubleVal::null();
}
return DoubleVal(fmod(v1.val, v2.val));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 04/06: [improvement][vectorized] support es node predicate peel (#8174)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a78bfecaaec7684a221d3b0593c9ffa94f31643e
Author: Pxl <952130...@qq.com>
AuthorDate: Sat Feb 26 17:02:54 2022 +0800
[improvement][vectorized] support es node predicate peel (#8174)
---
be/src/exec/es/es_predicate.cpp| 2 +-
be/src/exec/es/es_predicate.h | 4 +-
be/src/exec/es/es_query_builder.cpp| 121 ++---
be/src/exec/es/es_query_builder.h | 1 +
be/src/exec/es_http_scan_node.cpp | 24 +++-
be/src/exec/es_http_scan_node.h| 10 +-
be/src/exec/olap_scan_node.cpp | 58 ++
be/src/exec/olap_scan_node.h | 14 +--
be/src/exec/scan_node.cpp | 28 +
be/src/exec/scan_node.h| 9 +-
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/exec/ves_http_scanner.cpp | 8 +-
be/src/vec/exec/ves_http_scanner.h | 24 ++--
be/src/vec/functions/function_convert_tz.h | 2 +
.../function_date_or_datetime_computation.h| 2 +
.../function_date_or_datetime_to_something.h | 2 +
.../function_date_or_datetime_to_string.h | 2 +
.../functions/function_datetime_string_to_string.h | 2 +
.../{function_hash.h => function_fake.cpp} | 22 +---
...{function_multi_same_args.h => function_fake.h} | 36 +++---
be/src/vec/functions/function_grouping.h | 10 +-
be/src/vec/functions/function_hash.h | 1 +
be/src/vec/functions/function_ifnull.h | 49 -
be/src/vec/functions/function_multi_same_args.h| 4 +-
be/src/vec/functions/function_rpc.h| 2 +
be/src/vec/functions/function_string_to_string.h | 2 +
be/src/vec/functions/simple_function_factory.h | 3 +
be/src/vec/utils/util.hpp | 26 -
gensrc/script/doris_builtins_functions.py | 2 +-
29 files changed, 244 insertions(+), 227 deletions(-)
diff --git a/be/src/exec/es/es_predicate.cpp b/be/src/exec/es/es_predicate.cpp
index 3b8be66..7dab9ca 100644
--- a/be/src/exec/es/es_predicate.cpp
+++ b/be/src/exec/es/es_predicate.cpp
@@ -181,7 +181,7 @@ Status EsPredicate::build_disjuncts_list() {
}
// make sure to build by build_disjuncts_list
-const std::vector& EsPredicate::get_predicate_list() {
+const std::vector& EsPredicate::get_predicate_list() const {
return _disjuncts;
}
diff --git a/be/src/exec/es/es_predicate.h b/be/src/exec/es/es_predicate.h
index 825bc5e..826af0a 100644
--- a/be/src/exec/es/es_predicate.h
+++ b/be/src/exec/es/es_predicate.h
@@ -141,10 +141,10 @@ class EsPredicate {
public:
EsPredicate(ExprContext* context, const TupleDescriptor* tuple_desc,
ObjectPool* pool);
~EsPredicate();
-const std::vector& get_predicate_list();
+const std::vector& get_predicate_list() const;
Status build_disjuncts_list();
// public for tests
-EsPredicate(const std::vector& all_predicates) { _disjuncts
= all_predicates; };
+EsPredicate(const std::vector& all_predicates) { _disjuncts
= all_predicates; }
Status get_es_query_status() { return _es_query_status; }
diff --git a/be/src/exec/es/es_query_builder.cpp
b/be/src/exec/es/es_query_builder.cpp
index 5f8dcfd..1f9488e 100644
--- a/be/src/exec/es/es_query_builder.cpp
+++ b/be/src/exec/es/es_query_builder.cpp
@@ -54,19 +54,19 @@ TermQueryBuilder::TermQueryBuilder(const std::string&
field, const std::string&
TermQueryBuilder::TermQueryBuilder(const ExtBinaryPredicate& binary_predicate)
: _field(binary_predicate.col.name), _match_none(false) {
-if (binary_predicate.col.type.type == PrimitiveType::TYPE_BOOLEAN) {
-int val = atoi(binary_predicate.value.to_string().c_str());
-if (val == 1) {
-_term = std::string("true");
-} else if (val == 0){
-_term = std::string("false");
-} else {
-// keep semantic consistent with mysql
-_match_none = true;
-}
+if (binary_predicate.col.type.type == PrimitiveType::TYPE_BOOLEAN) {
+int val = atoi(binary_predicate.value.to_string().c_str());
+if (val == 1) {
+_term = std::string("true");
+} else if (val == 0) {
+_term = std::string("false");
} else {
-_term = binary_predicate.value.to_string();
+// keep semantic consistent with mysql
+_match_none = true;
}
+} else {
+_term = binary_predicate.value.to_string();
+}
}
void TermQueryBuilder::to_json(rapidjson::Document* document,
rapidjson::Value* query) {
@@ -82,7 +82,6 @@ void TermQueryBuilder::to_
[incubator-doris] 06/06: [fix](olap) add check statement to protect get_dict_word_info() from crash (#8241)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 699d0659fe61b7575673e49e2e799e0d8631bec1
Author: zuochunwei
AuthorDate: Sat Feb 26 17:05:58 2022 +0800
[fix](olap) add check statement to protect get_dict_word_info() from crash
(#8241)
check _num_elems firstly at the beginning of get_dict_word_info for safe
---
be/src/olap/rowset/segment_v2/binary_plain_page.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h
b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 2060bd5..061884f 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -269,6 +269,8 @@ public:
}
void get_dict_word_info(StringRef* dict_word_info) {
+if (_num_elems <= 0) [[unlikely]] return;
+
char* data_begin = (char*)&_data[0];
char* offset_ptr = (char*)&_data[_offsets_pos];
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 03/06: [Bug][Vectorized] Fix bug of decimal cast to double loss precision (#8221)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5f511854de3447325e5d4ba8dc9f72167a26c114
Author: HappenLee
AuthorDate: Fri Feb 25 11:04:11 2022 +0800
[Bug][Vectorized] Fix bug of decimal cast to double loss precision (#8221)
Co-authored-by: lihaopeng
---
be/src/vec/data_types/data_type_decimal.h | 2 +-
be/src/vec/exec/join/vhash_join_node.cpp | 67 +--
be/src/vec/exec/join/vhash_join_node.h| 2 -
3 files changed, 38 insertions(+), 33 deletions(-)
diff --git a/be/src/vec/data_types/data_type_decimal.h
b/be/src/vec/data_types/data_type_decimal.h
index 8792a95..21cdbed 100644
--- a/be/src/vec/data_types/data_type_decimal.h
+++ b/be/src/vec/data_types/data_type_decimal.h
@@ -290,7 +290,7 @@ convert_from_decimal(const typename
FromDataType::FieldType& value, UInt32 scale
using ToFieldType = typename ToDataType::FieldType;
if constexpr (std::is_floating_point_v)
-return static_cast(value) /
FromDataType::get_scale_multiplier(scale);
+return binary_cast(value);
else {
FromFieldType converted_value =
convert_decimals(value, scale, 0);
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp
b/be/src/vec/exec/join/vhash_join_node.cpp
index ead..f6a5422 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -158,8 +158,6 @@ template
struct ProcessHashTableProbe {
ProcessHashTableProbe(HashJoinNode* join_node, int batch_size, int
probe_rows)
: _join_node(join_node),
- _right_col_idx(join_node->_right_col_idx),
- _right_col_len(join_node->_right_col_len),
_batch_size(batch_size),
_probe_rows(probe_rows),
_build_blocks(join_node->_build_blocks),
@@ -180,6 +178,10 @@ struct ProcessHashTableProbe {
using KeyGetter = typename HashTableContext::State;
using Mapped = typename HashTableContext::Mapped;
+int right_col_idx = _join_node->_is_right_semi_anti ? 0 :
+_join_node->_left_table_data_types.size();
+int right_col_len = _join_node->_right_table_data_types.size();
+
KeyGetter key_getter(_probe_raw_ptrs, _join_node->_probe_key_sz,
nullptr);
auto& mcol = mutable_block.mutable_columns();
int current_offset = 0;
@@ -194,6 +196,10 @@ struct ProcessHashTableProbe {
constexpr auto is_right_semi_anti_join = JoinOpType::value ==
TJoinOp::RIGHT_ANTI_JOIN ||
JoinOpType::value ==
TJoinOp::RIGHT_SEMI_JOIN;
+constexpr auto is_semi_anti_join = is_right_semi_anti_join ||
+JoinOpType::value ==
TJoinOp::LEFT_ANTI_JOIN ||
+JoinOpType::value ==
TJoinOp::LEFT_SEMI_JOIN;
+
constexpr auto probe_all = JoinOpType::value ==
TJoinOp::LEFT_OUTER_JOIN ||
JoinOpType::value ==
TJoinOp::FULL_OUTER_JOIN;
@@ -269,34 +275,34 @@ struct ProcessHashTableProbe {
}
// insert all matched build rows
-if constexpr (!is_right_semi_anti_join) {
+if constexpr (!is_semi_anti_join) {
if (_build_blocks.size() == 1) {
-for (int i = 0; i < _right_col_len; i++) {
+for (int i = 0; i < right_col_len; i++) {
auto& column = *_build_blocks[0].get_by_position(i).column;
-mcol[i + _right_col_idx]->insert_indices_from(column,
+mcol[i + right_col_idx]->insert_indices_from(column,
_build_block_rows.data(), _build_block_rows.data()
+ current_offset);
}
} else {
-for (int i = 0; i < _right_col_len; i++) {
+for (int i = 0; i < right_col_len; i++) {
for (int j = 0; j < current_offset; j++) {
if constexpr (probe_all) {
if (_build_block_offsets[j] == -1) {
-DCHECK(mcol[i +
_right_col_idx]->is_nullable());
-assert_cast(mcol[i +
_right_col_idx].get())->insert_data(nullptr, 0);
+DCHECK(mcol[i + right_col_idx]->is_nullable());
+assert_cast(mcol[i +
right_col_idx].get())->insert_data(nullptr, 0);
} else {
auto& column =
*_build_blocks[_build_block_offsets[j]].get_by_position(i).column;
-mcol[i + _right_col_idx]->insert_from(column,
_build_block_rows[j]);
+mcol[i + right_col_idx]->insert_from(column,
_build_block_rows[j]);
}
[incubator-doris] 01/06: [Bug] group_concat(value, null) not return null
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 9196fbee6a7ed791215e90149c011c70f9e913b6
Author: Pxl <952130...@qq.com>
AuthorDate: Fri Feb 25 11:03:23 2022 +0800
[Bug] group_concat(value,null) not return null
---
be/src/exprs/aggregate_functions.cpp | 4 ++--
.../sql-functions/aggregate-functions/group_concat.md | 7 +++
.../sql-functions/aggregate-functions/group_concat.md | 7 +++
3 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/be/src/exprs/aggregate_functions.cpp
b/be/src/exprs/aggregate_functions.cpp
index f8efe67..9f059b7 100644
--- a/be/src/exprs/aggregate_functions.cpp
+++ b/be/src/exprs/aggregate_functions.cpp
@@ -784,7 +784,7 @@ void AggregateFunctions::max(FunctionContext*, const
DateTimeVal& src, DateTimeV
void AggregateFunctions::string_concat(FunctionContext* ctx, const StringVal&
src,
const StringVal& separator, StringVal*
result) {
-if (src.is_null) {
+if (src.is_null || separator.is_null) {
return;
}
@@ -819,7 +819,7 @@ void
AggregateFunctions::string_concat_update(FunctionContext* ctx, const String
void AggregateFunctions::string_concat_update(FunctionContext* ctx, const
StringVal& src,
const StringVal& separator,
StringVal* result) {
-if (src.is_null) {
+if (src.is_null || separator.is_null) {
return;
}
const StringVal* sep = separator.is_null ? &DEFAULT_STRING_CONCAT_DELIM :
&separator;
diff --git
a/docs/en/sql-reference/sql-functions/aggregate-functions/group_concat.md
b/docs/en/sql-reference/sql-functions/aggregate-functions/group_concat.md
index bdff733..b9273a9 100644
--- a/docs/en/sql-reference/sql-functions/aggregate-functions/group_concat.md
+++ b/docs/en/sql-reference/sql-functions/aggregate-functions/group_concat.md
@@ -58,6 +58,13 @@ mysql> select GROUP_CONCAT(value, " ") from test;
++
| a b c |
++
+
+mysql> select GROUP_CONCAT(value, NULL) from test;
+++
+| GROUP_CONCAT(`value`, NULL)|
+++
+| NULL |
+++
```
## keyword
GROUP_CONCAT,GROUP,CONCAT
diff --git
a/docs/zh-CN/sql-reference/sql-functions/aggregate-functions/group_concat.md
b/docs/zh-CN/sql-reference/sql-functions/aggregate-functions/group_concat.md
index bc43575..8e2a3f4 100644
--- a/docs/zh-CN/sql-reference/sql-functions/aggregate-functions/group_concat.md
+++ b/docs/zh-CN/sql-reference/sql-functions/aggregate-functions/group_concat.md
@@ -58,6 +58,13 @@ mysql> select GROUP_CONCAT(value, " ") from test;
++
| a b c |
++
+
+mysql> select GROUP_CONCAT(value, NULL) from test;
+++
+| GROUP_CONCAT(`value`, NULL)|
+++
+| NULL |
+++
```
## keyword
GROUP_CONCAT,GROUP,CONCAT
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch dev-1.0.0 updated (c302179 -> 699d065)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.0
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from c302179 [Feature][array-type]add proto for complex data type ARRAY
(#8212)
new 9196fbe [Bug] group_concat(value,null) not return null
new fc74376 [Bug] Double/Float % 0 should be NULL (#8230)
new 5f51185 [Bug][Vectorized] Fix bug of decimal cast to double loss
precision (#8221)
new a78bfec [improvement][vectorized] support es node predicate peel
(#8174)
new bc174fc [fix](routine-load) fix show routine load task error (#8195)
new 699d065 [fix](olap) add check statement to protect
get_dict_word_info() from crash (#8241)
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/es/es_predicate.cpp| 2 +-
be/src/exec/es/es_predicate.h | 4 +-
be/src/exec/es/es_query_builder.cpp| 121 ++---
be/src/exec/es/es_query_builder.h | 1 +
be/src/exec/es_http_scan_node.cpp | 24 +++-
be/src/exec/es_http_scan_node.h| 10 +-
be/src/exec/olap_scan_node.cpp | 58 ++
be/src/exec/olap_scan_node.h | 14 +--
be/src/exec/scan_node.cpp | 28 +
be/src/exec/scan_node.h| 9 +-
be/src/exprs/aggregate_functions.cpp | 4 +-
be/src/exprs/arithmetic_expr.cpp | 4 +-
be/src/olap/rowset/segment_v2/binary_plain_page.h | 2 +
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/data_types/data_type_decimal.h | 2 +-
be/src/vec/exec/join/vhash_join_node.cpp | 67 +++-
be/src/vec/exec/join/vhash_join_node.h | 2 -
be/src/vec/exec/ves_http_scanner.cpp | 8 +-
be/src/vec/exec/ves_http_scanner.h | 24 ++--
be/src/vec/functions/function_convert_tz.h | 2 +
.../function_date_or_datetime_computation.h| 2 +
.../function_date_or_datetime_to_something.h | 2 +
.../function_date_or_datetime_to_string.h | 2 +
.../functions/function_datetime_string_to_string.h | 2 +
.../{function_convert_tz.cpp => function_fake.cpp} | 6 +-
...{function_multi_same_args.h => function_fake.h} | 36 +++---
be/src/vec/functions/function_grouping.h | 10 +-
be/src/vec/functions/function_hash.h | 1 +
be/src/vec/functions/function_ifnull.h | 49 -
be/src/vec/functions/function_multi_same_args.h| 4 +-
be/src/vec/functions/function_rpc.h| 2 +
be/src/vec/functions/function_string_to_string.h | 2 +
be/src/vec/functions/simple_function_factory.h | 3 +
be/src/vec/utils/util.hpp | 26 -
.../aggregate-functions/group_concat.md| 7 ++
.../aggregate-functions/group_concat.md| 7 ++
.../doris/load/routineload/RoutineLoadJob.java | 10 +-
gensrc/script/doris_builtins_functions.py | 2 +-
38 files changed, 310 insertions(+), 250 deletions(-)
copy be/src/vec/functions/{function_convert_tz.cpp => function_fake.cpp} (83%)
copy be/src/vec/functions/{function_multi_same_args.h => function_fake.h} (72%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] amosbird opened a new pull request #8249: Support aarch64 target with ldb_toolchain
amosbird opened a new pull request #8249: URL: https://github.com/apache/incubator-doris/pull/8249 # Proposed changes Issue Number: close https://github.com/apache/incubator-doris/issues/3526 , https://github.com/apache/incubator-doris/issues/6065 ## Problem Summary: Some build fixes for aarch64 target. ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments Starting from v0.9.1 ldb_toolchain contains an aarch64 variant https://github.com/amosbird/ldb_toolchain_gen/releases/tag/v0.9.1 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8246: Remove old fe meta version
morningman commented on a change in pull request #8246:
URL: https://github.com/apache/incubator-doris/pull/8246#discussion_r815292493
##
File path: fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java
##
@@ -1030,10 +1029,6 @@ public void write(DataOutput out) throws IOException {
}
public void readFields(DataInput in) throws IOException {
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_94) {
-readFieldOld(in);
Review comment:
remove `readFieldOld` method
##
File path:
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##
@@ -523,86 +520,31 @@ public void write(DataOutput out) throws IOException {
}
public void readFields(DataInput in) throws IOException {
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_43) {
-// consume the flag of empty user name
-in.readBoolean();
-}
-
-// user name
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_30) {
-qualifiedUser =
ClusterNamespace.getFullName(SystemInfoService.DEFAULT_CLUSTER,
Text.readString(in));
-} else {
-qualifiedUser = Text.readString(in);
-}
-
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_43) {
-int passwordLen = in.readInt();
-byte[] password = new byte[passwordLen];
-in.readFully(password);
-
-// boolean isAdmin
-in.readBoolean();
-
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_1) {
-// boolean isSuperuser
-in.readBoolean();
-}
-}
+qualifiedUser = Text.readString(in);
if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_100) {
Review comment:
MINIMUM_VERSION_REQUIRED is 100, so I think we can remove this.
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -975,35 +973,19 @@ private void readFromStream(DataInput in) throws
IOException {
txIsolation = Text.readString(in);
autoCommit = in.readBoolean();
resourceGroup = Text.readString(in);
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_65) {
-sqlMode = in.readLong();
-} else {
-// read old version SQL mode
-Text.readString(in);
-sqlMode = 0L;
-}
+sqlMode = in.readLong();
enableProfile = in.readBoolean();
queryTimeoutS = in.readInt();
maxExecMemByte = in.readLong();
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_37) {
-collationServer = Text.readString(in);
-}
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_38) {
-batchSize = in.readInt();
-disableStreamPreaggregations = in.readBoolean();
-parallelExecInstanceNum = in.readInt();
-}
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_62) {
-exchangeInstanceParallel = in.readInt();
-}
+collationServer = Text.readString(in);
+batchSize = in.readInt();
+disableStreamPreaggregations = in.readBoolean();
+parallelExecInstanceNum = in.readInt();
+exchangeInstanceParallel = in.readInt();
}
public void readFields(DataInput in) throws IOException {
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_67) {
-readFromStream(in);
Review comment:
remove `readFromStream()` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 closed pull request #7925: [optimization]Remove the old http module and completely replace it with http v2
hf200012 closed pull request #7925: URL: https://github.com/apache/incubator-doris/pull/7925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
morningman commented on a change in pull request #8247:
URL: https://github.com/apache/incubator-doris/pull/8247#discussion_r815295347
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -50,6 +50,9 @@
public static final String RESOURCE_VARIABLE = "resource_group";
public static final String AUTO_COMMIT = "autocommit";
public static final String TX_ISOLATION = "tx_isolation";
+public static final String TX_READ_ONLY = "tx_read_only";
+ public static final String TRANSACTION_READ_ONLY =
"transaction_read_only";
Review comment:
tab to space
##
File path:
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlHandshakePacket.java
##
@@ -23,7 +23,7 @@
// Version of handshake packet, since MySQL 3.21.0, Handshake of protocol
10 is used
private static final int PROTOCOL_VERSION = 10;
// JDBC uses this version to check which protocol the server support
-private static final String SERVER_VERSION = "5.1.73";
+private static final String SERVER_VERSION = "5.7.37";
Review comment:
```suggestion
public static final String SERVER_VERSION = "5.7.37";
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
##
@@ -47,7 +47,7 @@
Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH;
@VariableMgr.VarAttr(name = VERSION, flag = VariableMgr.READ_ONLY)
-public static String version = "5.1.0";
+public static String version = "5.7.37";
Review comment:
```suggestion
public static String version = MysqlHandshakePacket.SERVER_VERSION
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -456,7 +471,19 @@ public void setSqlMode(long sqlMode) {
public boolean isAutoCommit() {
return autoCommit;
}
+
+ public boolean isTxReadonly() {
Review comment:
tab to space
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
Review comment:
This method is deprecated, no need to modify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
morningman commented on pull request #8247: URL: https://github.com/apache/incubator-doris/pull/8247#issuecomment-1051968033 Hi @aopangzi , thanks for your contribution. And please also modify the document: ``` docs/zh-CN/administrator-guide/variables.md docs/en/administrator-guide/variables.md ``` to add new variables -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8248: [fix] (rpc-udf) Fixed the problem that the query could not be interrupted if an error occurred in the rpc server during the execu
morningman commented on pull request #8248: URL: https://github.com/apache/incubator-doris/pull/8248#issuecomment-1051971283 compile failed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aopangzi commented on a change in pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
aopangzi commented on a change in pull request #8247: URL: https://github.com/apache/incubator-doris/pull/8247#discussion_r815300238 ## File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java ## @@ -50,6 +50,9 @@ public static final String RESOURCE_VARIABLE = "resource_group"; public static final String AUTO_COMMIT = "autocommit"; public static final String TX_ISOLATION = "tx_isolation"; +public static final String TX_READ_ONLY = "tx_read_only"; + public static final String TRANSACTION_READ_ONLY = "transaction_read_only"; Review comment: Thank you for your advice. It has been corrected -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aopangzi commented on a change in pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
aopangzi commented on a change in pull request #8247:
URL: https://github.com/apache/incubator-doris/pull/8247#discussion_r815300261
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
Review comment:
Yes, you're right. The tx_isolation and tx_read_only variables are
now(5.7.20) deprecated and will be removed in MySQL 8.0 .So add transaction_
isolation。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] aopangzi commented on a change in pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
aopangzi commented on a change in pull request #8247: URL: https://github.com/apache/incubator-doris/pull/8247#discussion_r815300365 ## File path: fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java ## @@ -47,7 +47,7 @@ Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH; @VariableMgr.VarAttr(name = VERSION, flag = VariableMgr.READ_ONLY) -public static String version = "5.1.0"; +public static String version = "5.7.37"; Review comment: Thank you for your advice. I'll study it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch revert-8218-upgrade-grpc-version created (now 406f9ed)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch revert-8218-upgrade-grpc-version in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. at 406f9ed Revert "[chore](dependency) upgrade-grpc-version (#8218)" This branch includes the following new commits: new 406f9ed Revert "[chore](dependency) upgrade-grpc-version (#8218)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] 01/01: Revert "[chore](dependency) upgrade-grpc-version (#8218)"
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch revert-8218-upgrade-grpc-version in repository https://gitbox.apache.org/repos/asf/incubator-doris.git commit 406f9ed6b908e14dff467c9bc5325f6b7f020067 Author: Mingyu Chen AuthorDate: Sat Feb 26 19:46:37 2022 +0800 Revert "[chore](dependency) upgrade-grpc-version (#8218)" This reverts commit df7e848cbbc8170c7bd83d812d7cac58b5574570. --- fe/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fe/pom.xml b/fe/pom.xml index 8e13dcb..36dd85c 100644 --- a/fe/pom.xml +++ b/fe/pom.xml @@ -111,7 +111,7 @@ under the License. UTF-8 1.8 1.8 -1.44.1 +1.30.0 3.14.0 false 2.3.7 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #8250: [revert] "[improvement]upgrade-grpc-version"
morningman opened a new pull request #8250: URL: https://github.com/apache/incubator-doris/pull/8250 Reverts apache/incubator-doris#8218 Because when using grpc 1.44.1, the corresponding `protoc-gen-grpc-java` plugin requried GLIBC_2.14, which is not found in CentOS 6. So I suggest to revert this commit this time. And considering upgrading this component after most systems have reached glibc version 2.14. And for Mac M1, you may have to change this version manually for now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8250: [revert] "[improvement]upgrade-grpc-version"
morningman commented on pull request #8250: URL: https://github.com/apache/incubator-doris/pull/8250#issuecomment-1052072597 CC @wunan1210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8249: Support aarch64 target with ldb_toolchain
github-actions[bot] commented on pull request #8249: URL: https://github.com/apache/incubator-doris/pull/8249#issuecomment-1052088536 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #8245: fix restore with partitions cause some query fail
morningman commented on a change in pull request #8245:
URL: https://github.com/apache/incubator-doris/pull/8245#discussion_r815304759
##
File path: fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
##
@@ -469,6 +470,16 @@ private void checkAndPrepareMeta() {
}
olapTbl.setState(OlapTableState.RESTORE);
+// set restore status for partitions
+BackupOlapTableInfo tblInfo =
jobInfo.backupOlapTableObjects.get(tableName);
+for (Map.Entry partitionEntry :
tblInfo.partitions.entrySet()) {
+String partitionName = partitionEntry.getKey();
+Partition partition = olapTbl.getPartition(partitionName);
+if (partition == null) {
+continue;
+}
+partition.setState(PartitionState.RESTORE);
Review comment:
you also need to modify `replayCheckAndPrepareMeta()` method.
##
File path: fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
##
@@ -585,7 +587,17 @@ public TableRef resolveTableRef(TableRef tableRef) throws
AnalysisException {
if (table.getType() == TableType.OLAP && (((OlapTable)
table).getState() == OlapTableState.RESTORE
|| ((OlapTable) table).getState() ==
OlapTableState.RESTORE_WITH_LOAD)) {
-ErrorReport.reportAnalysisException(ErrorCode.ERR_BAD_TABLE_STATE,
"RESTORING");
+Boolean isNotRestoring = ((OlapTable)
table).getPartitions().stream().filter(
+partition -> partition.getState() == PartitionState.RESTORE
+).collect(Collectors.toList()).isEmpty();
+
+if(!isNotRestoring){
+// if doing restore with partitions, the status check push
down to OlapScanNode::computePartitionInfo to
+// support query that partitions is not restoring.
+}else {
Review comment:
```suggestion
} else {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8243: [improvment] Ignore the case of column name, when do the stream load in json format
github-actions[bot] commented on pull request #8243: URL: https://github.com/apache/incubator-doris/pull/8243#issuecomment-1052112168 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #8243: [improvment] Ignore the case of column name, when do the stream load in json format
caiconghui commented on pull request #8243: URL: https://github.com/apache/incubator-doris/pull/8243#issuecomment-1052115998 I have two question: 1. this solution is only for simple json, and other json would cause different behavior? 2. it seems that if user not use fuzzy_parse for optimization that would cause different result? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #8251: [improvement](lateral-view) Add number rows filtered in profile
morningman opened a new pull request #8251: URL: https://github.com/apache/incubator-doris/pull/8251 # Proposed changes Issue Number: close #xxx ## Problem Summary: Add `RowsFiltered` counter in TableFunctionNode profile. So that we can know the total number of rows that TableFunctionNode processed ## Checklist(Required) 1. Does it affect the original behavior: (No) 2. Has unit tests been added: (No Need) 3. Has document been added or modified: (No Need) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #8252: [community] add more collaborators in .asf.yaml (#8029)
morningman opened a new pull request #8252: URL: https://github.com/apache/incubator-doris/pull/8252 # Proposed changes Issue Number: close #xxx ## Problem Summary: Add following 2 collaborators: @zuochunwei @thomascai126 ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on a change in pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
hf200012 commented on a change in pull request #8247:
URL: https://github.com/apache/incubator-doris/pull/8247#discussion_r815293589
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -50,6 +50,9 @@
public static final String RESOURCE_VARIABLE = "resource_group";
public static final String AUTO_COMMIT = "autocommit";
public static final String TX_ISOLATION = "tx_isolation";
+public static final String TX_READ_ONLY = "tx_read_only";
+ public static final String TRANSACTION_READ_ONLY =
"transaction_read_only";
Review comment:
```suggestion
public static final String TRANSACTION_READ_ONLY =
"transaction_read_only";
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
Review comment:
```suggestion
transactionIsolation = Text.readString(in);
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
+ txReadonly = in.readBoolean();
+ transactionReadonly = in.readBoolean();
Review comment:
code format
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
+ txReadonly = in.readBoolean();
+ transactionReadonly = in.readBoolean();
Review comment:
```suggestion
transactionReadonly = in.readBoolean();
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -50,6 +50,9 @@
public static final String RESOURCE_VARIABLE = "resource_group";
public static final String AUTO_COMMIT = "autocommit";
public static final String TX_ISOLATION = "tx_isolation";
+public static final String TX_READ_ONLY = "tx_read_only";
+ public static final String TRANSACTION_READ_ONLY =
"transaction_read_only";
+ public static final String TRANSACTION_ISOLATION =
"transaction_isolation";
Review comment:
```suggestion
public static final String TRANSACTION_ISOLATION =
"transaction_isolation";
```
##
File path: fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
##
@@ -973,6 +1000,9 @@ private void readFromStream(DataInput in) throws
IOException {
charsetConnection = Text.readString(in);
charsetClient = Text.readString(in);
txIsolation = Text.readString(in);
+ transactionIsolation = Text.readString(in);
+ txReadonly = in.readBoolean();
Review comment:
```suggestion
txReadonly = in.readBoolean();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8247: [refactor] Upgrade to MySQL 5.7.37 to reduce unnecessary CVE issues
github-actions[bot] commented on pull request #8247: URL: https://github.com/apache/incubator-doris/pull/8247#issuecomment-1052283144 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8252: [community] add more collaborators in .asf.yaml (#8029)
github-actions[bot] commented on pull request #8252: URL: https://github.com/apache/incubator-doris/pull/8252#issuecomment-1052284102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #8253: [fix](unit_test) Fix unit test failed for tablet_info_test
caiconghui opened a new pull request #8253: URL: https://github.com/apache/incubator-doris/pull/8253 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8253: [fix](unit_test) Fix unit test failed for tablet_info_test
github-actions[bot] commented on pull request #8253: URL: https://github.com/apache/incubator-doris/pull/8253#issuecomment-1052912012 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8249: Support aarch64 target with ldb_toolchain
morningman merged pull request #8249: URL: https://github.com/apache/incubator-doris/pull/8249 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #3526: aarch64上编译不通过
morningman closed issue #3526: URL: https://github.com/apache/incubator-doris/issues/3526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (351fbbc -> 7f3564c)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 351fbbc [fix](olap) add check statement to protect get_dict_word_info() from crash (#8241) add 7f3564c [chore] Support aarch64 target with ldb_toolchain (#8249) No new revisions were added by this update. Summary of changes: be/CMakeLists.txt | 1 - be/src/glibc-compatibility/CMakeLists.txt | 9 +- be/src/glibc-compatibility/FastMemcpy.c| 220 --- be/src/glibc-compatibility/FastMemcpy.h| 694 - be/src/glibc-compatibility/LICENSE_FastMemcpy | 22 - .../glibc-compatibility/memcpy/memcpy_aarch64.cpp | 245 .../glibc-compatibility/memcpy/memcpy_x86_64.cpp | 220 +++ be/src/glibc-compatibility/memcpy_wrapper.c| 6 - thirdparty/patches/libhdfs3-master.patch | 24 +- 9 files changed, 492 insertions(+), 949 deletions(-) delete mode 100644 be/src/glibc-compatibility/FastMemcpy.c delete mode 100644 be/src/glibc-compatibility/FastMemcpy.h delete mode 100644 be/src/glibc-compatibility/LICENSE_FastMemcpy create mode 100644 be/src/glibc-compatibility/memcpy/memcpy_aarch64.cpp create mode 100644 be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp delete mode 100644 be/src/glibc-compatibility/memcpy_wrapper.c - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #8254: [doc] Modify document of compilation on ARM64
morningman opened a new pull request #8254: URL: https://github.com/apache/incubator-doris/pull/8254 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #8253: [fix](unit_test) Fix unit test failed for tablet_info_test
morningman commented on pull request #8253: URL: https://github.com/apache/incubator-doris/pull/8253#issuecomment-1052989913 Quick merge for unit test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #8253: [fix](unit_test) Fix unit test failed for tablet_info_test
morningman merged pull request #8253: URL: https://github.com/apache/incubator-doris/pull/8253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [fix](be-ut) fix unit test bug for tablet_info_test (#8253)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new c66a9bf [fix](be-ut) fix unit test bug for tablet_info_test (#8253)
c66a9bf is described below
commit c66a9bf64b3d135844390a663329bcf3203ef1de
Author: caiconghui <55968745+caicong...@users.noreply.github.com>
AuthorDate: Sun Feb 27 10:44:20 2022 +0800
[fix](be-ut) fix unit test bug for tablet_info_test (#8253)
introduced from #8041
---
be/test/exec/tablet_info_test.cpp | 39 +--
1 file changed, 13 insertions(+), 26 deletions(-)
diff --git a/be/test/exec/tablet_info_test.cpp
b/be/test/exec/tablet_info_test.cpp
index 01d325e..9b676d5 100644
--- a/be/test/exec/tablet_info_test.cpp
+++ b/be/test/exec/tablet_info_test.cpp
@@ -159,9 +159,8 @@ TEST_F(OlapTablePartitionParamTest, normal) {
memcpy(str_val->ptr, "abc", str_val->len);
// 9:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(10, partition->id);
}
@@ -178,9 +177,8 @@ TEST_F(OlapTablePartitionParamTest, normal) {
memcpy(str_val->ptr, "abcd", str_val->len);
// 25:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(11, partition->id);
}
@@ -197,9 +195,8 @@ TEST_F(OlapTablePartitionParamTest, normal) {
memcpy(str_val->ptr, "abcde", str_val->len);
// 50:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_FALSE(found);
}
@@ -216,9 +213,8 @@ TEST_F(OlapTablePartitionParamTest, normal) {
memcpy(str_val->ptr, "abcdef", str_val->len);
// 60:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(12, partition->id);
}
@@ -356,9 +352,8 @@ TEST_F(OlapTablePartitionParamTest, single_list_partition) {
memcpy(str_val->ptr, "abc", str_val->len);
// 1:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(10, partition->id);
}
@@ -375,9 +370,8 @@ TEST_F(OlapTablePartitionParamTest, single_list_partition) {
memcpy(str_val->ptr, "abcd", str_val->len);
// 3:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(11, partition->id);
}
@@ -394,9 +388,8 @@ TEST_F(OlapTablePartitionParamTest, single_list_partition) {
memcpy(str_val->ptr, "abcde", str_val->len);
// 50:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_FALSE(found);
}
@@ -413,9 +406,8 @@ TEST_F(OlapTablePartitionParamTest, single_list_partition) {
memcpy(str_val->ptr, "abcdef", str_val->len);
// 6:
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(12, partition->id);
}
@@ -572,9 +564,8 @@ TEST_F(OlapTablePartitionParamTest, multi_list_partition) {
memcpy(str_val->ptr, "beijing", str_val->len);
// 1, beijing
-uint32_t dist_hash = 0;
const OlapTablePartition* partition = nullptr;
-auto found = part.find_tablet(tuple, &partition, &dist_hash);
+auto found = part.find_partition(tuple, &partition);
ASSERT_TRUE(found);
ASSERT_EQ(10, partition->id);
}
@@ -591,9 +582,8 @@ TEST_F(OlapTablePartitionParamTest, multi_list_partition) {
memcpy(str_val->ptr, "shanghai", str_val->len);
[GitHub] [incubator-doris] yiguolei commented on a change in pull request #8246: Remove old fe meta version
yiguolei commented on a change in pull request #8246:
URL: https://github.com/apache/incubator-doris/pull/8246#discussion_r815379917
##
File path:
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java
##
@@ -523,86 +520,31 @@ public void write(DataOutput out) throws IOException {
}
public void readFields(DataInput in) throws IOException {
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_43) {
-// consume the flag of empty user name
-in.readBoolean();
-}
-
-// user name
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_30) {
-qualifiedUser =
ClusterNamespace.getFullName(SystemInfoService.DEFAULT_CLUSTER,
Text.readString(in));
-} else {
-qualifiedUser = Text.readString(in);
-}
-
-if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_43) {
-int passwordLen = in.readInt();
-byte[] password = new byte[passwordLen];
-in.readFully(password);
-
-// boolean isAdmin
-in.readBoolean();
-
-if (Catalog.getCurrentCatalogJournalVersion() >=
FeMetaVersion.VERSION_1) {
-// boolean isSuperuser
-in.readBoolean();
-}
-}
+qualifiedUser = Text.readString(in);
if (Catalog.getCurrentCatalogJournalVersion() <
FeMetaVersion.VERSION_100) {
Review comment:
Actually, I am removing code that FeMetaVersion < VERSION_96. I am
afraid of version 100 is too latest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xiaokang commented on pull request #7999: [Enhancement] Optimization for string_search
xiaokang commented on pull request #7999: URL: https://github.com/apache/incubator-doris/pull/7999#issuecomment-1052994475 @BiteThet Should the following license comment for python be removed? // This is taken from the python search string function doing string search (substring) // using an optimized boyer-moore-horspool algorithm. // http://hg.python.org/cpython/file/6b6c79eba944/Objects/stringlib/fastsearch.h -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] BiteTheDDDDt commented on pull request #7999: [Enhancement] Optimization for string_search
BiteThet commented on pull request #7999: URL: https://github.com/apache/incubator-doris/pull/7999#issuecomment-1053013421 > @BiteThet Should the following license comment for python be removed? > > // This is taken from the python search string function doing string search (substring) // using an optimized boyer-moore-horspool algorithm. // http://hg.python.org/cpython/file/6b6c79eba944/Objects/stringlib/fastsearch.h yes, and seems like should add apache license. Do you want add a pull request to fix it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] morningman commented on pull request #11: [feature]Supports traversal of Doris FE nodes when searching for Doris BE on branch flink-before-1.13
morningman commented on pull request #11: URL: https://github.com/apache/incubator-doris-flink-connector/pull/11#issuecomment-1053017735 LGTM No need to update document, because we already support specify multi FE nodes before -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] branch branch-for-flink-before-1.13 updated: [improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-for-flink-before-1.13
in repository
https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git
The following commit(s) were added to refs/heads/branch-for-flink-before-1.13
by this push:
new 3ebb83a [improvement] Supports traversal of Doris FE nodes when
searching for Doris BE (#11)
3ebb83a is described below
commit 3ebb83a9c65b275b538c1458be8cf4c0a77a68c1
Author: Jiangqiao Xu <96433131+bridgedr...@users.noreply.github.com>
AuthorDate: Sun Feb 27 11:08:39 2022 +0800
[improvement] Supports traversal of Doris FE nodes when searching for Doris
BE (#11)
---
.../org/apache/doris/flink/rest/RestService.java | 45 ++
1 file changed, 37 insertions(+), 8 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
index 0c4264f..c229cf0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
@@ -244,6 +244,26 @@ public class RestService implements Serializable {
}
/**
+ * choice a Doris FE node to request.
+ *
+ * @param feNodes Doris FE node list, separate be comma
+ * @param logger slf4j logger
+ * @return the array of Doris FE nodes
+ * @throws IllegalArgumentException fe nodes is illegal
+ */
+@VisibleForTesting
+static List allEndpoints(String feNodes, Logger logger) throws
IllegalArgumentException {
+logger.trace("Parse fenodes '{}'.", feNodes);
+if (StringUtils.isEmpty(feNodes)) {
+logger.error(ILLEGAL_ARGUMENT_MESSAGE, "fenodes", feNodes);
+throw new IllegalArgumentException("fenodes", feNodes);
+}
+List nodes =
Arrays.stream(feNodes.split(",")).map(String::trim).collect(Collectors.toList());
+Collections.shuffle(nodes);
+return nodes;
+}
+
+/**
* choice a Doris BE node to request.
*
* @param options configuration of request
@@ -322,19 +342,28 @@ public class RestService implements Serializable {
*
* @param options configuration of request
* @param logger slf4j logger
- * @return the chosen one Doris BE node
+ * @return all Doris BE node
* @throws IllegalArgumentException BE nodes is illegal
*/
@VisibleForTesting
static List getBackendsV2(DorisOptions options,
DorisReadOptions readOptions, Logger logger) throws DorisException, IOException
{
String feNodes = options.getFenodes();
-String feNode = randomEndpoint(feNodes, logger);
-String beUrl = "http://"; + feNode + BACKENDS_V2;
-HttpGet httpGet = new HttpGet(beUrl);
-String response = send(options, readOptions, httpGet, logger);
-logger.info("Backend Info:{}", response);
-List backends = parseBackendV2(response,
logger);
-return backends;
+List feNodeList = allEndpoints(feNodes, logger);
+for (String feNode: feNodeList) {
+try {
+String beUrl = "http://"; + feNode + BACKENDS_V2;
+HttpGet httpGet = new HttpGet(beUrl);
+String response = send(options, readOptions, httpGet, logger);
+logger.info("Backend Info:{}", response);
+List backends =
parseBackendV2(response, logger);
+return backends;
+} catch (ConnectedFailedException e) {
+logger.info("Doris FE node {} is unavailable: {}, Request the
next Doris FE node", feNode, e.getMessage());
+}
+}
+String errMsg = "No Doris FE is available, please check configuration";
+logger.error(errMsg);
+throw new DorisException(errMsg);
}
static List parseBackendV2(String response, Logger
logger) throws DorisException, IOException {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] morningman merged pull request #11: [feature]Supports traversal of Doris FE nodes when searching for Doris BE on branch flink-before-1.13
morningman merged pull request #11: URL: https://github.com/apache/incubator-doris-flink-connector/pull/11 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris-flink-connector] branch master updated: [improvement] Supports traversal of Doris FE nodes when searching for Doris BE (#11)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/incubator-doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 7effefa [improvement] Supports traversal of Doris FE nodes when
searching for Doris BE (#11)
7effefa is described below
commit 7effefa160f7b6a8e9753ad07c516f0ee160906e
Author: Jiangqiao Xu <96433131+bridgedr...@users.noreply.github.com>
AuthorDate: Sun Feb 27 11:08:53 2022 +0800
[improvement] Supports traversal of Doris FE nodes when searching for Doris
BE (#11)
---
.../org/apache/doris/flink/rest/RestService.java | 43 ++
1 file changed, 36 insertions(+), 7 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
index 9812840..f1d0512 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
@@ -244,6 +244,26 @@ public class RestService implements Serializable {
}
/**
+ * choice a Doris FE node to request.
+ *
+ * @param feNodes Doris FE node list, separate be comma
+ * @param logger slf4j logger
+ * @return the array of Doris FE nodes
+ * @throws IllegalArgumentException fe nodes is illegal
+ */
+@VisibleForTesting
+static List allEndpoints(String feNodes, Logger logger) throws
IllegalArgumentException {
+logger.trace("Parse fenodes '{}'.", feNodes);
+if (StringUtils.isEmpty(feNodes)) {
+logger.error(ILLEGAL_ARGUMENT_MESSAGE, "fenodes", feNodes);
+throw new IllegalArgumentException("fenodes", feNodes);
+}
+List nodes =
Arrays.stream(feNodes.split(",")).map(String::trim).collect(Collectors.toList());
+Collections.shuffle(nodes);
+return nodes;
+}
+
+/**
* choice a Doris BE node to request.
*
* @param options configuration of request
@@ -328,13 +348,22 @@ public class RestService implements Serializable {
@VisibleForTesting
static List getBackendsV2(DorisOptions options,
DorisReadOptions readOptions, Logger logger) throws DorisException, IOException
{
String feNodes = options.getFenodes();
-String feNode = randomEndpoint(feNodes, logger);
-String beUrl = "http://"; + feNode + BACKENDS_V2;
-HttpGet httpGet = new HttpGet(beUrl);
-String response = send(options, readOptions, httpGet, logger);
-logger.info("Backend Info:{}", response);
-List backends = parseBackendV2(response,
logger);
-return backends;
+List feNodeList = allEndpoints(feNodes, logger);
+for (String feNode: feNodeList) {
+try {
+String beUrl = "http://"; + feNode + BACKENDS_V2;
+HttpGet httpGet = new HttpGet(beUrl);
+String response = send(options, readOptions, httpGet, logger);
+logger.info("Backend Info:{}", response);
+List backends =
parseBackendV2(response, logger);
+return backends;
+} catch (ConnectedFailedException e) {
+logger.info("Doris FE node {} is unavailable: {}, Request the
next Doris FE node", feNode, e.getMessage());
+}
+}
+String errMsg = "No Doris FE is available, please check configuration";
+logger.error(errMsg);
+throw new DorisException(errMsg);
}
static List parseBackendV2(String response, Logger
logger) throws DorisException, IOException {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris-flink-connector] morningman merged pull request #9: [feature]Supports traversal of Doris FE nodes when searching for Doris BE
morningman merged pull request #9: URL: https://github.com/apache/incubator-doris-flink-connector/pull/9 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #8246: Remove old fe meta version
github-actions[bot] commented on pull request #8246: URL: https://github.com/apache/incubator-doris/pull/8246#issuecomment-1053069476 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new issue #8255: [Enhancement] Support routine load task succeed with empty data consumed
caiconghui opened a new issue #8255: URL: https://github.com/apache/incubator-doris/issues/8255 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description routine load task would be marked as canceled, which would cause user confused. but now Doris should process this well for that Doris can commit empty data transaction after PR #7240. ### Solution Support routine load task succeed with empty data consumed ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] caiconghui opened a new pull request #8256: [Enhancement](routine-load) Support routine load task succeed with empty data consumed
caiconghui opened a new pull request #8256: URL: https://github.com/apache/incubator-doris/pull/8256 # Proposed changes Issue Number: close #8255 ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes) 2. Has unit tests been added: (No) 3. Has document been added or modified: (No) 4. Does it need to update dependencies: (No) 5. Are there any changes that cannot be rolled back: (No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
