[doris] branch master updated: [typo](docs) Modify parameter description (#23782)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0585beee02a [typo](docs) Modify parameter description (#23782)
0585beee02a is described below
commit 0585beee02ab7c533897aebf884fc63a8b0bc691
Author: yuanyuan8983 <99315889+yuanyuan8...@users.noreply.github.com>
AuthorDate: Mon Oct 16 14:29:00 2023 +0800
[typo](docs) Modify parameter description (#23782)
---
docs/en/docs/advanced/variables.md| 2 +-
docs/zh-CN/docs/advanced/variables.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/en/docs/advanced/variables.md
b/docs/en/docs/advanced/variables.md
index 0ac94b7087d..59ec3992cfd 100644
--- a/docs/en/docs/advanced/variables.md
+++ b/docs/en/docs/advanced/variables.md
@@ -639,7 +639,7 @@ Note that the comment must start with /*+ and can only
follow the SELECT.
-Use a fixed replica to query. If use_fix_replica is 1, the smallest one is
used, if use_fix_replica is 2, the second smallest one is used, and so on. The
default value is -1, which means it is not enabled.
+Use a fixed replica to query. replica starts with 0 and if use_fix_replica
is 0, the smallest is used, if use_fix_replica is 1, the second smallest is
used, and so on. The default value is -1, indicating that the function is
disabled.
* `dry_run_query`
diff --git a/docs/zh-CN/docs/advanced/variables.md
b/docs/zh-CN/docs/advanced/variables.md
index bee0717993e..17196070417 100644
--- a/docs/zh-CN/docs/advanced/variables.md
+++ b/docs/zh-CN/docs/advanced/variables.md
@@ -627,7 +627,7 @@ try (Connection conn =
DriverManager.getConnection("jdbc:mysql://127.0.0.1:9030/
-使用固定的replica进行查询，该值表示固定使用第几小的replica，默认为-1表示不启用。
+
使用固定replica进行查询。replica从0开始，如果use_fix_replica为0，则使用最小的，如果use_fix_replica为1，则使用第二个最小的，依此类推。默认值为-1，表示未启用。
* `dry_run_query`
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](jdbc catalog) fix handle oracle date format (#25487)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 652d6c57c03 [fix](jdbc catalog) fix handle oracle date format (#25487)
652d6c57c03 is described below
commit 652d6c57c03e25a0f957a5c08534ff6949c93fe7
Author: zy-kkk
AuthorDate: Tue Oct 17 02:10:28 2023 -0500
[fix](jdbc catalog) fix handle oracle date format (#25487)
---
.../main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java | 3 +--
.../data/external_table_p0/jdbc/test_oracle_jdbc_catalog.out | 3 +++
.../suites/external_table_p0/jdbc/test_oracle_jdbc_catalog.groovy | 1 +
3 files changed, 5 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
index 614b71484ce..3295284da92 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
@@ -32,7 +32,6 @@ import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.JdbcTable;
-import org.apache.doris.catalog.Type;
import org.apache.doris.catalog.external.JdbcExternalTable;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
@@ -364,7 +363,7 @@ public class JdbcScanNode extends ExternalScanNode {
private static String handleOracleDateFormat(Expr expr) {
if (expr.isConstant()
-&& (expr.getType().equals(Type.DATETIME) ||
expr.getType().equals(Type.DATETIMEV2))) {
+&& (expr.getType().isDatetime() ||
expr.getType().isDatetimeV2())) {
return "to_date('" + expr.getStringValue() + "', '-mm-dd
hh24:mi:ss')";
}
return expr.toMySql();
diff --git
a/regression-test/data/external_table_p0/jdbc/test_oracle_jdbc_catalog.out
b/regression-test/data/external_table_p0/jdbc/test_oracle_jdbc_catalog.out
index ad2572ddb35..1be0feb1a90 100644
--- a/regression-test/data/external_table_p0/jdbc/test_oracle_jdbc_catalog.out
+++ b/regression-test/data/external_table_p0/jdbc/test_oracle_jdbc_catalog.out
@@ -99,6 +99,9 @@
1 2022-01-21T05:23:01 \N \N
2 2022-11-12T20:32:56 \N \N
+-- !date7 --
+3 \N 2019-11-12T20:33:57.999 \N \N \N \N \N
+
-- !test_insert1 --
doris1 18
diff --git
a/regression-test/suites/external_table_p0/jdbc/test_oracle_jdbc_catalog.groovy
b/regression-test/suites/external_table_p0/jdbc/test_oracle_jdbc_catalog.groovy
index 79e4367793e..77d3015239c 100644
---
a/regression-test/suites/external_table_p0/jdbc/test_oracle_jdbc_catalog.groovy
+++
b/regression-test/suites/external_table_p0/jdbc/test_oracle_jdbc_catalog.groovy
@@ -80,6 +80,7 @@ suite("test_oracle_jdbc_catalog",
"p0,external,oracle,external_docker,external_d
order_qt_date4 """ select * from TEST_DATE where (T1 > '2022-01-21
00:00:00' and T1 < '2022-01-22 00:00:00') or (T1 > '2022-01-20 00:00:00' and T1
< '2022-01-23 00:00:00'); """
order_qt_date5 """ select * from TEST_DATE where T1 < '2022-01-22
00:00:00' or T1 = '2022-01-21 05:23:01'; """
order_qt_date6 """ select * from TEST_DATE where (T1 < '2022-01-22
00:00:00' or T1 > '2022-01-20 00:00:00') and (T1 < '2022-01-23 00:00:00' or T1
> '2022-01-19 00:00:00'); """
+order_qt_date7 """select * from TEST_TIMESTAMP where T2 <
str_to_date('2020-12-21 12:34:56', '%Y-%m-%d %H:%i:%s');"""
// The result of TEST_RAW will change
// So instead of qt, we're using sql here.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (db16a149704 -> 76abbfc5556)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from db16a149704 [fix](backup) fix backup fail on s3 (#25496) add 76abbfc5556 [fix](mvn source) fix audit compile java-cup and cup-maven-plugin not found (#25564) No new revisions were added by this update. Summary of changes: fe_plugins/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhance](regression)change external default docker list (#25573)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 0533911275b [Enhance](regression)change external default docker list (#25573) 0533911275b is described below commit 0533911275b0192f0ddebe9b5dda6e47a7a993d5 Author: zhangguoqiang <18372634...@163.com> AuthorDate: Wed Oct 18 15:56:10 2023 +0800 [Enhance](regression)change external default docker list (#25573) --- docker/thirdparties/run-thirdparties-docker.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/thirdparties/run-thirdparties-docker.sh b/docker/thirdparties/run-thirdparties-docker.sh index de55f9c3396..32a6db41f8b 100755 --- a/docker/thirdparties/run-thirdparties-docker.sh +++ b/docker/thirdparties/run-thirdparties-docker.sh @@ -60,7 +60,7 @@ STOP=0 if [[ "$#" == 1 ]]; then # default - COMPONENTS="mysql,pg,oracle,sqlserver,clickhouse,hive,iceberg,hudi,trino,kafka" +COMPONENTS="mysql,es,hive,pg,oracle,sqlserver,clickhouse" else while true; do case "$1" in - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch master updated: [feature] support struct and map type (#212)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 9f30144 [feature] support struct and map type (#212)
9f30144 is described below
commit 9f30144120964eeaf9c2757afcfc1242e588a8f0
Author: wudi <676366...@qq.com>
AuthorDate: Thu Oct 19 10:05:09 2023 +0800
[feature] support struct and map type (#212)
Support doris map, struct type reading and writing
```java
//doris create table
CREATE TABLE `simple_map2` (
`id` int(11) NULL,
`m` MAP NULL,
`s_info` STRUCT NULL
) ENGINE=OLAP
DUPLICATE KEY(`id`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"is_being_synced" = "false",
"storage_format" = "V2",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false"
);
//datagen->doris
tEnv.executeSql(
"CREATE TABLE doris_test (" +
" id int,\n" +
" task Map,\n" +
" buyer ROW\n" +
") " +
"WITH (\n" +
" 'connector' = 'datagen', \n" +
" 'number-of-rows' = '11' \n" +
")");
tEnv.executeSql("CREATE TABLE blackhole_table (" +
"id int," +
"m Map," +
"s_info Row" +
") WITH (" +
" 'connector' = 'doris',\n" +
" 'fenodes' = '127.0.0.1:8030',\n" +
" 'table.identifier' = 'test.simple_map2',\n" +
" 'sink.enable-2pc' = 'false',\n" +
" 'username' = 'root',\n" +
" 'password' = '',\n" +
" 'sink.properties.format' = 'json',\n" +
" 'sink.properties.read_json_by_line' = 'true'\n" +
");");
tEnv.executeSql("INSERT INTO blackhole_table select * from doris_test");
//doris->doris
tEnv.executeSql(
"CREATE TABLE doris_source (" +
"id int," +
"m Map," +
"s_info Row" +
") " +
"WITH (\n" +
" 'connector' = 'doris',\n" +
" 'fenodes' = '127.0.0.1:8030',\n" +
" 'table.identifier' = 'test.simple_map',\n" +
" 'username' = 'root',\n" +
" 'password' = ''\n" +
")");
tEnv.executeSql("CREATE TABLE blackhole_table (" +
"id int," +
"m Map," +
"s_info Row" +
") WITH (" +
" 'connector' = 'doris',\n" +
" 'fenodes' = '127.0.0.1:8030',\n" +
" 'table.identifier' = 'test.simple_map2',\n" +
" 'sink.enable-2pc' = 'false',\n" +
" 'username' = 'root',\n" +
" 'password' = '',\n" +
" 'sink.properties.format' = 'json',\n" +
" 'sink.properties.read_json_by_line' = 'true'\n" +
");");
tEnv.executeSql("insert into blackhole_table select * from doris_source");
```
---
.../converter/DorisRowConverter.java | 29 +
.../apache/doris/flink/serialization/RowBatch.java | 30 +
.../doris/flink/serialization/TestRowBatch.java| 144 +
3 files changed, 203 insertions(+)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/converter/DorisRowConverter.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/deserialization/converter/DorisRowConverter.java
index 6fa3be9..ebc0ff6 100644
---
a/flink-do
[doris] branch master updated: [typo](docs) Rename Import Advanced to CN Version (#25374)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fcf7bdc9e0e [typo](docs) Rename Import Advanced to CN Version (#25374)
fcf7bdc9e0e is described below
commit fcf7bdc9e0e1a65e1bda0694092ce4dfb040d718
Author: KassieZ <139741991+kass...@users.noreply.github.com>
AuthorDate: Thu Oct 19 10:18:30 2023 +0800
[typo](docs) Rename Import Advanced to CN Version (#25374)
---
docs/dev.json | 6 +-
.../data-operate/import/import-advanced/min-load-replica-num.md | 4 ++--
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/docs/dev.json b/docs/dev.json
index 646bbdeed64..4caad6c57cd 100644
--- a/docs/dev.json
+++ b/docs/dev.json
@@ -47,6 +47,10 @@
"message": "按方式导入",
"description": "The label for category Import Way in sidebar docs"
},
+ "sidebar.docs.category.Import Advanced": {
+"message": "高级导入",
+"description": "The label for category Import Advanced in sidebar docs"
+ },
"sidebar.docs.category.Export": {
"message": "导出",
"description": "The label for category Export in sidebar docs"
@@ -311,4 +315,4 @@
"message": "版本发布",
"description": "The label for category Release notes in sidebar docs"
}
-}
\ No newline at end of file
+}
diff --git
a/docs/zh-CN/docs/data-operate/import/import-advanced/min-load-replica-num.md
b/docs/zh-CN/docs/data-operate/import/import-advanced/min-load-replica-num.md
index 827f6baf9f3..f629268d913 100644
---
a/docs/zh-CN/docs/data-operate/import/import-advanced/min-load-replica-num.md
+++
b/docs/zh-CN/docs/data-operate/import/import-advanced/min-load-replica-num.md
@@ -1,6 +1,6 @@
---
{
-"title": "Min Load Replica Num",
+"title": "最小写入副本数",
"language": "zh-CN"
}
---
@@ -24,7 +24,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Min Load Replica Num
+# 最小写入副本数
默认情况下，数据导入要求至少有超过半数的副本写入成功，导入才算成功。然而，这种方式不够灵活，在某些场景会带来不便。
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.2-lts updated: [typo](doc)delete error doc (#25624)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new dd54c462e71 [typo](doc)delete error doc (#25624)
dd54c462e71 is described below
commit dd54c462e717087a736f296aacfe8461b9468e3f
Author: ZhenchaoXu <49646212+i...@users.noreply.github.com>
AuthorDate: Thu Oct 19 15:16:26 2023 +0800
[typo](doc)delete error doc (#25624)
---
.../sql-functions/string-functions/char.md | 69 --
docs/sidebars.json | 1 -
.../sql-functions/string-functions/char.md | 69 --
3 files changed, 139 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-functions/string-functions/char.md
b/docs/en/docs/sql-manual/sql-functions/string-functions/char.md
deleted file mode 100644
index 71e574077cb..000
--- a/docs/en/docs/sql-manual/sql-functions/string-functions/char.md
+++ /dev/null
@@ -1,69 +0,0 @@
-{
-"title": "char",
-"language": "en"
-}
-
-
-
-
-
-## function char
-### description
- Syntax
-
-`VARCHAR char(INT,..., [USING charset_name])`
-
-Interprets each argument as an integer and returns a string consisting of the
characters given by the code values of those integers. `NULL` values are
skipped.
-
-If the result string is illegal for the given character set, the result from
`CHAR()` becomes `NULL`.
-
-Arguments larger than `255` are converted into multiple result bytes. For
example, `char(15049882)` is equivalent to `char(229, 164, 154)`.
-
-Currently only `utf8` is supported for `charset_name`.
-
-
-### example
-
-```
-mysql> select char(68, 111, 114, 105, 115);
-+--+
-| char('utf8', 68, 111, 114, 105, 115) |
-+--+
-| Doris|
-+--+
-
-mysql> select char(15049882, 15179199, 14989469);
-++
-| char('utf8', 15049882, 15179199, 14989469) |
-++
-| 多睿丝 |
-++
-
-mysql> select char(255);
-+---+
-| char('utf8', 255) |
-+---+
-| NULL |
-+---+
-```
-### keywords
-CHAR
diff --git a/docs/sidebars.json b/docs/sidebars.json
index 9b7068d649e..2cf79010dbd 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -421,7 +421,6 @@
"sql-manual/sql-functions/string-functions/initcap",
"sql-manual/sql-functions/string-functions/repeat",
"sql-manual/sql-functions/string-functions/reverse",
-
"sql-manual/sql-functions/string-functions/char",
"sql-manual/sql-functions/string-functions/concat",
"sql-manual/sql-functions/string-functions/concat_ws",
"sql-manual/sql-functions/string-functions/substr",
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/string-functions/char.md
b/docs/zh-CN/docs/sql-manual/sql-functions/string-functions/char.md
deleted file mode 100644
index e3d57f3fd31..000
--- a/docs/zh-CN/docs/sql-manual/sql-functions/string-functions/char.md
+++ /dev/null
@@ -1,69 +0,0 @@
-{
-"title": "char",
-"language": "zh-CN"
-}
-
-
-
-
-
-## function char
-### description
- Syntax
-
-`VARCHAR char(INT,..., [USING charset_name])`
-
-将每个参数解释为整数，并返回一个字符串，该字符串由这些整数的代码值给出的字符组成。忽略`NULL`值。
-
-如果结果字符串对于给定字符集是非法的，相应的转换结果为`NULL`值。
-
-大于 `255` 的参数将转换为多个结果字节。例如，`char(15049882)`等价于`char(229, 164, 154)`。
-
-`charset_name`目前只支持`utf8`。
-
-
-### example
-
-```
-mysql> select char(68, 111, 114, 105, 115);
-+--+
-| char('utf8', 68, 111, 114, 105, 115) |
-+--+
-| Doris|
-+--+
-
-mysql> select char(15049882, 15179199, 14989469);
-++
-| char('utf8', 15049882, 15179199, 14989469) |
-++
-| 多睿丝 |
-++
-
-mysql> select char(255);
-+---+
-| char('utf8', 255) |
-+---+
-| NULL |
-+---+
-```
-### keywords
-CHAR
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (9ea8efe5fae -> 32fe78511a8)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9ea8efe5fae [coverage](fe)add jacoco coverage option on start_fe.sh (#25598) add 32fe78511a8 [typo](doc) update spark connector two phase commit option doc (#24458) No new revisions were added by this update. Summary of changes: docs/en/docs/ecosystem/spark-doris-connector.md| 1 + docs/zh-CN/docs/ecosystem/spark-doris-connector.md | 1 + 2 files changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](docs) Fix mistakes in flink-doris-connector docs (#24512) (#25557)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 49ca36720d6 [fix](docs) Fix mistakes in flink-doris-connector docs (#24512) (#25557) 49ca36720d6 is described below commit 49ca36720d69df3ec2ce5b539e8ac1345f853c57 Author: HowardQin AuthorDate: Fri Oct 20 17:53:18 2023 +0800 [fix](docs) Fix mistakes in flink-doris-connector docs (#24512) (#25557) --- docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md index f70aeb79ed5..abad86201eb 100644 --- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md +++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md @@ -30,7 +30,7 @@ under the License. -[Flink Doris Connector](https://github.com/apache/doris-flink-connector) 可以支持通过 Flink 操作（读取、插入、修改、删除） Doris 中存储的数据。本文档介绍如何通过Flink如果通过Datastream和SQL操作Doris。 +[Flink Doris Connector](https://github.com/apache/doris-flink-connector) 可以支持通过 Flink 操作（读取、插入、修改、删除） Doris 中存储的数据。本文档介绍Flink如何通过Datastream和SQL操作Doris。 >**注意：** > - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [build](extension) Update build-extension.yml (#25721)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 387a9c74489 [build](extension) Update build-extension.yml (#25721) 387a9c74489 is described below commit 387a9c74489c47930d3bb2506c70fdd06f0cefde Author: Jeffrey AuthorDate: Sat Oct 21 16:52:55 2023 +0800 [build](extension) Update build-extension.yml (#25721) --- .github/workflows/build-extension.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/build-extension.yml b/.github/workflows/build-extension.yml index 4a9fc5d38ef..d58e21a75dc 100644 --- a/.github/workflows/build-extension.yml +++ b/.github/workflows/build-extension.yml @@ -100,6 +100,7 @@ jobs: run: | git clone https://github.com/apache/doris-website.git website cd website + echo "[\"current\"]" > versions.json mkdir -p docs cp -R ../docs/en/docs/* docs/ cp -R ../docs/sidebars.json sidebars.json - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regresstion-test](jdbc catalog)Mariadb compatible test (#25664)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 7de3d9882c4 [regresstion-test](jdbc catalog)Mariadb compatible test (#25664) 7de3d9882c4 is described below commit 7de3d9882c4c699ef81e7274652288e2c81ed20e Author: Petrichor <31833513+vinle...@users.noreply.github.com> AuthorDate: Sun Oct 22 22:51:03 2023 -0500 [regresstion-test](jdbc catalog)Mariadb compatible test (#25664) --- .../docker-compose/mariadb/init/01-drop-db.sql | 18 .../docker-compose/mariadb/init/02-create-db.sql | 18 .../mariadb/init/03-create-table.sql | 76 + .../docker-compose/mariadb/init/04-insert.sql | 36 +++ .../docker-compose/mariadb/mariadb-10.env | 19 .../docker-compose/mariadb/mariadb-10.yaml.tpl | 49 + docker/thirdparties/run-thirdparties-docker.sh | 22 +++- regression-test/conf/regression-conf.groovy| 1 + .../jdbc/test_mariadb_jdbc_catalog.out | 117 + .../pipeline/p0/conf/regression-conf.groovy| 2 +- .../jdbc/test_mariadb_jdbc_catalog.groovy | 82 +++ 11 files changed, 435 insertions(+), 5 deletions(-) diff --git a/docker/thirdparties/docker-compose/mariadb/init/01-drop-db.sql b/docker/thirdparties/docker-compose/mariadb/init/01-drop-db.sql new file mode 100644 index 000..d0db4a7e1ac --- /dev/null +++ b/docker/thirdparties/docker-compose/mariadb/init/01-drop-db.sql @@ -0,0 +1,18 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +drop database if exists doris_test; diff --git a/docker/thirdparties/docker-compose/mariadb/init/02-create-db.sql b/docker/thirdparties/docker-compose/mariadb/init/02-create-db.sql new file mode 100644 index 000..f87b6922d9f --- /dev/null +++ b/docker/thirdparties/docker-compose/mariadb/init/02-create-db.sql @@ -0,0 +1,18 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +create database doris_test; diff --git a/docker/thirdparties/docker-compose/mariadb/init/03-create-table.sql b/docker/thirdparties/docker-compose/mariadb/init/03-create-table.sql new file mode 100644 index 000..e82da905dca --- /dev/null +++ b/docker/thirdparties/docker-compose/mariadb/init/03-create-table.sql @@ -0,0 +1,76 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +CREATE TABLE doris_test.ex_tb0 +( +`id` int PRIMARY KEY, +`name` varchar(128) +); + +create table doris_test.all
[doris] branch master updated: [typo](doc)Adjust tablet_rowset_stale_sweep_time_sec parameter default value to 300 (#25584)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new f23fdcbbef1 [typo](doc)Adjust tablet_rowset_stale_sweep_time_sec parameter default value to 300 (#25584) f23fdcbbef1 is described below commit f23fdcbbef1467696311e4b9c2d83e289b9733f7 Author: DongLiang-0 <46414265+donglian...@users.noreply.github.com> AuthorDate: Tue Oct 24 10:32:08 2023 +0800 [typo](doc)Adjust tablet_rowset_stale_sweep_time_sec parameter default value to 300 (#25584) --- docs/en/docs/admin-manual/config/be-config.md| 2 +- docs/zh-CN/docs/admin-manual/config/be-config.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md index 05a8a5d16a0..0cfbbd8a192 100644 --- a/docs/en/docs/admin-manual/config/be-config.md +++ b/docs/en/docs/admin-manual/config/be-config.md @@ -1265,7 +1265,7 @@ BaseCompaction:546859: * Type: int64 * Description: It is used to control the expiration time of cleaning up the merged rowset version. When the current time now() minus the max created rowset‘s create time in a version path is greater than tablet_rowset_stale_sweep_time_sec, the current path is cleaned up and these merged rowsets are deleted, the unit is second. - - When writing is too frequent and the disk time is insufficient, you can configure less tablet_rowset_stale_sweep_time_sec. However, if this time is less than 5 minutes, it may cause fe to query the version that has been merged, causing a query -230 error. + - When writing is too frequent, Fe may not be able to query the merged version, resulting in a query -230 error. This problem can be avoided by increasing this parameter. * Default value: 300 `tablet_writer_open_rpc_timeout_sec` diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md index 4a1476a980e..36dd0a345c9 100644 --- a/docs/zh-CN/docs/admin-manual/config/be-config.md +++ b/docs/zh-CN/docs/admin-manual/config/be-config.md @@ -1292,7 +1292,7 @@ BaseCompaction:546859: * 类型：int64 * 描述：用来表示清理合并版本的过期时间，当当前时间 now() 减去一个合并的版本路径中rowset最近创建创建时间大于tablet_rowset_stale_sweep_time_sec时，对当前路径进行清理，删除这些合并过的rowset, 单位为s。 - - 当写入过于频繁，磁盘空间不足时，可以配置较少这个时间。不过这个时间过短小于5分钟时，可能会引发fe查询不到已经合并过的版本，引发查询-230错误。 + - 当写入过于频繁，可能会引发fe查询不到已经合并过的版本，引发查询-230错误。可以通过调大该参数避免该问题。 * 默认值：300 `tablet_writer_open_rpc_timeout_sec` - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-spark-connector] branch master updated: [improve] add auto redirect options (#150)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-spark-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 1ee1cae [improve] add auto redirect options (#150)
1ee1cae is described below
commit 1ee1cae591853de1f852c09c6c89ba8d221899a7
Author: wudi <676366...@qq.com>
AuthorDate: Thu Oct 26 10:52:42 2023 +0800
[improve] add auto redirect options (#150)
Add redirection parameters. After opening, there is no need to obtain the
be list, and streamloading is performed through fe.
---
.../doris/spark/cfg/ConfigurationOptions.java | 5 +++
.../apache/doris/spark/load/DorisStreamLoad.java | 46 --
.../org/apache/doris/spark/rest/RestService.java | 2 +-
.../doris/spark/rest/models/RespContent.java | 4 +-
.../spark/listener/DorisTransactionListener.scala | 6 +--
.../apache/doris/spark/writer/DorisWriter.scala| 12 +++---
.../doris/spark/sql/TestSparkConnector.scala | 1 +
7 files changed, 51 insertions(+), 25 deletions(-)
diff --git
a/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
b/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
index 6498916..7877bc8 100644
---
a/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
+++
b/spark-doris-connector/src/main/java/org/apache/doris/spark/cfg/ConfigurationOptions.java
@@ -119,4 +119,9 @@ public interface ConfigurationOptions {
String DORIS_SINK_TXN_RETRIES = "doris.sink.txn.retries";
int DORIS_SINK_TXN_RETRIES_DEFAULT = 3;
+/**
+ * Use automatic redirection of fe without explicitly obtaining the be list
+ */
+String DORIS_SINK_AUTO_REDIRECT = "doris.sink.auto-redirect";
+boolean DORIS_SINK_AUTO_REDIRECT_DEFAULT = false;
}
diff --git
a/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
b/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
index 338ffbe..3d5bf36 100644
---
a/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
+++
b/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
@@ -16,14 +16,6 @@
// under the License.
package org.apache.doris.spark.load;
-import org.apache.doris.spark.cfg.ConfigurationOptions;
-import org.apache.doris.spark.cfg.SparkSettings;
-import org.apache.doris.spark.exception.StreamLoadException;
-import org.apache.doris.spark.rest.RestService;
-import org.apache.doris.spark.rest.models.BackendV2;
-import org.apache.doris.spark.rest.models.RespContent;
-import org.apache.doris.spark.util.ResponseUtil;
-
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -32,6 +24,14 @@ import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import org.apache.commons.collections.MapUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.doris.spark.cfg.ConfigurationOptions;
+import org.apache.doris.spark.cfg.SparkSettings;
+import org.apache.doris.spark.exception.IllegalArgumentException;
+import org.apache.doris.spark.exception.StreamLoadException;
+import org.apache.doris.spark.rest.RestService;
+import org.apache.doris.spark.rest.models.BackendV2;
+import org.apache.doris.spark.rest.models.RespContent;
+import org.apache.doris.spark.util.ResponseUtil;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
@@ -41,7 +41,9 @@ import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.entity.BufferedHttpEntity;
import org.apache.http.entity.InputStreamEntity;
import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.DefaultRedirectStrategy;
import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.types.StructType;
@@ -94,6 +96,7 @@ public class DorisStreamLoad implements Serializable {
private boolean addDoubleQuotes;
private static final long cacheExpireTimeout = 4 * 60;
private final LoadingCache> cache;
+private final String fenodes;
private final String fileType;
private String FIELD_DELIMITER;
private final String LINE_DELIMITER;
@@ -101,11 +104,13 @@ public class DorisStreamLoad implements Serializable {
private final boolean enable2PC;
private final Integer txnRetries;
private final Integer txnIntervalMs;
+private final boolean autoRedirect;
public DorisStreamLoad(SparkSettings settings) {
String[] db
(doris) branch revert-25639-add_tpcds_shape_sf100_hive created (now 093dae1a0a5)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch revert-25639-add_tpcds_shape_sf100_hive in repository https://gitbox.apache.org/repos/asf/doris.git at 093dae1a0a5 Revert "[Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)" This branch includes the following new commits: new 093dae1a0a5 Revert "[Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6a85f46ff33 -> 501c6096dd4)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6a85f46ff33 [refactor](move-memtable) rename open_stream_sink rpc to open_load_stream (#25883) add 501c6096dd4 Revert "[Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)" (#26069) No new revisions were added by this update. Summary of changes: .licenserc.yaml|2 - .../hive/scripts/create_tpcds100_orc.hql | 625 -- .../hive/scripts/create_tpcds100_parquet.hql | 625 -- .../docker-compose/hive/scripts/hive-metastore.sh |6 - .../tpcds100_orc_shape/shape/query1.out| 42 - .../tpcds100_orc_shape/shape/query10.out | 57 - .../tpcds100_orc_shape/shape/query11.out | 65 - .../tpcds100_orc_shape/shape/query12.out | 28 - .../tpcds100_orc_shape/shape/query13.out | 41 - .../tpcds100_orc_shape/shape/query14.out | 170 --- .../tpcds100_orc_shape/shape/query15.out | 29 - .../tpcds100_orc_shape/shape/query16.out | 38 - .../tpcds100_orc_shape/shape/query17.out | 52 - .../tpcds100_orc_shape/shape/query18.out | 50 - .../tpcds100_orc_shape/shape/query19.out | 42 - .../tpcds100_orc_shape/shape/query2.out| 46 - .../tpcds100_orc_shape/shape/query20.out | 28 - .../tpcds100_orc_shape/shape/query21.out | 28 - .../tpcds100_orc_shape/shape/query22.out | 24 - .../tpcds100_orc_shape/shape/query23.out | 95 -- .../tpcds100_orc_shape/shape/query24.out | 59 - .../tpcds100_orc_shape/shape/query25.out | 51 - .../tpcds100_orc_shape/shape/query26.out | 36 - .../tpcds100_orc_shape/shape/query27.out | 37 - .../tpcds100_orc_shape/shape/query28.out | 63 - .../tpcds100_orc_shape/shape/query29.out | 50 - .../tpcds100_orc_shape/shape/query3.out| 25 - .../tpcds100_orc_shape/shape/query30.out | 48 - .../tpcds100_orc_shape/shape/query31.out | 76 -- .../tpcds100_orc_shape/shape/query32.out | 28 - .../tpcds100_orc_shape/shape/query33.out | 83 -- .../tpcds100_orc_shape/shape/query34.out | 35 - .../tpcds100_orc_shape/shape/query35.out | 59 - .../tpcds100_orc_shape/shape/query36.out | 37 - .../tpcds100_orc_shape/shape/query37.out | 30 - .../tpcds100_orc_shape/shape/query38.out | 62 - .../tpcds100_orc_shape/shape/query39.out | 40 - .../tpcds100_orc_shape/shape/query4.out| 94 -- .../tpcds100_orc_shape/shape/query40.out | 33 - .../tpcds100_orc_shape/shape/query41.out | 24 - .../tpcds100_orc_shape/shape/query42.out | 23 - .../tpcds100_orc_shape/shape/query43.out | 24 - .../tpcds100_orc_shape/shape/query44.out | 72 -- .../tpcds100_orc_shape/shape/query45.out | 42 - .../tpcds100_orc_shape/shape/query46.out | 45 - .../tpcds100_orc_shape/shape/query47.out | 50 - .../tpcds100_orc_shape/shape/query48.out | 34 - .../tpcds100_orc_shape/shape/query49.out | 93 -- .../tpcds100_orc_shape/shape/query5.out| 75 -- .../tpcds100_orc_shape/shape/query50.out | 33 - .../tpcds100_orc_shape/shape/query51.out | 52 - .../tpcds100_orc_shape/shape/query52.out | 24 - .../tpcds100_orc_shape/shape/query53.out | 37 - .../tpcds100_orc_shape/shape/query54.out | 86 -- .../tpcds100_orc_shape/shape/query55.out | 24 - .../tpcds100_orc_shape/shape/query56.out | 80 -- .../tpcds100_orc_shape/shape/query57.out | 51 - .../tpcds100_orc_shape/shape/query58.out | 97 -- .../tpcds100_orc_shape/shape/query59.out | 52 - .../tpcds100_orc_shape/shape/query6.out| 54 - .../tpcds100_orc_shape/shape/query60.out | 86 -- .../tpcds100_orc_shape/shape/query61.out | 82 -- .../tpcds100_orc_shape/shape/query62.out | 31 - .../tpcds100_orc_shape/shape/query63.out | 37 - .../tpcds100_orc_shape/shape/query64.out | 130 -- .../tpcds100_orc_shape/shape/query65.out | 46 - .../tpcds100_orc_shape/shape/query66.out | 53 - .../tpcds100_orc_shape/shape/query67.out | 34 - .../tpcds100_orc_shape/shape/query68.out | 47 - .../tpcds100_orc_shape/shape/query69.out | 58 - .../tpcds100_orc_shape/shape/query7.out| 36 - .../tpcds100_orc_shape/shape/query70.out | 55 - .../tpcds100_orc_shape/shape/query71.out | 52 - .../tpcds100_orc_shape/shape/query72.out
(doris) branch revert-25639-add_tpcds_shape_sf100_hive deleted (was 093dae1a0a5)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch revert-25639-add_tpcds_shape_sf100_hive in repository https://gitbox.apache.org/repos/asf/doris.git was 093dae1a0a5 Revert "[Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Feature](Job)Support manual and refactor some execution logic (#26082)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 08c78a11359 [Feature](Job)Support manual and refactor some execution
logic (#26082)
08c78a11359 is described below
commit 08c78a11359e3d0c9e6bc89cf32d4d9456c6c4d5
Author: Calvin Kirs
AuthorDate: Tue Oct 31 20:35:55 2023 +0800
[Feature](Job)Support manual and refactor some execution logic (#26082)
Supports manually triggered JOBs and Tasks
Optimize JOB&TASK display logic
Refactor the executor to support context passing
---
fe/fe-core/src/main/cup/sql_parser.cup | 6 +-
.../org/apache/doris/analysis/CreateJobStmt.java | 36 +-
.../org/apache/doris/analysis/ShowJobStmt.java | 11 +--
.../org/apache/doris/analysis/ShowJobTaskStmt.java | 4 +-
.../main/java/org/apache/doris/catalog/Env.java| 5 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 7 +-
.../apache/doris/scheduler/constants/JobType.java | 7 +-
.../apache/doris/scheduler/constants/TaskType.java | 16 -
.../doris/scheduler/disruptor/TaskDisruptor.java | 34 +++--
.../doris/scheduler/disruptor/TaskHandler.java | 26 ---
.../scheduler/executor/AbstractJobExecutor.java| 54 ++
.../doris/scheduler/executor/JobExecutor.java | 9 +--
.../doris/scheduler/executor/SqlJobExecutor.java | 28 +++-
.../java/org/apache/doris/scheduler/job/Job.java | 82 +-
.../org/apache/doris/scheduler/job/JobTask.java| 37 +-
.../doris/scheduler/manager/JobTaskManager.java| 17 +++--
.../doris/scheduler/manager/TimerJobManager.java | 75 +++-
.../scheduler/registry/PersistentJobRegister.java | 13
.../doris/scheduler/registry/TimerJobRegister.java | 5 ++
.../apache/doris/scheduler/disruptor/JobTest.java | 3 +-
.../scheduler/disruptor/TaskDisruptorTest.java | 13 +++-
.../scheduler/disruptor/TimerJobManagerTest.java | 14 ++--
22 files changed, 352 insertions(+), 150 deletions(-)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index 80a71b7490d..5933fcffbcb 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -2551,17 +2551,17 @@ resource_desc ::=
create_job_stmt ::=
KW_CREATE KW_JOB job_label:jobLabel KW_ON KW_SCHEDULER KW_EVERY
INTEGER_LITERAL:time_interval ident:time_unit opt_job_starts:startsTime
opt_job_ends:endsTime opt_comment:comment KW_DO stmt:executeSql
{:
-CreateJobStmt stmt = new
CreateJobStmt(jobLabel,null,false,time_interval,time_unit, startsTime,
endsTime,comment,executeSql);
+CreateJobStmt stmt = new
CreateJobStmt(jobLabel,"RECURRING",null,time_interval,time_unit, startsTime,
endsTime,comment,executeSql);
RESULT = stmt;
:}
| KW_CREATE KW_JOB job_label:jobLabel KW_ON KW_SCHEDULER KW_STREAMING
KW_AT STRING_LITERAL:atTime opt_comment:comment KW_DO stmt:executeSql
{:
-CreateJobStmt stmt = new
CreateJobStmt(jobLabel,atTime,true,null,null,null,null,comment,executeSql);
+CreateJobStmt stmt = new
CreateJobStmt(jobLabel,"STREAMING",atTime,null,null,null,null,comment,executeSql);
RESULT = stmt;
:}
| KW_CREATE KW_JOB job_label:jobLabel KW_ON KW_SCHEDULER KW_AT
STRING_LITERAL:atTime opt_comment:comment KW_DO stmt:executeSql
{:
-CreateJobStmt stmt = new
CreateJobStmt(jobLabel,atTime,false,null,null,null,null,comment,executeSql);
+CreateJobStmt stmt = new
CreateJobStmt(jobLabel,"ONE_TIME",atTime,null,null,null,null,comment,executeSql);
RESULT = stmt;
:}
;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateJobStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateJobStmt.java
index 999d630153d..57f976712c9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateJobStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateJobStmt.java
@@ -28,6 +28,7 @@ import org.apache.doris.qe.ConnectContext;
import org.apache.doris.scheduler.common.IntervalUnit;
import org.apache.doris.scheduler.constants.JobCategory;
import org.apache.doris.scheduler.constants.JobStatus;
+import org.apache.doris.scheduler.constants.JobType;
import org.apache.doris.scheduler.executor.SqlJobExecutor;
import org.apache.doris.scheduler.job.Job;
@@ -41,22 +42,21 @@ import java.util.HashSet;
/**
* syntax:
* CREATE
- * [DEFINER = user]
- * JOB
- * event_name
- * ON SCHEDULE schedule
- * [COMMENT 'string']
- * DO event_body;
+ * [DEFINER = user]
+ * JOB
+ * event_name
+ * ON SCHEDULE schedule
+ * [COMMENT 'string']
+ * DO event_body;
* schedule: {
- *[STREAMIN
[doris] branch revert-23803-eh/remove_outdated_tablestats created (now e25f7b60cb)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch revert-23803-eh/remove_outdated_tablestats in repository https://gitbox.apache.org/repos/asf/doris.git at e25f7b60cb Revert "[opt](stats) remove table stats when table has been removed (#23803)" This branch includes the following new commits: new e25f7b60cb Revert "[opt](stats) remove table stats when table has been removed (#23803)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/01: Revert "[opt](stats) remove table stats when table has been removed (#23803)"
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch revert-23803-eh/remove_outdated_tablestats
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e25f7b60cb057cd73a0db2b47932b86a3de83d45
Author: zy-kkk
AuthorDate: Thu Sep 7 21:20:08 2023 +0800
Revert "[opt](stats) remove table stats when table has been removed
(#23803)"
This reverts commit 66d3371400207f568c7ff6ff6bf5f4f0da32bd2c.
---
.../org/apache/doris/datasource/CatalogIf.java | 4 +--
.../org/apache/doris/datasource/CatalogMgr.java| 4 +--
.../apache/doris/datasource/ExternalCatalog.java | 3 +-
.../apache/doris/datasource/InternalCatalog.java | 3 +-
.../java/org/apache/doris/persist/EditLog.java | 8 -
.../org/apache/doris/persist/OperationType.java| 2 --
.../doris/persist/TableStatsDeletionLog.java | 42 --
.../org/apache/doris/statistics/AnalysisInfo.java | 2 --
.../apache/doris/statistics/AnalysisManager.java | 40 ++---
.../suites/statistics/analyze_stats.groovy | 4 ---
10 files changed, 9 insertions(+), 103 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
index 17874ba0f8..d135018e75 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
@@ -20,7 +20,6 @@ package org.apache.doris.datasource;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
@@ -169,7 +168,8 @@ public interface CatalogIf {
}
// Return a copy of all db collection.
-public Collection> getAllDbs();
+@SuppressWarnings({"rawtypes", "unchecked"})
+public Collection getAllDbs();
public boolean enableAutoAnalyze();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
index f8725a2d2a..253cfa2f69 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
@@ -289,7 +289,7 @@ public class CatalogMgr implements Writable,
GsonPostProcessable {
LOG.warn("Non catalog {} is found.", stmt.getCatalogName());
return;
}
-CatalogIf> catalog =
nameToCatalog.get(stmt.getCatalogName());
+CatalogIf catalog = nameToCatalog.get(stmt.getCatalogName());
if (catalog == null) {
throw new DdlException("No catalog found with name: " +
stmt.getCatalogName());
}
@@ -299,7 +299,7 @@ public class CatalogMgr implements Writable,
GsonPostProcessable {
lastDBOfCatalog.remove(stmt.getCatalogName());
Env.getCurrentEnv().getQueryStats().clear(catalog.getId());
-
Env.getCurrentEnv().getAnalysisManager().removeExternalTableStats(catalog);
+
} finally {
writeUnlock();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index 55a0e9a1fd..35d03dfabc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -21,7 +21,6 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.Resource;
-import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.external.DeltaLakeExternalDataBase;
import org.apache.doris.catalog.external.EsExternalDatabase;
import org.apache.doris.catalog.external.ExternalDatabase;
@@ -589,7 +588,7 @@ public abstract class ExternalCatalog
}
@Override
-public Collection> getAllDbs() {
+public Collection getAllDbs() {
makeSureInitialized();
return new HashSet<>(idToDb.values());
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index e6f4f65e02..4114386680 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -919,7 +919,6 @@ public class InternalCatalog implements CatalogIf
{
Env.getCurrentEnv().getEditLog().logDropTable(info);
Env.getCurrentEnv().getQueryStats().clear(Env.getCurrentEnv().getCurre
[doris] branch master updated (cdb1b341c7 -> a27349c83a)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cdb1b341c7 [pipelineX](runtime filter) Support runtime filter (#24054) add a27349c83a [fix](Export) Concatenation the outfile sql for Export (#23635) No new revisions were added by this update. Summary of changes: .../Manipulation/EXPORT.md |2 - .../Manipulation/EXPORT.md |2 - .../antlr4/org/apache/doris/nereids/DorisParser.g4 |2 +- .../java/org/apache/doris/analysis/BrokerDesc.java | 12 +- .../org/apache/doris/analysis/BrokerLoadStmt.java |5 +- .../java/org/apache/doris/analysis/ExportStmt.java | 60 +- .../java/org/apache/doris/analysis/LoadStmt.java |5 +- .../org/apache/doris/analysis/StorageBackend.java | 39 +- .../main/java/org/apache/doris/catalog/Env.java|8 + .../main/java/org/apache/doris/load/ExportJob.java | 200 ++- .../main/java/org/apache/doris/load/ExportMgr.java | 53 +- .../org/apache/doris/load/ExportTaskExecutor.java | 39 +- .../doris/nereids/parser/LogicalPlanBuilder.java | 24 +- .../trees/plans/commands/ExportCommand.java| 327 +++- .../java/org/apache/doris/persist/EditLog.java |1 + .../java/org/apache/doris/qe/StmtExecutor.java | 18 +- .../tablefunction/HdfsTableValuedFunction.java |4 +- .../analysis/ExportToOutfileLogicalPlanTest.java | 462 ++ .../data/export_p0/test_export_basic.out | 169 +- regression-test/data/export_p0/test_export_csv.out | 270 .../export_p2/test_export_with_parallelism.out | 1633 .../suites/export_p0/test_export_basic.groovy | 140 +- .../suites/export_p0/test_export_csv.groovy| 19 +- .../suites/export_p0/test_export_data_types.groovy |5 + .../export_p0/test_export_empty_table.groovy |5 + .../suites/export_p0/test_export_orc.groovy|5 + .../suites/export_p0/test_export_parquet.groovy|5 + .../suites/export_p0/test_outfile_expr.groovy |5 + .../suites/export_p0/test_outfile_parquet.groovy |5 + .../suites/export_p0/test_outfile_separator.groovy |5 + .../suites/export_p2/test_export_big_data.groovy |5 + .../export_p2/test_export_max_file_size.groovy | 10 +- .../suites/export_p2/test_export_with_hdfs.groovy |5 + .../export_p2/test_export_with_parallelism.groovy | 165 ++ .../suites/export_p2/test_export_with_s3.groovy|4 + .../test_outfile_orc_max_file_size.groovy |5 + 36 files changed, 3213 insertions(+), 510 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/analysis/ExportToOutfileLogicalPlanTest.java create mode 100644 regression-test/data/export_p2/test_export_with_parallelism.out create mode 100644 regression-test/suites/export_p2/test_export_with_parallelism.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (cb29d1a395 -> c68e6a9ca8)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cb29d1a395 fix compile error with gcc12 (#24049) add c68e6a9ca8 [Fix](catalog) Doris datetime type conversion failed (#23906) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (894aa48743 -> 69cc6fee97)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 894aa48743 [fix](Nereids) remove PARTITIONS from non-reserved list (#24110) add 69cc6fee97 [fix](explain) fix explain physical plan with external table issue (#23845) No new revisions were added by this update. Summary of changes: .../doris/nereids/trees/plans/logical/LogicalFileScan.java | 12 +--- .../suites/external_table_p0/hive/test_hive_parquet.groovy | 6 ++ 2 files changed, 15 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-thirdparty] branch clucene updated: [Fix](PFOR) fix build error in arm (#120)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch clucene
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/clucene by this push:
new 2761b1af [Fix](PFOR) fix build error in arm (#120)
2761b1af is described below
commit 2761b1afe48cb1bd272ce8959e3aa3049b6e63ac
Author: airborne12
AuthorDate: Sat Sep 9 13:29:03 2023 +0800
[Fix](PFOR) fix build error in arm (#120)
---
src/core/CLucene/util/PFORUtil.cpp | 5 -
src/core/CLucene/util/PFORUtil.h | 1 -
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/core/CLucene/util/PFORUtil.cpp
b/src/core/CLucene/util/PFORUtil.cpp
index eaf628c7..a9a32dfb 100644
--- a/src/core/CLucene/util/PFORUtil.cpp
+++ b/src/core/CLucene/util/PFORUtil.cpp
@@ -16,6 +16,9 @@
// under the License.
#include "PFORUtil.h"
#include "vp4.h"
+#if (defined(__i386) || defined(__x86_64__))
+#include
+#endif
namespace {
using DEC_FUNC = size_t (*)(unsigned char *__restrict, size_t, uint32_t
*__restrict);
@@ -27,12 +30,12 @@ ENC_FUNC g_p4nzenc;
} // anonymous namespace
__attribute__((constructor)) void SelectPFORFunctions() {
+#if (defined(__i386) || defined(__x86_64__))
uint32_t eax, ebx, ecx, edx;
__cpuid(1, eax, ebx, ecx, edx);
bool sse2 = (edx & bit_SSE2) != 0;
bool sse42 = (ecx & bit_SSE4_2) != 0;
-#if (defined(__i386) || defined(__x86_64__))
#if defined(USE_AVX2)
g_p4nd1dec = p4nd1dec256v32;
g_p4nzdec = p4nzdec256v32;
diff --git a/src/core/CLucene/util/PFORUtil.h b/src/core/CLucene/util/PFORUtil.h
index 5ba664aa..29acb7fe 100644
--- a/src/core/CLucene/util/PFORUtil.h
+++ b/src/core/CLucene/util/PFORUtil.h
@@ -18,7 +18,6 @@
#include
#include
-#include
size_t P4DEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out);
size_t P4NZDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict
out);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Fix](clucene) fix clucene build error in arm (#24130)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new c3f3195721 [Fix](clucene) fix clucene build error in arm (#24130) c3f3195721 is described below commit c3f3195721f7cc504ac164b9a9118b41dc046af2 Author: airborne12 AuthorDate: Sat Sep 9 15:31:40 2023 +0800 [Fix](clucene) fix clucene build error in arm (#24130) --- be/src/clucene | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/src/clucene b/be/src/clucene index fd45366505..2761b1afe4 16 --- a/be/src/clucene +++ b/be/src/clucene @@ -1 +1 @@ -Subproject commit fd453665055c65b94892d13a93ac47180afd72bb +Subproject commit 2761b1afe48cb1bd272ce8959e3aa3049b6e63ac - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch revert-23803-eh/remove_outdated_tablestats deleted (was e25f7b60cb)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch revert-23803-eh/remove_outdated_tablestats in repository https://gitbox.apache.org/repos/asf/doris.git was e25f7b60cb Revert "[opt](stats) remove table stats when table has been removed (#23803)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](jdbc) add profile for jdbc read and convert phase (#23962)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f85da7d942 [improvement](jdbc) add profile for jdbc read and convert
phase (#23962)
f85da7d942 is described below
commit f85da7d942e6102dd61060e60e23fe62ff48cdd1
Author: Mingyu Chen
AuthorDate: Sun Sep 10 21:42:06 2023 +0800
[improvement](jdbc) add profile for jdbc read and convert phase (#23962)
Add 2 metrics in jdbc scan node profile:
- `CallJniNextTime`: call get next from jdbc result set
- `ConvertBatchTime`: call convert jobject to columm block
Also fix a potential concurrency issue when init jdbc connection cache pool
---
be/src/vec/exec/scan/new_jdbc_scanner.cpp | 6 +++
be/src/vec/exec/scan/new_jdbc_scanner.h| 2 +
be/src/vec/exec/vjdbc_connector.h | 2 +
.../java/org/apache/doris/jdbc/JdbcExecutor.java | 56 ++
4 files changed, 45 insertions(+), 21 deletions(-)
diff --git a/be/src/vec/exec/scan/new_jdbc_scanner.cpp
b/be/src/vec/exec/scan/new_jdbc_scanner.cpp
index bc8bf6e044..3d71aeb4f7 100644
--- a/be/src/vec/exec/scan/new_jdbc_scanner.cpp
+++ b/be/src/vec/exec/scan/new_jdbc_scanner.cpp
@@ -50,6 +50,10 @@ NewJdbcScanner::NewJdbcScanner(RuntimeState* state,
NewJdbcScanNode* parent, int
_init_connector_timer = ADD_TIMER(get_parent()->_scanner_profile,
"InitConnectorTime");
_check_type_timer = ADD_TIMER(get_parent()->_scanner_profile,
"CheckTypeTime");
_get_data_timer = ADD_TIMER(get_parent()->_scanner_profile, "GetDataTime");
+_call_jni_next_timer =
+ADD_CHILD_TIMER(get_parent()->_scanner_profile, "CallJniNextTime",
"GetDataTime");
+_convert_batch_timer =
+ADD_CHILD_TIMER(get_parent()->_scanner_profile,
"ConvertBatchTime", "GetDataTime");
_execte_read_timer = ADD_TIMER(get_parent()->_scanner_profile,
"ExecteReadTime");
_connector_close_timer = ADD_TIMER(get_parent()->_scanner_profile,
"ConnectorCloseTime");
}
@@ -186,6 +190,8 @@ void NewJdbcScanner::_update_profile() {
COUNTER_UPDATE(_init_connector_timer,
jdbc_statistic._init_connector_timer);
COUNTER_UPDATE(_check_type_timer, jdbc_statistic._check_type_timer);
COUNTER_UPDATE(_get_data_timer, jdbc_statistic._get_data_timer);
+COUNTER_UPDATE(_call_jni_next_timer, jdbc_statistic._call_jni_next_timer);
+COUNTER_UPDATE(_convert_batch_timer, jdbc_statistic._convert_batch_timer);
COUNTER_UPDATE(_execte_read_timer, jdbc_statistic._execte_read_timer);
COUNTER_UPDATE(_connector_close_timer,
jdbc_statistic._connector_close_timer);
}
diff --git a/be/src/vec/exec/scan/new_jdbc_scanner.h
b/be/src/vec/exec/scan/new_jdbc_scanner.h
index 1fb1b64874..f52e711056 100644
--- a/be/src/vec/exec/scan/new_jdbc_scanner.h
+++ b/be/src/vec/exec/scan/new_jdbc_scanner.h
@@ -60,6 +60,8 @@ protected:
RuntimeProfile::Counter* _load_jar_timer = nullptr;
RuntimeProfile::Counter* _init_connector_timer = nullptr;
RuntimeProfile::Counter* _get_data_timer = nullptr;
+RuntimeProfile::Counter* _call_jni_next_timer = nullptr;
+RuntimeProfile::Counter* _convert_batch_timer = nullptr;
RuntimeProfile::Counter* _check_type_timer = nullptr;
RuntimeProfile::Counter* _execte_read_timer = nullptr;
RuntimeProfile::Counter* _connector_close_timer = nullptr;
diff --git a/be/src/vec/exec/vjdbc_connector.h
b/be/src/vec/exec/vjdbc_connector.h
index 75ff1ab36b..50f7d669ca 100644
--- a/be/src/vec/exec/vjdbc_connector.h
+++ b/be/src/vec/exec/vjdbc_connector.h
@@ -64,6 +64,8 @@ public:
int64_t _load_jar_timer = 0;
int64_t _init_connector_timer = 0;
int64_t _get_data_timer = 0;
+int64_t _call_jni_next_timer = 0;
+int64_t _convert_batch_timer = 0;
int64_t _check_type_timer = 0;
int64_t _execte_read_timer = 0;
int64_t _connector_close_timer = 0;
diff --git
a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcExecutor.java
b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcExecutor.java
index f84761a20e..4541de0f4c 100644
---
a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcExecutor.java
+++
b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcExecutor.java
@@ -85,6 +85,7 @@ public class JdbcExecutor {
private int curBlockRows = 0;
private static final byte[] emptyBytes = new byte[0];
private DruidDataSource druidDataSource = null;
+private byte[] druidDataSourceLock = new byte[0];
private int minPoolSize;
private int maxPoolSize;
private int minIdleSize;
@@ -410,29 +411,41 @@ pu
[doris] branch master updated: Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle for performance boost (#15972)" (#24146)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ebac816e85 Revert "[improvement](bitshuffle)Enable avx512 support in
bitshuffle for performance boost (#15972)" (#24146)
ebac816e85 is described below
commit ebac816e85ea93b07c77509a231362f92fe24b5f
Author: Kang
AuthorDate: Sun Sep 10 23:06:21 2023 +0800
Revert "[improvement](bitshuffle)Enable avx512 support in bitshuffle for
performance boost (#15972)" (#24146)
This reverts commit 28fcc093a8958a6870fec9802b23db07a42bbd7b.
---
be/src/gutil/cpu.cc | 4
be/src/gutil/cpu.h | 3 ---
be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp | 17 +
3 files changed, 1 insertion(+), 23 deletions(-)
diff --git a/be/src/gutil/cpu.cc b/be/src/gutil/cpu.cc
index 299703c15d..3aff485f39 100644
--- a/be/src/gutil/cpu.cc
+++ b/be/src/gutil/cpu.cc
@@ -69,7 +69,6 @@ CPU::CPU()
has_popcnt_(false),
has_avx_(false),
has_avx2_(false),
- has_avx512_(false),
has_aesni_(false),
has_non_stop_time_stamp_counter_(false),
is_running_in_vm_(false),
@@ -201,8 +200,6 @@ void CPU::Initialize() {
(xgetbv(0) & 6) == 6 /* XSAVE enabled by kernel */;
has_aesni_ = (cpu_info[2] & 0x0200) != 0;
has_avx2_ = has_avx_ && (cpu_info7[1] & 0x0020) != 0;
-has_avx512_ = has_avx2_ && (cpu_info7[1] & 0x0001) != 0 &&
- (cpu_info7[1] & 0x4000) != 0 && (cpu_info7[1] &
0x8000) != 0;
}
// Get the brand string of the cpu.
__cpuid(cpu_info, 0x8000);
@@ -255,7 +252,6 @@ void CPU::Initialize() {
#endif
}
CPU::IntelMicroArchitecture CPU::GetIntelMicroArchitecture() const {
-if (has_avx512()) return AVX512;
if (has_avx2()) return AVX2;
if (has_avx()) return AVX;
if (has_sse42()) return SSE42;
diff --git a/be/src/gutil/cpu.h b/be/src/gutil/cpu.h
index 82b87a1fb3..f7a12bbe43 100644
--- a/be/src/gutil/cpu.h
+++ b/be/src/gutil/cpu.h
@@ -60,7 +60,6 @@ public:
SSE42,
AVX,
AVX2,
-AVX512,
MAX_INTEL_MICRO_ARCHITECTURE
};
// Accessors for CPU information.
@@ -82,7 +81,6 @@ public:
bool has_popcnt() const { return has_popcnt_; }
bool has_avx() const { return has_avx_; }
bool has_avx2() const { return has_avx2_; }
-bool has_avx512() const { return has_avx512_; }
bool has_aesni() const { return has_aesni_; }
bool has_non_stop_time_stamp_counter() const { return
has_non_stop_time_stamp_counter_; }
bool is_running_in_vm() const { return is_running_in_vm_; }
@@ -109,7 +107,6 @@ private:
bool has_popcnt_;
bool has_avx_;
bool has_avx2_;
-bool has_avx512_;
bool has_aesni_;
bool has_non_stop_time_stamp_counter_;
bool is_running_in_vm_;
diff --git a/be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp
b/be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp
index 7e569f92b2..7ad20f210c 100644
--- a/be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp
+++ b/be/src/olap/rowset/segment_v2/bitshuffle_wrapper.cpp
@@ -34,17 +34,6 @@
#undef bshuf_compress_lz4
#undef bshuf_decompress_lz4
-// Include the bitshuffle header again, but this time importing the
-// AVX512-compiled symbols by defining some macros.
-#undef BITSHUFFLE_H
-#define bshuf_compress_lz4_bound bshuf_compress_lz4_bound_avx512
-#define bshuf_compress_lz4 bshuf_compress_lz4_avx512
-#define bshuf_decompress_lz4 bshuf_decompress_lz4_avx512
-#include // NOLINT(*)
-#undef bshuf_compress_lz4_bound
-#undef bshuf_compress_lz4
-#undef bshuf_decompress_lz4
-
using base::CPU;
namespace doris {
@@ -65,11 +54,7 @@ decltype(&bshuf_decompress_lz4) g_bshuf_decompress_lz4;
// the cost of a 'std::once' call.
__attribute__((constructor)) void SelectBitshuffleFunctions() {
#if (defined(__i386) || defined(__x86_64__))
-if (CPU().has_avx512()) {
-g_bshuf_compress_lz4_bound = bshuf_compress_lz4_bound_avx512;
-g_bshuf_compress_lz4 = bshuf_compress_lz4_avx512;
-g_bshuf_decompress_lz4 = bshuf_decompress_lz4_avx512;
-} else if (CPU().has_avx2()) {
+if (CPU().has_avx2()) {
g_bshuf_compress_lz4_bound = bshuf_compress_lz4_bound_avx2;
g_bshuf_compress_lz4 = bshuf_compress_lz4_avx2;
g_bshuf_decompress_lz4 = bshuf_decompress_lz4_avx2;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (cd13f9e8c6 -> 480fcef0a1)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from cd13f9e8c6 [BUG](view) fix can't create view with lambda function (#23942) add 480fcef0a1 [typo](errmsg) Improve partition error message (#23968) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/util/DynamicPartitionUtil.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](jdbc catalog) optimize the JDBC Catalog connection error message (#23868)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d2cd0c30c7 [improvement](jdbc catalog) optimize the JDBC Catalog
connection error message (#23868)
d2cd0c30c7 is described below
commit d2cd0c30c770a8cd1bd2395680b260e9dfca355e
Author: zy-kkk
AuthorDate: Mon Sep 11 10:26:54 2023 +0800
[improvement](jdbc catalog) optimize the JDBC Catalog connection error
message (#23868)
---
.../java/org/apache/doris/datasource/jdbc/client/JdbcClient.java | 5 +++--
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java | 5 +
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
index 90070bdba7..8ffc5df3ca 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
@@ -162,8 +162,9 @@ public abstract class JdbcClient {
try {
conn = dataSource.getConnection();
} catch (Exception e) {
-throw new JdbcClientException("Can not connect to jdbc due to
error: %s, catalog name: %s", e.getMessage(),
-this.catalog, e);
+String errorMessage = String.format("Can not connect to jdbc due
to error: %s, Catalog name: %s", e,
+this.getCatalog());
+throw new JdbcClientException(errorMessage, e);
}
return conn;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index a4d1e1880a..280729b01b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -104,6 +104,7 @@ import
org.apache.doris.common.util.ProfileManager.ProfileType;
import org.apache.doris.common.util.SqlParserUtils;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.jdbc.client.JdbcClientException;
import org.apache.doris.load.EtlJobType;
import org.apache.doris.load.LoadJobRowResult;
import org.apache.doris.load.loadv2.LoadManager;
@@ -778,6 +779,10 @@ public class StmtExecutor {
LOG.warn("execute Exception. {}", context.getQueryIdentifier(), e);
context.getState().setError(e.getMysqlErrorCode(), e.getMessage());
context.getState().setErrType(QueryState.ErrType.ANALYSIS_ERR);
+} catch (JdbcClientException e) {
+LOG.warn("execute Exception. {}", context.getQueryIdentifier(), e);
+context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR,
+e.getMessage());
} catch (Exception e) {
LOG.warn("execute Exception. {}", context.getQueryIdentifier(), e);
context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR,
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](jdbc catalog) Added create jdbc catalog properties validation (#23764)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d18d272ac2 [improvement](jdbc catalog) Added create jdbc catalog
properties validation (#23764)
d18d272ac2 is described below
commit d18d272ac2f26586de26ccbb45035f193ff4d562
Author: zy-kkk
AuthorDate: Mon Sep 11 10:38:53 2023 +0800
[improvement](jdbc catalog) Added create jdbc catalog properties validation
(#23764)
---
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 17 ++
.../datasource/jdbc/JdbcExternalCatalogTest.java | 63 ++
2 files changed, 80 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
index 7617ad7180..9b2d9b00f8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
@@ -177,5 +177,22 @@ public class JdbcExternalCatalog extends ExternalCatalog {
properties.put(JdbcResource.CHECK_SUM,
JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
}
+String onlySpecifiedDatabase = getOnlySpecifiedDatabase();
+if (!onlySpecifiedDatabase.equalsIgnoreCase("true") &&
!onlySpecifiedDatabase.equalsIgnoreCase("false")) {
+throw new DdlException("only_specified_database must be true or
false");
+}
+String lowerCaseTableNames = getLowerCaseTableNames();
+if (!lowerCaseTableNames.equalsIgnoreCase("true") &&
!lowerCaseTableNames.equalsIgnoreCase("false")) {
+throw new DdlException("lower_case_table_names must be true or
false");
+}
+if (!onlySpecifiedDatabase.equalsIgnoreCase("true")) {
+Map includeDatabaseList = getIncludeDatabaseMap();
+Map excludeDatabaseList = getExcludeDatabaseMap();
+if ((includeDatabaseList != null && !includeDatabaseList.isEmpty())
+|| (excludeDatabaseList != null &&
!excludeDatabaseList.isEmpty())) {
+throw new DdlException("include_database_list and
exclude_database_list can not be set when "
++ "only_specified_database is false");
+}
+}
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
new file mode 100644
index 00..0f2977a988
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.jdbc;
+
+import org.apache.doris.catalog.JdbcResource;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.FeConstants;
+
+import org.junit.Assert;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class JdbcExternalCatalogTest {
+private JdbcExternalCatalog jdbcExternalCatalog;
+
+@BeforeEach
+public void setUp() throws DdlException {
+FeConstants.runningUnitTest = true;
+Map properties = new HashMap<>();
+properties.put(JdbcResource.DRIVER_URL, "ojdbc8.jar");
+properties.put(JdbcResource.JDBC_URL,
"jdbc:oracle:thin:@127.0.0.1:1521:XE");
+properties.put(JdbcResource.DRIVER_CLASS,
"oracle.jdbc.driver.OracleDriver");
+jdbcExternalCatalog = new JdbcExternalCatalog(1L, "testCatalog",
"testResource", properties, "testComment");
+}
+
+@Test
+public void setDefaultPropsWhenCreatingTest() {
+
jdbcExternalCatalog.ge
[doris] branch master updated (d18d272ac2 -> fb6cb88341)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d18d272ac2 [improvement](jdbc catalog) Added create jdbc catalog properties validation (#23764) add fb6cb88341 [feature-wip](dbt) dbt view columns comment and view rename change (#23917) No new revisions were added by this update. Summary of changes: .../dbt-doris/dbt/adapters/doris/__version__.py| 2 +- extension/dbt-doris/dbt/adapters/doris/impl.py | 9 +++- .../dbt-doris/dbt/include/doris/dbt_project.yml| 2 +- .../dbt/include/doris/macros/adapters/columns.sql | 7 ++- .../materializations/view/create_view_as.sql | 9 +++- .../doris/macros/materializations/view/view.sql| 52 +++--- extension/dbt-doris/setup.py | 2 +- 7 files changed, 40 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (c94e47583c -> db139cfd6e)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c94e47583c [fix](join) avoid DCHECK failed in '_filter_data_and_build_output' (#24162) add db139cfd6e [fix](log) delete useless log (#24161) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/qe/StmtExecutor.java | 16 1 file changed, 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improve](Job)Support other types of Job query interfaces (#24172)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 232f120edc [Improve](Job)Support other types of Job query interfaces
(#24172)
232f120edc is described below
commit 232f120edc713955741898af94698dca247ae7de
Author: Calvin Kirs
AuthorDate: Tue Sep 12 13:55:56 2023 +0800
[Improve](Job)Support other types of Job query interfaces (#24172)
- Support MTMV job
- Task info add create time and sql
- Optimize scheduling logic
---
.../main/java/org/apache/doris/common/Config.java | 4 +-
fe/fe-core/src/main/cup/sql_parser.cup | 6 +-
.../org/apache/doris/analysis/ShowJobStmt.java | 44 +-
.../org/apache/doris/analysis/ShowJobTaskStmt.java | 29 ++---
.../main/java/org/apache/doris/catalog/Env.java| 1 +
.../java/org/apache/doris/qe/ShowExecutor.java | 5 +-
.../doris/scheduler/constants/JobCategory.java | 13 +++--
.../doris/scheduler/disruptor/TaskDisruptor.java | 20 +++
.../doris/scheduler/disruptor/TaskEvent.java | 2 +
.../doris/scheduler/disruptor/TaskHandler.java | 32 +++---
.../doris/scheduler/executor/JobExecutor.java | 3 +-
.../doris/scheduler/executor/SqlJobExecutor.java | 14 +++--
.../scheduler/executor/TransientTaskExecutor.java | 2 +-
.../ExecutorResult.java} | 29 +
.../java/org/apache/doris/scheduler/job/Job.java | 13 -
.../org/apache/doris/scheduler/job/JobTask.java| 18 +-
.../apache/doris/scheduler/job/TimerJobTask.java | 9 ++-
.../doris/scheduler/manager/JobTaskManager.java| 26 +
.../doris/scheduler/manager/TimerJobManager.java | 68 --
.../scheduler/disruptor/TaskDisruptorTest.java | 7 ++-
.../scheduler/disruptor/TimerJobManagerTest.java | 6 +-
21 files changed, 241 insertions(+), 110 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 54a0e814a4..fd5258e0c0 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1554,7 +1554,7 @@ public class Config extends ConfigBase {
public static boolean enable_pipeline_load = false;
@ConfField
-public static int scheduler_job_task_max_saved_count = 10;
+public static int scheduler_job_task_max_saved_count = 20;
/**
* The number of async tasks that can be queued. @See TaskDisruptor
@@ -1568,7 +1568,7 @@ public class Config extends ConfigBase {
* if we have a lot of async tasks, we need more threads to consume them.
Sure, it's depends on the cpu cores.
*/
@ConfField
-public static int async_task_consumer_thread_num = 10;
+public static int async_task_consumer_thread_num = 5;
// enable_workload_group should be immutable and temporarily set to
mutable during the development test phase
@ConfField(mutable = true, varType = VariableAnnotation.EXPERIMENTAL)
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup
b/fe/fe-core/src/main/cup/sql_parser.cup
index 94fcd2260f..99c65ffee2 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -2689,15 +2689,15 @@ resume_job_stmt ::=
show_job_stmt ::=
KW_SHOW KW_JOBS
{:
-RESULT = new ShowJobStmt(null,null);
+RESULT = new ShowJobStmt(null,null,null);
:}
| KW_SHOW KW_JOB KW_FOR job_label:jobLabel
{:
-RESULT = new ShowJobStmt(jobLabel,null);
+RESULT = new ShowJobStmt(null,jobLabel,null);
:}
| KW_SHOW KW_JOB KW_TASKS KW_FOR job_label:jobLabel
{:
-RESULT = new ShowJobTaskStmt(jobLabel);
+RESULT = new ShowJobTaskStmt(null,jobLabel);
:}
;
pause_job_stmt ::=
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowJobStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowJobStmt.java
index 56fc916b2a..7ad3ce343c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowJobStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowJobStmt.java
@@ -26,18 +26,21 @@ import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.scheduler.constants.JobCategory;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
+import lombok.Getter;
+import org.apache.commons.lang3.StringUtils;
import java.util.List;
/**
* SHOW JOB [FOR JobName]
* eg: show event
- * return all job in connection db
+ * return all job in connection db
* eg: show event for test
- * return job named te
[doris] branch master updated (dad671af8e -> 5238be24a2)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from dad671af8e [feature](nereids)prune runtime filter (tpch part) #19312 add 5238be24a2 [fix](jdbc catalog) Ensure Thread Safety by Refactoring isDoris&convertDateToNull Static Variable in JdbcMySQLClient (#24253) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (07dd6830e8 -> dbfacdc4af)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 07dd6830e8 [pipelineX](refactor) add union node in pipelineX (#24286) add dbfacdc4af [improvement](jdbc catalog) Optimize Loop Performance by Caching `isNebula` Method Result (#24260) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/jdbc/JdbcExecutor.java | 30 ++ 1 file changed, 20 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch master updated: [fix](oracle) add oracle regexp like compatible (#194)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new da648ac [fix](oracle) add oracle regexp like compatible (#194)
da648ac is described below
commit da648ac4c1a53bb5769b1fc193e054f9f880a62a
Author: wudi <676366...@qq.com>
AuthorDate: Thu Sep 14 15:53:59 2023 +0800
[fix](oracle) add oracle regexp like compatible (#194)
When include-tables has too many table names, and debezium incrementally
reads, it will be judged based on `regexp_like`. When the regular length
exceeds 512, an error will be reported, like `ORA-12733: regular expression too
long`
---
.../main/java/org/apache/doris/flink/lookup/ExecutionPool.java | 4 ++--
.../main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java | 4
.../apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java | 9 -
3 files changed, 14 insertions(+), 3 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/lookup/ExecutionPool.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/lookup/ExecutionPool.java
index 9b930ff..94cb35c 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/lookup/ExecutionPool.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/lookup/ExecutionPool.java
@@ -104,9 +104,9 @@ public class ExecutionPool implements Closeable {
public void close() throws IOException {
if (started.compareAndSet(true, false)) {
LOG.info("close executorService");
-actionWatcherExecutorService.shutdownNow();
+actionWatcherExecutorService.shutdown();
+workerExecutorService.shutdown();
workerStated.set(false);
-workerExecutorService.shutdownNow();
this.actionWatcherExecutorService = null;
this.workerExecutorService = null;
this.semaphore = null;
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
index 455000e..e36590c 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
@@ -61,6 +61,8 @@ public abstract class DatabaseSync {
public StreamExecutionEnvironment env;
private boolean createTableOnly = false;
private boolean newSchemaChange;
+protected String includingTables;
+protected String excludingTables;
public abstract Connection getConnection() throws SQLException;
@@ -76,6 +78,8 @@ public abstract class DatabaseSync {
this.config = config;
this.database = database;
this.converter = new TableNameConverter(tablePrefix, tableSuffix);
+this.includingTables = includingTables;
+this.excludingTables = excludingTables;
this.includingPattern = includingTables == null ? null :
Pattern.compile(includingTables);
this.excludingPattern = excludingTables == null ? null :
Pattern.compile(excludingTables);
this.ignoreDefaultValue = ignoreDefaultValue;
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java
index 3b4bc31..bfd974f 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/oracle/OracleDatabaseSync.java
@@ -107,11 +107,19 @@ public class OracleDatabaseSync extends DatabaseSync {
@Override
public DataStreamSource buildCdcSource(StreamExecutionEnvironment
env) {
+Properties debeziumProperties = new Properties();
String databaseName = config.get(OracleSourceOptions.DATABASE_NAME);
String schemaName = config.get(OracleSourceOptions.SCHEMA_NAME);
Preconditions.checkNotNull(databaseName, "database-name in oracle is
required");
Preconditions.checkNotNull(schemaName, "schema-name in oracle is
required");
String tableName = config.get(OracleSourceOptions.TABLE_NAME);
+//When debezium incrementally reads, it will be judged based on
regexp_like.
+//When the regular length exceeds 512, an error will be reported, like
ORA-12733: regular expression too long
+if(tableName.length() > 384){
+//max database name length 128
+tableName = StringUtils.isNullOrWhitespaceOnly(includingTables) ?
".*" : includingTables;
+}
+
String url = config.get(OracleSourceOptions.URL);
String h
[doris-flink-connector] branch master updated: [Fix](mysql) fix mysql datetime missing millisecond digits (#195)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 666411c [Fix](mysql) fix mysql datetime missing millisecond digits
(#195)
666411c is described below
commit 666411c6a1a1cbb12be263944645d1e24665b3fc
Author: wudi <676366...@qq.com>
AuthorDate: Fri Sep 15 11:33:35 2023 +0800
[Fix](mysql) fix mysql datetime missing millisecond digits (#195)
Co-authored-by: wudi <>
---
.../doris/flink/tools/cdc/{ => mysql}/DateToStringConverter.java| 6 +++---
.../org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java | 1 -
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DateToStringConverter.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java
similarity index 98%
rename from
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DateToStringConverter.java
rename to
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java
index 60a9718..e779be0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DateToStringConverter.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/DateToStringConverter.java
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.flink.tools.cdc;
+package org.apache.doris.flink.tools.cdc.mysql;
import
com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.data.SchemaBuilder;
import io.debezium.spi.converter.CustomConverter;
@@ -47,9 +47,9 @@ public class DateToStringConverter implements
CustomConverter
[doris] branch master updated (699023069d -> c270a2d89f)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 699023069d [regression](lateral view) add test case for explode_bitmap (#24421) add c270a2d89f [typo] (docs)Row storage mode can only be turned on when creating a table (#24257) No new revisions were added by this update. Summary of changes: docs/en/docs/query-acceleration/hight-concurrent-point-query.md| 4 ++-- docs/zh-CN/docs/query-acceleration/hight-concurrent-point-query.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](jdbc catalog) refactor JdbcFunctionPushDownRule (#23826)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1c142309a6 [refactor](jdbc catalog) refactor JdbcFunctionPushDownRule
(#23826)
1c142309a6 is described below
commit 1c142309a6e69a78c4a3db7a336c40ffccd69e47
Author: zy-kkk
AuthorDate: Fri Sep 15 22:16:07 2023 +0800
[refactor](jdbc catalog) refactor JdbcFunctionPushDownRule (#23826)
1. Change from using string matching function to using Expr matching
2. Replace the `nvl` function with `ifnull` when pushed down to MySQL
3. Adapt ClickHouse's `from_unixtime` function to push down
4. Non-function filtering can still be pushed down when
`enable_func_pushdown` is set to false
---
.../docker-compose/clickhouse/clickhouse.yaml.tpl | 2 +-
.../clickhouse/init/03-create-table.sql| 8 ++
.../docker-compose/clickhouse/init/04-insert.sql | 3 +
.../external/jdbc/JdbcFunctionPushDownRule.java| 123 ++---
.../doris/planner/external/jdbc/JdbcScanNode.java | 60 +++---
.../doris/planner/external/odbc/OdbcScanNode.java | 15 ++-
.../jdbc/test_clickhouse_jdbc_catalog.out | Bin 2766 -> 2782 bytes
.../jdbc/test_mysql_jdbc_catalog.out | 46
.../jdbc/test_clickhouse_jdbc_catalog.groovy | 9 ++
.../jdbc/test_mysql_jdbc_catalog.groovy| 61 --
10 files changed, 264 insertions(+), 63 deletions(-)
diff --git a/docker/thirdparties/docker-compose/clickhouse/clickhouse.yaml.tpl
b/docker/thirdparties/docker-compose/clickhouse/clickhouse.yaml.tpl
index 3832680dea..6e34c459be 100644
--- a/docker/thirdparties/docker-compose/clickhouse/clickhouse.yaml.tpl
+++ b/docker/thirdparties/docker-compose/clickhouse/clickhouse.yaml.tpl
@@ -19,7 +19,7 @@ version: "2.1"
services:
doris--clickhouse:
-image: "clickhouse/clickhouse-server:latest"
+image: "clickhouse/clickhouse-server:23.3"
restart: always
environment:
CLICKHOUSE_PASSWORD: 123456
diff --git
a/docker/thirdparties/docker-compose/clickhouse/init/03-create-table.sql
b/docker/thirdparties/docker-compose/clickhouse/init/03-create-table.sql
index c13fac5cf9..46d2e6ffc3 100644
--- a/docker/thirdparties/docker-compose/clickhouse/init/03-create-table.sql
+++ b/docker/thirdparties/docker-compose/clickhouse/init/03-create-table.sql
@@ -131,3 +131,11 @@ CREATE TABLE doris_test.final_test
)
ENGINE = ReplacingMergeTree
ORDER BY key;
+
+CREATE TABLE doris_test.ts
+(
+id Int64,
+ts UInt64
+)
+ENGINE = MergeTree
+ORDER BY id;
\ No newline at end of file
diff --git a/docker/thirdparties/docker-compose/clickhouse/init/04-insert.sql
b/docker/thirdparties/docker-compose/clickhouse/init/04-insert.sql
index 69c2ebd25f..c5e83eefe0 100644
--- a/docker/thirdparties/docker-compose/clickhouse/init/04-insert.sql
+++ b/docker/thirdparties/docker-compose/clickhouse/init/04-insert.sql
@@ -39,3 +39,6 @@ INSERT INTO doris_test.json VALUES ('1','{"a": 1, "b": { "c":
2, "d": [1, 2, 3]
INSERT INTO doris_test.final_test Values (1, 'first');
INSERT INTO doris_test.final_test Values (1, 'second');
+INSERT INTO doris_test.ts values (1,1694438743);
+
+
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
index bac9d62dbe..d1a44cb700 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
@@ -17,33 +17,130 @@
package org.apache.doris.planner.external.jdbc;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.FunctionCallExpr;
+import org.apache.doris.analysis.FunctionName;
import org.apache.doris.thrift.TOdbcTableType;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Maps;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Map;
import java.util.TreeSet;
+import java.util.function.Predicate;
public class JdbcFunctionPushDownRule {
-private static final TreeSet UNSUPPORTED_MYSQL_FUNCTIONS = new
TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+private static final Logger LOG =
LogManager.getLogger(JdbcFunctionPushDownRule.class);
+private static final TreeSet MYSQL_UNSUPPORTED_FUNCTIONS = new
TreeSet<>(String.CASE_INSENSITIVE_ORDER);
+
+static {
+MYSQL_UNSUPPORTED_FUNCTIONS.add("date_trunc");
+MYSQL_UNSUPPORTED_FUNCTIONS.add("money_format");
+}
+
+private static final Tr
[doris] branch master updated (da9eb79ac4 -> 1fd637eab7)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from da9eb79ac4 [Enhancement](Schema hash) Remove schema hash in tablet info (#23516) add 1fd637eab7 [Docs](Job)Add Job and Job tasks docs (#23576) No new revisions were added by this update. Summary of changes: .../Data-Definition-Statements/Alter/PAUSE-JOB.md | 53 + .../Data-Definition-Statements/Alter/RESUME-JOB.md | 54 + .../Data-Definition-Statements/Alter/STOP-JOB.md | 54 + .../Create/CREATE-JOB.md | 129 .../sql-reference/Show-Statements/SHOW-JOB-TASK.md | 73 .../sql-reference/Show-Statements/SHOW-JOB.md | 94 +++ docs/sidebars.json | 8 +- .../Data-Definition-Statements/Alter/PAUSE-JOB.md | 54 + .../Data-Definition-Statements/Alter/RESUME-JOB.md | 54 + .../Data-Definition-Statements/Alter/STOP-JOB.md | 54 + .../Create/CREATE-JOB.md | 130 + .../sql-reference/Show-Statements/SHOW-JOB-TASK.md | 74 .../sql-reference/Show-Statements/SHOW-JOB.md | 94 +++ 13 files changed, 924 insertions(+), 1 deletion(-) create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md create mode 100644 docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md create mode 100644 docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (d8f159728b -> 5b641ebd40)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from d8f159728b [fix](planner) only forbid substitute literal expr in
function call expr (#23532)
add 5b641ebd40 [feature-wip](catalog) support deltalake catalog
step1-metadata (#22493)
No new revisions were added by this update.
Summary of changes:
fe/fe-core/pom.xml | 6 +
.../java/org/apache/doris/catalog/TableIf.java | 5 +-
...atabase.java => DeltaLakeExternalDataBase.java} | 12 +-
.../catalog/external/DeltaLakeExternalTable.java | 159 +
.../catalog/external/HMSExternalDatabase.java | 4 +
.../doris/catalog/external/HMSExternalTable.java | 14 +-
.../apache/doris/datasource/CatalogFactory.java| 4 +
.../apache/doris/datasource/ExternalCatalog.java | 3 +
.../doris/datasource/HMSExternalCatalog.java | 7 +
.../apache/doris/datasource/InitCatalogLog.java| 1 +
.../apache/doris/datasource/InitDatabaseLog.java | 1 +
.../deltalake/DeltaLakeExternalCatalog.java| 69 +
.../org/apache/doris/persist/gson/GsonUtils.java | 12 +-
.../apache/doris/planner/SingleNodePlanner.java| 4 +-
fe/pom.xml | 1 +
15 files changed, 285 insertions(+), 17 deletions(-)
copy
fe/fe-core/src/main/java/org/apache/doris/catalog/external/{TestExternalDatabase.java
=> DeltaLakeExternalDataBase.java} (65%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/catalog/external/DeltaLakeExternalTable.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/deltalake/DeltaLakeExternalCatalog.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](odbc) fix odbc insert string data to sqlserve (#23364)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 030df6db35 [fix](odbc) fix odbc insert string data to sqlserve (#23364)
030df6db35 is described below
commit 030df6db35ffad95ada0c8d1014fa1c77374a5b5
Author: zy-kkk
AuthorDate: Tue Aug 29 21:47:50 2023 +0800
[fix](odbc) fix odbc insert string data to sqlserve (#23364)
---
be/src/exec/table_connector.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/exec/table_connector.cpp b/be/src/exec/table_connector.cpp
index 94c7d889be..7aa90eda08 100644
--- a/be/src/exec/table_connector.cpp
+++ b/be/src/exec/table_connector.cpp
@@ -205,7 +205,7 @@ Status TableConnector::convert_column_data(const
vectorized::ColumnPtr& column_p
// for oracle/pg database string must be '
if (table_type == TOdbcTableType::ORACLE || table_type ==
TOdbcTableType::POSTGRESQL ||
table_type == TOdbcTableType::SAP_HANA || table_type ==
TOdbcTableType::MYSQL ||
-table_type == TOdbcTableType::CLICKHOUSE) {
+table_type == TOdbcTableType::CLICKHOUSE || table_type ==
TOdbcTableType::SQLSERVER) {
fmt::format_to(_insert_stmt_buffer, "'{}'",
fmt::basic_string_view(item, size));
} else {
fmt::format_to(_insert_stmt_buffer, "\"{}\"",
fmt::basic_string_view(item, size));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [refactor](TableRowCountAction) Fine-tune sql execution code (#23541)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f786689044 [refactor](TableRowCountAction) Fine-tune sql execution
code (#23541)
f786689044 is described below
commit f786689044882083f1773c4731ebb2f3dd43d071
Author: Guangdong Liu
AuthorDate: Wed Aug 30 10:11:30 2023 +0800
[refactor](TableRowCountAction) Fine-tune sql execution code (#23541)
---
.../org/apache/doris/httpv2/rest/StmtExecutionAction.java | 7 ++-
.../java/org/apache/doris/httpv2/util/StatementSubmitter.java | 11 +--
2 files changed, 7 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/StmtExecutionAction.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/StmtExecutionAction.java
index 7bfc0eb418..d9e07eb9b3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/StmtExecutionAction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/StmtExecutionAction.java
@@ -113,7 +113,7 @@ public class StmtExecutionAction extends RestBaseController
{
stmtRequestBody.limit);
ConnectContext.get().changeDefaultCatalog(ns);
-ConnectContext.get().setDatabase(getFullDbName(dbName));
+ConnectContext.get().setDatabase(fullDbName);
String streamHeader = request.getHeader("X-Doris-Stream");
boolean isStream = !("false".equalsIgnoreCase(streamHeader));
@@ -176,10 +176,7 @@ public class StmtExecutionAction extends
RestBaseController {
return null;
}
return ResponseEntityBuilder.ok(resultSet.getResult());
-} catch (InterruptedException e) {
-LOG.warn("failed to execute stmt", e);
-return ResponseEntityBuilder.okWithCommonError("Failed to
execute sql: " + e.getMessage());
-} catch (ExecutionException e) {
+} catch (InterruptedException | ExecutionException e) {
LOG.warn("failed to execute stmt", e);
return ResponseEntityBuilder.okWithCommonError("Failed to
execute sql: " + e.getMessage());
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/util/StatementSubmitter.java
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/util/StatementSubmitter.java
index 5817f1d252..83c5e075f2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/httpv2/util/StatementSubmitter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/httpv2/util/StatementSubmitter.java
@@ -73,7 +73,7 @@ public class StatementSubmitter {
private static final String JDBC_DRIVER = "org.mariadb.jdbc.Driver";
private static final String DB_URL_PATTERN =
"jdbc:mariadb://127.0.0.1:%d/%s";
-private ThreadPoolExecutor executor =
ThreadPoolManager.newDaemonCacheThreadPool(2, "SQL submitter", true);
+private final ThreadPoolExecutor executor =
ThreadPoolManager.newDaemonCacheThreadPool(2, "SQL submitter", true);
public Future submit(StmtContext queryCtx) {
Worker worker = new Worker(ConnectContext.get(), queryCtx);
@@ -81,8 +81,8 @@ public class StatementSubmitter {
}
private static class Worker implements Callable {
-private ConnectContext ctx;
-private StmtContext queryCtx;
+private final ConnectContext ctx;
+private final StmtContext queryCtx;
public Worker(ConnectContext ctx, StmtContext queryCtx) {
this.ctx = ctx;
@@ -104,7 +104,7 @@ public class StatementSubmitter {
stmt = conn.prepareStatement(
queryCtx.stmt, ResultSet.TYPE_FORWARD_ONLY,
ResultSet.CONCUR_READ_ONLY);
// set fetch size to 1 to enable streaming result set to
avoid OOM.
-((PreparedStatement) stmt).setFetchSize(1000);
+stmt.setFetchSize(1000);
ResultSet rs = ((PreparedStatement) stmt).executeQuery();
if (queryCtx.isStream) {
StreamResponseInf streamResponse = new
JsonStreamResponse(queryCtx.response);
@@ -123,8 +123,7 @@ public class StatementSubmitter {
streamResponse.handleDdlAndExport(startTime);
return new ExecutionResultSet(null);
}
-ExecutionResultSet resultSet =
generateExecStatus(startTime);
-return resultSet;
+return generateExecStatus(startTime);
} else {
throw new Exception("Unsupported statement type");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](jdbc catalog) Remove useless mysql jdbc connection parameters (#23647)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cc6cbc04af [improvement](jdbc catalog) Remove useless mysql jdbc
connection parameters (#23647)
cc6cbc04af is described below
commit cc6cbc04af32832ada0361664eeaec9c5ca4fc3b
Author: zy-kkk
AuthorDate: Wed Aug 30 17:47:04 2023 +0800
[improvement](jdbc catalog) Remove useless mysql jdbc connection parameters
(#23647)
---
fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java | 6 ++
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index dcf144b615..e9fc898b2e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -300,11 +300,9 @@ public class JdbcResource extends Resource {
newJdbcUrl = checkAndSetJdbcBoolParam(newJdbcUrl,
"yearIsDateType", "true", "false");
// MySQL Types and Return Values for GetColumnTypeName and
GetColumnClassName
// are presented in
https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-type-conversions.html
-// However when tinyInt1isBit=false, GetColumnClassName of MySQL
returns java.lang.Boolean,
-// while that of Doris returns java.lang.Integer. In order to be
compatible with both MySQL and Doris,
-// Jdbc params should set
tinyInt1isBit=true&transformedBitIsBoolean=true
+// When mysql's tinyint stores non-0 or 1, we need to read the
data correctly,
+// so we need tinyInt1isBit=false
newJdbcUrl = checkAndSetJdbcBoolParam(newJdbcUrl, "tinyInt1isBit",
"true", "false");
-newJdbcUrl = checkAndSetJdbcBoolParam(newJdbcUrl,
"transformedBitIsBoolean", "false", "true");
// set useUnicode and characterEncoding to false and utf-8
newJdbcUrl = checkAndSetJdbcBoolParam(newJdbcUrl, "useUnicode",
"false", "true");
newJdbcUrl = checkAndSetJdbcBoolParam(newJdbcUrl,
"rewriteBatchedStatements", "false", "true");
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](catalog) avoid calling checksum when replaying creating jdbc catalog and fix ranger issue (#22369)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b7404896fa [improvement](catalog) avoid calling checksum when
replaying creating jdbc catalog and fix ranger issue (#22369)
b7404896fa is described below
commit b7404896fa8e22716288c0f956633a8f8d12ba90
Author: Mingyu Chen
AuthorDate: Wed Aug 30 19:24:11 2023 +0800
[improvement](catalog) avoid calling checksum when replaying creating jdbc
catalog and fix ranger issue (#22369)
1. jdbc
Before, in the constructor of Jdbc catalog, we may call checksum action of
the jdbc driver.
But the download link of the jdbc driver may not be available when
replaying, causing replay error.
This PR change the logic to avoid calling checksum when replaying creating
jdbc catalog.
2. ranger
After this PR, when creating catalog, it will try to init access controller
to make sure the config is ok.
3. catalog priv check
When creating/dropping/altering/ catalog, doris will only use internal
access controller to check catalog level priv.
---
.../org/apache/doris/datasource/CatalogFactory.java| 10 +-
.../org/apache/doris/datasource/ExternalCatalog.java | 8 +---
.../apache/doris/datasource/HMSExternalCatalog.java| 5 -
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 12
.../doris/mysql/privilege/AccessControllerManager.java | 18 --
.../hive/test_external_catalog_hive.groovy | 15 +++
6 files changed, 57 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
index 2293250d5c..07a03ed861 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java
@@ -146,7 +146,15 @@ public class CatalogFactory {
if (!isReplay) {
// set some default properties when creating catalog.
// do not call this method when replaying edit log. Because we
need to keey the original properties.
-catalog.setDefaultProps();
+catalog.setDefaultPropsWhenCreating(isReplay);
+// This will check if the customized access controller can be
created successfully.
+// If failed, it will throw exception and the catalog will not be
created.
+try {
+catalog.initAccessController(true);
+} catch (Exception e) {
+LOG.warn("Failed to init access controller", e);
+throw new DdlException("Failed to init access controller: " +
e.getMessage());
+}
}
return catalog;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index 89083a1b30..f213f9a302 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -110,7 +110,7 @@ public abstract class ExternalCatalog
+ "listDatabaseNames from remote client when init catalog with
" + logType.name());
}
-public void setDefaultProps() {
+public void setDefaultPropsWhenCreating(boolean isReplay) throws
DdlException {
// set some default properties when creating catalog
}
@@ -203,8 +203,10 @@ public abstract class ExternalCatalog
* "access_controller.properties.prop1" = "xxx",
* "access_controller.properties.prop2" = "yyy",
* )
+ *
+ * isDryRun: if true, it will try to create the custom access controller,
but will not add it to the access manager.
*/
-public void initAccessController() {
+public void initAccessController(boolean isDryRun) {
Map properties = getCatalogProperty().getProperties();
// 1. get access controller class name
String className =
properties.getOrDefault(CatalogMgr.ACCESS_CONTROLLER_CLASS_PROP, "");
@@ -224,7 +226,7 @@ public abstract class ExternalCatalog
}
// 3. create access controller
-Env.getCurrentEnv().getAccessManager().createAccessController(name,
className, acProperties);
+Env.getCurrentEnv().getAccessManager().createAccessController(name,
className, acProperties, isDryRun);
}
// init schema related objects
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
index 8a076fabec..8bdea35e45 100644
---
a/fe/fe-
[doris-flink-connector] branch master updated: [mysql] add mysql chunk column options (#188)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new f8ea59b [mysql] add mysql chunk column options (#188)
f8ea59b is described below
commit f8ea59ba79419018dd495bb107410718d708b61d
Author: wudi <676366...@qq.com>
AuthorDate: Thu Aug 31 10:14:36 2023 +0800
[mysql] add mysql chunk column options (#188)
1、add mysql column key column:
`--mysql-conf scan.incremental.snapshot.chunk.key-column =
db.tbl1:col1,db.tbl2:col2`
mysqlcdc version must be greater than 2.4
2、fix sqlserver type
-
Co-authored-by: wudi <>
---
.../apache/doris/flink/tools/cdc/DatabaseSync.java | 3 +-
.../flink/tools/cdc/mysql/MysqlDatabaseSync.java | 51 +-
.../flink/tools/cdc/sqlserver/SqlServerType.java | 7 ++-
3 files changed, 45 insertions(+), 16 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
index 29c554e..bc15987 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java
@@ -94,6 +94,7 @@ public abstract class DatabaseSync {
DorisSystem dorisSystem = new DorisSystem(options);
List schemaList = getSchemaList();
+Preconditions.checkState(!schemaList.isEmpty(), "No tables to be
synchronized.");
if (!dorisSystem.databaseExists(database)) {
LOG.info("database {} not exist, created", database);
dorisSystem.createDatabase(database);
@@ -118,9 +119,7 @@ public abstract class DatabaseSync {
System.exit(0);
}
-Preconditions.checkState(!syncTables.isEmpty(), "No tables to be
synchronized.");
config.setString(TABLE_NAME_OPTIONS, "(" + String.join("|",
syncTables) + ")");
-
DataStreamSource streamSource = buildCdcSource(env);
SingleOutputStreamOperator parsedStream =
streamSource.process(new ParsingProcessFunction(converter));
for (String table : dorisTables) {
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java
index 735fdb9..05dd298 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/mysql/MysqlDatabaseSync.java
@@ -26,7 +26,7 @@ import
com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.json.JsonCon
import com.ververica.cdc.debezium.DebeziumDeserializationSchema;
import com.ververica.cdc.debezium.JsonDebeziumDeserializationSchema;
import com.ververica.cdc.debezium.table.DebeziumOptions;
-
+import org.apache.doris.flink.catalog.doris.DataModel;
import
org.apache.doris.flink.deserialization.DorisJsonDebeziumDeserializationSchema;
import org.apache.doris.flink.tools.cdc.DatabaseSync;
import org.apache.doris.flink.tools.cdc.DateToStringConverter;
@@ -34,7 +34,9 @@ import org.apache.doris.flink.tools.cdc.SourceSchema;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.catalog.ObjectPath;
import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -48,6 +50,8 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
public class MysqlDatabaseSync extends DatabaseSync {
private static final Logger LOG =
LoggerFactory.getLogger(MysqlDatabaseSync.class);
@@ -83,13 +87,8 @@ public class MysqlDatabaseSync extends DatabaseSync {
}
SourceSchema sourceSchema =
new MysqlSchema(metaData, databaseName, tableName,
tableComment);
-if (sourceSchema.primaryKeys.size() > 0) {
-//Only sync tables with primary keys
-schemaList.add(sourceSchema);
-} else {
-LOG.warn("table {} has no primary key, skip",
tableName);
-System.out.println("table " + tableName + " has no
primary key, skip.");
-
[doris] branch master updated: [fix](jdbc catalog) fix jdbc driver cache load error (#23656)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3e4ee3c1e6 [fix](jdbc catalog) fix jdbc driver cache load error
(#23656)
3e4ee3c1e6 is described below
commit 3e4ee3c1e67a7cfb99f0c16ab03fa6077328e363
Author: zy-kkk
AuthorDate: Thu Aug 31 10:17:15 2023 +0800
[fix](jdbc catalog) fix jdbc driver cache load error (#23656)
log error:
`W20230830 11:19:47.495721 3046231 status.h:363] meet error status:
[INTERNAL_ERROR]user function's name should be
function_id.checksum[.file_name].file_type, now the all split parts are by
delimiter(.):
7119053928154065546.20c8228267b6c9ce620fddb39467d3eb.postgresql-42.5.0.jar`
When the jdbc driver had `.` in its name we failed to split it properly
---
be/src/runtime/user_function_cache.cpp | 25 -
be/src/runtime/user_function_cache.h | 1 +
be/test/runtime/user_function_cache_test.cpp | 28
3 files changed, 53 insertions(+), 1 deletion(-)
diff --git a/be/src/runtime/user_function_cache.cpp
b/be/src/runtime/user_function_cache.cpp
index 9edc4dbd46..39507ec222 100644
--- a/be/src/runtime/user_function_cache.cpp
+++ b/be/src/runtime/user_function_cache.cpp
@@ -151,7 +151,7 @@ Status UserFunctionCache::_load_entry_from_lib(const
std::string& dir, const std
file);
}
-std::vector split_parts = strings::Split(file, ".");
+std::vector split_parts = _split_string_by_checksum(file);
if (split_parts.size() != 3 && split_parts.size() != 4) {
return Status::InternalError(
"user function's name should be
function_id.checksum[.file_name].file_type, now "
@@ -379,4 +379,27 @@ Status UserFunctionCache::get_jarpath(int64_t fid, const
std::string& url,
return Status::OK();
}
+std::vector UserFunctionCache::_split_string_by_checksum(const
std::string& file) {
+std::vector result;
+
+// Find the first dot from the start
+size_t firstDot = file.find('.');
+if (firstDot == std::string::npos) return {};
+
+// Find the second dot starting from the first dot's position
+size_t secondDot = file.find('.', firstDot + 1);
+if (secondDot == std::string::npos) return {};
+
+// Find the last dot from the end
+size_t lastDot = file.rfind('.');
+if (lastDot == std::string::npos || lastDot <= secondDot) return {};
+
+// Split based on these dots
+result.push_back(file.substr(0, firstDot));
+result.push_back(file.substr(firstDot + 1, secondDot - firstDot - 1));
+result.push_back(file.substr(secondDot + 1, lastDot - secondDot - 1));
+result.push_back(file.substr(lastDot + 1));
+
+return result;
+}
} // namespace doris
diff --git a/be/src/runtime/user_function_cache.h
b/be/src/runtime/user_function_cache.h
index e58f02294b..5d1bff8b86 100644
--- a/be/src/runtime/user_function_cache.h
+++ b/be/src/runtime/user_function_cache.h
@@ -74,6 +74,7 @@ private:
std::string _get_real_url(const std::string& url);
std::string _get_file_name_from_url(const std::string& url) const;
+std::vector _split_string_by_checksum(const std::string&
file);
private:
std::string _lib_dir;
diff --git a/be/test/runtime/user_function_cache_test.cpp
b/be/test/runtime/user_function_cache_test.cpp
index be1705ba9a..b9a4694982 100644
--- a/be/test/runtime/user_function_cache_test.cpp
+++ b/be/test/runtime/user_function_cache_test.cpp
@@ -16,3 +16,31 @@
// under the License.
#include "runtime/user_function_cache.h"
+
+#include
+#include
+
+#include
+
+#include "gtest/gtest.h"
+
+namespace doris {
+
+class UserFunctionCacheTest : public ::testing::Test {
+protected:
+UserFunctionCache ufc;
+};
+
+TEST_F(UserFunctionCacheTest, SplitStringByChecksumTest) {
+// Test valid string format
+std::string valid_str =
+
"7119053928154065546.20c8228267b6c9ce620fddb39467d3eb.postgresql-42.5.0.jar";
+auto result = ufc._split_string_by_checksum(valid_str);
+ASSERT_EQ(result.size(), 4);
+EXPECT_EQ(result[0], "7119053928154065546");
+EXPECT_EQ(result[1], "20c8228267b6c9ce620fddb39467d3eb");
+EXPECT_EQ(result[2], "postgresql-42.5.0");
+EXPECT_EQ(result[3], "jar");
+}
+
+} // namespace doris
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Doc](flink connector) Add new configuration be nodes (#23698)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 6b4d1c2d86 [Doc](flink connector) Add new configuration be nodes (#23698) 6b4d1c2d86 is described below commit 6b4d1c2d8636d647bec7abf022310fcabde1c5fc Author: DongLiang-0 <46414265+donglian...@users.noreply.github.com> AuthorDate: Fri Sep 1 00:16:08 2023 +0800 [Doc](flink connector) Add new configuration be nodes (#23698) --- docs/en/docs/ecosystem/flink-doris-connector.md| 19 ++- docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 19 ++- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/en/docs/ecosystem/flink-doris-connector.md b/docs/en/docs/ecosystem/flink-doris-connector.md index f2dcb85743..dbb0e9d99d 100644 --- a/docs/en/docs/ecosystem/flink-doris-connector.md +++ b/docs/en/docs/ecosystem/flink-doris-connector.md @@ -301,15 +301,16 @@ ON a.city = c.city ### General configuration items -| Key | Default Value | Required | Comment | -| | - | | | -| fenodes | --| Y| Doris FE http address, multiple addresses are supported, separated by commas | -| table.identifier | --| Y| Doris table name, such as: db.tbl| -| username | --| Y| username to access Doris | -| password | --| Y| Password to access Doris | -| doris.request.retries| 3 | N| Number of retries to send requests to Doris | -| doris.request.connect.timeout.ms | 3 | N| Connection timeout for sending requests to Doris | -| doris.request.read.timeout.ms| 3 | N| Read timeout for sending requests to Doris | +| Key | Default Value | Required | Comment | +|--|---|--|-| +| fenodes | --| Y| Doris FE http address, multiple addresses are supported, separated by commas | +| benodes | --| N| Doris BE http address, multiple addresses are supported, separated by commas. refer to [#187](https://github.com/apache/doris-flink-connector/pull/187) | +| table.identifier | --| Y| Doris table name, such as: db.tbl | +| username | --| Y| username to access Doris | +| password | --| Y| Password to access Doris | +| doris.request.retries| 3 | N| Number of retries to send requests to Doris | +| doris.request.connect.timeout.ms | 3 | N| Connection timeout for sending requests to Doris | +| doris.request.read.timeout.ms| 3 | N| Read timeout for sending requests to Doris | ### Source configuration item diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md index 3b7c929b4e..a170f3b02e 100644 --- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md +++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md @@ -303,15 +303,16 @@ ON a.city = c.city ### 通用配置项 -| Key | Default Value | Required | C
[doris] branch master updated: [Docs](alter partition) Fix the docs of adding default partition (#23705)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new d0e906f329 [Docs](alter partition) Fix the docs of adding default partition (#23705) d0e906f329 is described below commit d0e906f329578b5122b45e7780210fa227af1f6c Author: bobhan1 AuthorDate: Fri Sep 1 00:20:12 2023 +0800 [Docs](alter partition) Fix the docs of adding default partition (#23705) according to https://github.com/apache/doris/pull/15509, add a default list partition don't need the keyword `DEFAULT` --- .../Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md | 4 ++-- .../Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md index d1774fcf47..03aa6968ea 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md @@ -64,8 +64,8 @@ Notice: - If the bucketing method is specified, only the number of buckets can be modified, not the bucketing method or the bucketing column. If the bucketing method is specified but the number of buckets not be specified, the default value `10` will be used for bucket number instead of the number specified when the table is created. If the number of buckets modified, the bucketing method needs to be specified simultaneously. - The ["key"="value"] section can set some attributes of the partition, see [CREATE TABLE](../Create/CREATE-TABLE.md) - If the user does not explicitly create a partition when creating a table, adding a partition by ALTER is not supported -- If the user uses list partition then they can add default partition to the table - - ALTER TABLE ADD PARTITION DEFAULT +- If the user uses list partition then they can add default partition to the table. The default partition will store all data not satisfying prior partition key's constraints. + - ALTER TABLE table_name ADD PARTITION partition_name 2. Delete the partition diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md index 43987d00e5..c9ea4a4c7d 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/ALTER-TABLE-PARTITION.md @@ -64,8 +64,8 @@ partition_desc ["key"="value"] - 如指定分桶方式，只能修改分桶数，不可修改分桶方式或分桶列。如果指定了分桶方式，但是没有指定分桶数，则分桶数会使用默认值10，不会使用建表时指定的分桶数。如果要指定分桶数，则必须指定分桶方式。 - ["key"="value"] 部分可以设置分区的一些属性，具体说明见 [CREATE TABLE](../Create/CREATE-TABLE.md) - 如果建表时用户未显式创建Partition,则不支持通过ALTER的方式增加分区 -- 如果用户使用的是List Partition则可以增加default partition - - ALTER TABLE ADD PARTITION DEFAULT +- 如果用户使用的是List Partition则可以增加default partition，default partition将会存储所有不满足其他分区键要求的数据。 + - ALTER TABLE table_name ADD PARTITION partition_name 2. 删除分区 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Improve](Job)Job internal interface provides immediate scheduling (#23735)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e88c218390 [Improve](Job)Job internal interface provides immediate
scheduling (#23735)
e88c218390 is described below
commit e88c218390cede307b593044be2f0dfebc9bd0d3
Author: Calvin Kirs
AuthorDate: Fri Sep 1 12:50:08 2023 +0800
[Improve](Job)Job internal interface provides immediate scheduling (#23735)
Delete meaningless job status
System scheduling is executed in the time wheel
Optimize window calculation code
---
.../sql-reference/Show-Statements/SHOW-JOB.md | 1 -
.../sql-reference/Show-Statements/SHOW-JOB.md | 1 -
.../doris/scheduler/constants/JobStatus.java | 2 --
.../doris/scheduler/constants/SystemJob.java | 42 --
.../doris/scheduler/disruptor/TaskHandler.java | 35 +++---
.../java/org/apache/doris/scheduler/job/Job.java | 29 +++
.../doris/scheduler/manager/TimerJobManager.java | 33 +++--
.../scheduler/disruptor/TimerJobManagerTest.java | 12 +++
8 files changed, 51 insertions(+), 104 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
index 60c1dfd1cd..ec11fd3168 100644
--- a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
+++ b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
@@ -71,7 +71,6 @@ Result description:
* PAUSED: Paused
* STOPPED: end (manually triggered by the user)
* FINISHED: Finished
- * WAITING_FINISH: pending completion
### Example
diff --git
a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
index 009e2cb4e4..1b5da59908 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md
@@ -71,7 +71,6 @@ SHOW JOBS 用于展示当前 DB 下所有作业的运行状态，SHOW JOB FOR jo
* PAUSED：暂停
* STOPPED：结束（用户手动触发）
* FINISHED: 完成
-* WAITING_FINISH: 待结束
### Example
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/JobStatus.java
b/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/JobStatus.java
index ae204ef948..f01686a521 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/JobStatus.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/JobStatus.java
@@ -37,8 +37,6 @@ public enum JobStatus {
*/
STOPPED,
-WAITING_FINISH,
-
/**
* When the task is finished, the finished state will be triggered.
*/
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/SystemJob.java
b/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/SystemJob.java
deleted file mode 100644
index 3428c1724f..00
---
a/fe/fe-core/src/main/java/org/apache/doris/scheduler/constants/SystemJob.java
+++ /dev/null
@@ -1,42 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.scheduler.constants;
-
-import lombok.Getter;
-
-/**
- * System scheduler event job
- * They will start when scheduler starts,don't use this job in other place,it
just for system inner scheduler
- */
-public enum SystemJob {
-
-/**
- * System cycle scheduler event job, it will start cycle scheduler
- */
-SYSTEM_SCHEDULER_JOB("system_scheduler_event_job", 1L);
-
-@Getter
-private final String description;
-@Getter
-private final Long id;
-
-SystemJob(String description, Long id) {
-this.description = description;
-this.id = id;
-}
-}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/scheduler/disruptor/TaskHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/scheduler/disruptor/TaskHandler.java
index 297537f712..0b309ec3ce 100644
---
a/fe/fe-core/src/main/java/o
[doris] branch master updated (a8de805a7a -> 0d50c11d5c)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a8de805a7a [fix](Nereids) fix stats inject in or_expansion.groovy (#23748) add 0d50c11d5c [Doc](AuditLoader) improvement auditLoader doc (#23758) No new revisions were added by this update. Summary of changes: docs/en/docs/ecosystem/audit-plugin.md| 75 +-- docs/zh-CN/docs/ecosystem/audit-plugin.md | 63 -- 2 files changed, 101 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Feature](Multi-Catalog) support query doris bitmap column in external jdbc catalog (#23021)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 228f0ac5bb [Feature](Multi-Catalog) support query doris bitmap column
in external jdbc catalog (#23021)
228f0ac5bb is described below
commit 228f0ac5bb1c91464eb70cb6e365dac7e1c639a1
Author: GoGoWen <82132356+gogo...@users.noreply.github.com>
AuthorDate: Sat Sep 2 12:46:33 2023 +0800
[Feature](Multi-Catalog) support query doris bitmap column in external jdbc
catalog (#23021)
---
be/src/runtime/descriptors.h | 9 ++
be/src/util/bitmap_value.h | 98
be/src/vec/data_types/data_type_bitmap.cpp | 12 ++
be/src/vec/data_types/data_type_bitmap.h | 3 +-
be/src/vec/exec/vjdbc_connector.cpp| 82 +-
be/src/vec/exec/vjdbc_connector.h | 7 ++
be/src/vec/functions/function_cast.h | 24
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 1 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 3 +-
.../java/org/apache/doris/jdbc/JdbcExecutor.java | 50 +
.../datasource/jdbc/client/JdbcMySQLClient.java| 4 +
.../jdbc/test_doris_jdbc_catalog_query_bitmap.out | 51 +
.../test_doris_jdbc_catalog_query_bitmap.groovy| 124 +
13 files changed, 442 insertions(+), 26 deletions(-)
diff --git a/be/src/runtime/descriptors.h b/be/src/runtime/descriptors.h
index 2d9ecf9a32..af59b287c4 100644
--- a/be/src/runtime/descriptors.h
+++ b/be/src/runtime/descriptors.h
@@ -367,6 +367,15 @@ public:
return false;
}
+bool has_bitmap_slot() const {
+for (auto slot : _slots) {
+if (slot->type().is_bitmap_type()) {
+return true;
+}
+}
+return false;
+}
+
TupleId id() const { return _id; }
std::string debug_string() const;
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index 02a5595440..e67e665155 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1176,48 +1176,102 @@ public:
BitmapValue(const BitmapValue& other) {
_type = other._type;
-_sv = other._sv;
-_bitmap = other._bitmap;
-_set = other._set;
-_is_shared = true;
-// should also set other's state to shared, so that other bitmap value
will
-// create a new bitmap when it wants to modify it.
-const_cast(other)._is_shared = true;
+switch (other._type) {
+case EMPTY:
+break;
+case SINGLE:
+_sv = other._sv;
+break;
+case BITMAP:
+_bitmap = other._bitmap;
+break;
+case SET:
+_set = other._set;
+break;
+}
+
+if (other._type != EMPTY) {
+_is_shared = true;
+// should also set other's state to shared, so that other bitmap
value will
+// create a new bitmap when it wants to modify it.
+const_cast(other)._is_shared = true;
+}
}
BitmapValue(BitmapValue&& other) {
_type = other._type;
-_sv = other._sv;
+switch (other._type) {
+case EMPTY:
+break;
+case SINGLE:
+_sv = other._sv;
+break;
+case BITMAP:
+_bitmap = std::move(other._bitmap);
+other._bitmap = nullptr;
+break;
+case SET:
+_set = std::move(other._set);
+break;
+}
_is_shared = other._is_shared;
-_bitmap = std::move(other._bitmap);
-_set = std::move(other._set);
-
other._type = EMPTY;
other._is_shared = false;
-other._bitmap = nullptr;
}
BitmapValue& operator=(const BitmapValue& other) {
_type = other._type;
-_sv = other._sv;
-_bitmap = other._bitmap;
-_is_shared = true;
-_set = other._set;
-// should also set other's state to shared, so that other bitmap value
will
-// create a new bitmap when it wants to modify it.
-const_cast(other)._is_shared = true;
+switch (other._type) {
+case EMPTY:
+break;
+case SINGLE:
+_sv = other._sv;
+break;
+case BITMAP:
+_bitmap = other._bitmap;
+break;
+case SET:
+_set = other._set;
+break;
+}
+
+if (other._type != EMPTY) {
+_is_shared = true;
+// should also set other's state to shared, so that other bitmap
value will
+// create a new bitmap when it wants to modify it.
+const_cast(other)._is_shared = true;
+
[doris] branch master updated: [typo](docs) modify standard-deployment doc (#23473)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new a1400677b6 [typo](docs) modify standard-deployment doc (#23473) a1400677b6 is described below commit a1400677b6e0747e7b9a7b58f11120be50671da5 Author: wyx123654 <45102878+wyx123...@users.noreply.github.com> AuthorDate: Sat Sep 2 21:24:55 2023 +0800 [typo](docs) modify standard-deployment doc (#23473) --- docs/en/docs/install/standard-deployment.md| 2 +- docs/zh-CN/docs/install/standard-deployment.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/install/standard-deployment.md b/docs/en/docs/install/standard-deployment.md index 33d277a16b..e546d64c58 100644 --- a/docs/en/docs/install/standard-deployment.md +++ b/docs/en/docs/install/standard-deployment.md @@ -177,7 +177,7 @@ See the `lower_case_table_names` section in [Variables](../advanced/variables.md **Note: For production environments, it is better not to put the directory under the Doris installation directory but in a separate disk (SSD would be the best); for test and development environments, you may use the default configuration.** - 2. The default maximum Java heap memory of JAVA_OPTS in fe.conf is 4GB. For production environments, we recommend that it be adjusted to more than 8G. + 2. The default maximum Java heap memory of JAVA_OPTS in fe.conf is 8GB. * Start FE diff --git a/docs/zh-CN/docs/install/standard-deployment.md b/docs/zh-CN/docs/install/standard-deployment.md index cc1b34f73a..a338ab5f35 100644 --- a/docs/zh-CN/docs/install/standard-deployment.md +++ b/docs/zh-CN/docs/install/standard-deployment.md @@ -172,7 +172,7 @@ doris默认为表名大小写敏感，如有表名大小写不敏感的需求需 **注意：生产环境强烈建议单独指定目录不要放在Doris安装目录下，最好是单独的磁盘（如果有SSD最好），测试开发环境可以使用默认配置** -2. fe.conf 中 JAVA_OPTS 默认 java 最大堆内存为 4GB，**建议生产环境调整至 8G 以上**。 +2. fe.conf 中 JAVA_OPTS 默认 java 最大堆内存为 8GB。 * 启动FE - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (c1620f9e1a -> 2a3fc92d13)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from c1620f9e1a [chore](case) Update regression-conf.groovy (#23873) add 2a3fc92d13 [fix](auth)fix after setting the user password to expire, changing the password again will not take effect (#23426) No new revisions were added by this update. Summary of changes: .../doris/mysql/privilege/PasswordPolicy.java | 8 + .../mysql/privilege/PasswordPolicyManager.java | 7 + .../suites/account_p0/test_alter_user.groovy | 34 +- 3 files changed, 48 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](auth)Fix create user no permissions of information_schema database (#23898)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e40a4d4ae3 [fix](auth)Fix create user no permissions of
information_schema database (#23898)
e40a4d4ae3 is described below
commit e40a4d4ae3a01f21575994d573bf3c1e6716ed00
Author: DongLiang-0 <46414265+donglian...@users.noreply.github.com>
AuthorDate: Thu Sep 7 12:55:52 2023 +0800
[fix](auth)Fix create user no permissions of information_schema database
(#23898)
When creating a new user, this user does not have the information_schema
database permission.
---
.../java/org/apache/doris/mysql/privilege/Role.java | 17 -
.../org/apache/doris/mysql/privilege/RoleManager.java | 15 +--
.../suites/account_p0/test_alter_user.groovy| 10 ++
3 files changed, 31 insertions(+), 11 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
index 025b86fa63..7f1f8bb8ef 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
@@ -45,6 +45,7 @@ import org.apache.logging.log4j.Logger;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
+import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Objects;
@@ -138,17 +139,23 @@ public class Role implements Writable,
GsonPostProcessable {
grantPrivs(workloadGroupPattern, privs.copy());
}
-public Role(String roleName, TablePattern tablePattern, PrivBitSet
tablePrivs,
+public Role(String roleName, List tablePatterns, PrivBitSet
tablePrivs,
WorkloadGroupPattern workloadGroupPattern, PrivBitSet
workloadGroupPrivs) {
this.roleName = roleName;
-this.tblPatternToPrivs.put(tablePattern, tablePrivs);
this.workloadGroupPatternToPrivs.put(workloadGroupPattern,
workloadGroupPrivs);
-//for init admin role,will not generate exception
+tablePatterns.forEach(tablePattern -> {
+// for init admin role,will not generate exception
+try {
+this.tblPatternToPrivs.put(tablePattern, tablePrivs);
+grantPrivs(tablePattern, tablePrivs.copy());
+} catch (DdlException e) {
+LOG.warn("grant table failed,", e);
+}
+});
try {
-grantPrivs(tablePattern, tablePrivs.copy());
grantPrivs(workloadGroupPattern, workloadGroupPrivs.copy());
} catch (DdlException e) {
-LOG.warn("grant failed,", e);
+LOG.warn("grant workload group failed,", e);
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RoleManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RoleManager.java
index 7b37b6f40e..7df0baf495 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RoleManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RoleManager.java
@@ -192,17 +192,20 @@ public class RoleManager implements Writable {
if (roles.containsKey(userDefaultRoleName)) {
return roles.get(userDefaultRoleName);
}
+
// grant read privs to database information_schema & mysql
-TablePattern tblPattern = new TablePattern(Auth.DEFAULT_CATALOG,
InfoSchemaDb.DATABASE_NAME, "*");
+List tablePatterns = Lists.newArrayList();
+TablePattern informationTblPattern = new
TablePattern(Auth.DEFAULT_CATALOG, InfoSchemaDb.DATABASE_NAME, "*");
try {
-tblPattern.analyze(SystemInfoService.DEFAULT_CLUSTER);
+informationTblPattern.analyze(SystemInfoService.DEFAULT_CLUSTER);
+tablePatterns.add(informationTblPattern);
} catch (AnalysisException e) {
LOG.warn("should not happen", e);
}
-
-tblPattern = new TablePattern(Auth.DEFAULT_CATALOG,
MysqlDb.DATABASE_NAME, "*");
+TablePattern mysqlTblPattern = new TablePattern(Auth.DEFAULT_CATALOG,
MysqlDb.DATABASE_NAME, "*");
try {
-tblPattern.analyze(SystemInfoService.DEFAULT_CLUSTER);
+mysqlTblPattern.analyze(SystemInfoService.DEFAULT_CLUSTER);
+tablePatterns.add(mysqlTblPattern);
} catch (AnalysisException e) {
LOG.warn("should not happen", e);
}
@@ -214,7 +217,7 @@ public class RoleManager implements Writable {
} catch (AnalysisException e) {
LOG.warn("should not happen", e);
}
-Role role = new Role(userDefaultRoleName, tb
(doris-flink-connector) branch master updated: add multisink to DorisBatchSink (#223)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 32fd54e add multisink to DorisBatchSink (#223)
32fd54e is described below
commit 32fd54e3c1595897d02a188b00badebd0b9ef4e2
Author: wudi <676366...@qq.com>
AuthorDate: Mon Nov 6 16:27:08 2023 +0800
add multisink to DorisBatchSink (#223)
Support multi-table writing on DorisBatchSink
Example：
```java
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
DorisBatchSink.Builder builder =
DorisBatchSink.builder();
final DorisReadOptions.Builder readOptionBuilder =
DorisReadOptions.builder();
Properties properties = new Properties();
properties.setProperty("column_separator", ",");
properties.setProperty("line_delimiter", "\n");
properties.setProperty("format", "csv");
DorisOptions.Builder dorisBuilder = DorisOptions.builder();
dorisBuilder.setFenodes("127.0.0.1:8030")
.setTableIdentifier("")
.setUsername("root")
.setPassword("");
DorisExecutionOptions.Builder executionBuilder =
DorisExecutionOptions.builder();
executionBuilder.setLabelPrefix("label")
.setStreamLoadProp(properties)
.setDeletable(false)
.setBufferFlushMaxBytes(8 * 1024)
.setBufferFlushMaxRows(10)
.setBufferFlushIntervalMs(1000 * 10);
builder.setDorisReadOptions(readOptionBuilder.build())
.setDorisExecutionOptions(executionBuilder.build())
.setDorisOptions(dorisBuilder.build());
//Multiple table writing
RecordWithMeta record = new RecordWithMeta("test",
"test_flink_tmp1", "wangwu,1");
RecordWithMeta record1 = new RecordWithMeta("test",
"test_flink_tmp", "wangwu,1");
DataStreamSource source =
env.fromCollection(Arrays.asList(record, record1));
source.sinkTo(builder.build());
```
---
.../doris/flink/sink/batch/BatchRecordBuffer.java | 25 +
.../doris/flink/sink/batch/DorisBatchSink.java | 1 -
.../flink/sink/batch/DorisBatchStreamLoad.java | 109 ++---
.../doris/flink/sink/batch/DorisBatchWriter.java | 29 +-
.../doris/flink/sink/batch/RecordWithMeta.java | 62
.../doris/flink/sink/writer/LabelGenerator.java| 4 +
.../doris/flink/DorisSinkMultiTableExample.java| 101 +++
7 files changed, 271 insertions(+), 60 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
index 5fa601d..1de6253 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
@@ -36,6 +36,8 @@ public class BatchRecordBuffer {
private int numOfRecords = 0;
private int bufferSizeBytes = 0;
private boolean loadBatchFirstRecord = true;
+private String database;
+private String table;
public BatchRecordBuffer(){}
@@ -45,6 +47,14 @@ public class BatchRecordBuffer {
this.buffer = ByteBuffer.allocate(bufferSize);
}
+public BatchRecordBuffer(String database, String table, byte[]
lineDelimiter, int bufferSize) {
+super();
+this.database = database;
+this.table = table;
+this.lineDelimiter = lineDelimiter;
+this.buffer = ByteBuffer.allocate(bufferSize);
+}
+
public void insert(byte[] record) {
ensureCapacity(record.length);
if(loadBatchFirstRecord){
@@ -141,4 +151,19 @@ public class BatchRecordBuffer {
this.bufferSizeBytes = bufferSizeBytes;
}
+public String getDatabase() {
+return database;
+}
+
+public void setDatabase(String database) {
+this.database = database;
+}
+
+public String getTable() {
+return table;
+}
+
+public void setTable(String table) {
+this.table = table;
+}
}
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java
index 2c578d4..37d3973 100644
---
a/flink-doris-connector/src/main/java/org/a
(doris-flink-connector) branch master updated: [improve] add multi table sink to DorisSink (#224)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 0b2a14a [improve] add multi table sink to DorisSink (#224)
0b2a14a is described below
commit 0b2a14a93d3b65115c25fe59d84d6b3375da517d
Author: wudi <676366...@qq.com>
AuthorDate: Mon Nov 6 16:56:04 2023 +0800
[improve] add multi table sink to DorisSink (#224)
DorisSink supports multi-table import.
Example:
```java
Configuration config = new Configuration();
//
config.setString("execution.savepoint.path","/tmp/checkpoint/chk-6");
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment(config);
env.setParallelism(1);
env.getCheckpointConfig().setCheckpointStorage("file:///tmp/checkpoint/");
env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
env.setRestartStrategy(RestartStrategies.fixedDelayRestart(5,
Time.milliseconds(1)));
env.enableCheckpointing(1);
DorisSink.Builder builder = DorisSink.builder();
final DorisReadOptions.Builder readOptionBuilder =
DorisReadOptions.builder();
Properties properties = new Properties();
properties.setProperty("column_separator", ",");
properties.setProperty("line_delimiter", "\n");
properties.setProperty("format", "csv");
DorisOptions.Builder dorisBuilder = DorisOptions.builder();
dorisBuilder.setFenodes("127.0.0.1:8030")
.setTableIdentifier("")
.setUsername("root")
.setPassword("");
DorisExecutionOptions.Builder executionBuilder =
DorisExecutionOptions.builder();
executionBuilder.setLabelPrefix("xxx12")
.setStreamLoadProp(properties)
.setDeletable(false).enable2PC();
builder.setDorisReadOptions(readOptionBuilder.build())
.setDorisExecutionOptions(executionBuilder.build())
.setDorisOptions(dorisBuilder.build());
RecordWithMeta record = new RecordWithMeta("test",
"test_flink_tmp1", "wangwu,1");
RecordWithMeta record1 = new RecordWithMeta("test",
"test_flink_tmp", "wangwu,1");
DataStreamSource stringDataStreamSource =
env.fromCollection(
Arrays.asList(record, record1));
stringDataStreamSource.sinkTo(builder.build());
```
For details, please refer to
`org.apache.doris.flink.DorisSinkStreamMultiTableExample.java`
---
.../org/apache/doris/flink/cfg/DorisOptions.java | 4 +
.../org/apache/doris/flink/sink/BackendUtil.java | 12 ++
.../org/apache/doris/flink/sink/DorisSink.java | 1 -
.../doris/flink/sink/writer/DorisStreamLoad.java | 12 +-
.../doris/flink/sink/writer/DorisWriter.java | 211 +++--
.../doris/flink/sink/writer/DorisWriterState.java | 32 +++-
.../sink/writer/DorisWriterStateSerializer.java| 17 +-
.../doris/flink/sink/writer/LabelGenerator.java| 19 +-
.../flink/DorisSinkStreamMultiTableExample.java| 100 ++
.../org/apache/doris/flink/sink/HttpTestUtil.java | 41
.../flink/sink/writer/TestDorisStreamLoad.java | 12 +-
.../doris/flink/sink/writer/TestDorisWriter.java | 12 +-
.../writer/TestDorisWriterStateSerializer.java | 4 +-
13 files changed, 394 insertions(+), 83 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
index f560eae..6391e91 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisOptions.java
@@ -52,6 +52,10 @@ public class DorisOptions extends DorisConnectionOptions {
return tableIdentifier;
}
+public void setTableIdentifier(String tableIdentifier) {
+this.tableIdentifier = tableIdentifier;
+}
+
public String save() throws IllegalArgumentException {
Properties copy = new Properties();
return IOUtils.propsToString(copy);
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/BackendUtil.java
index 0d45e2f..954bdd0 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/Back
(doris-flink-connector) branch master updated: skip table model check in multi table (#225)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 26bff2c skip table model check in multi table (#225)
26bff2c is described below
commit 26bff2cff4063a0676a75a4d94e9d3be10ab8593
Author: wudi <676366...@qq.com>
AuthorDate: Tue Nov 7 11:47:43 2023 +0800
skip table model check in multi table (#225)
skip table model check in multi table
Co-authored-by: wudi <>
---
.../src/main/java/org/apache/doris/flink/rest/RestService.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
index 1633c96..cd02209 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/rest/RestService.java
@@ -458,6 +458,11 @@ public class RestService implements Serializable {
public static boolean isUniqueKeyType(DorisOptions options,
DorisReadOptions readOptions, Logger logger)
throws DorisRuntimeException {
+//Enable 2pc in multi-table scenario
+if(StringUtils.isBlank(options.getTableIdentifier())){
+logger.info("table model verification is skipped in multi-table
scenarios.");
+return false;
+}
try {
return UNIQUE_KEYS_TYPE.equals(getSchema(options, readOptions,
logger).getKeysType());
} catch (Exception e) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [docs](fix) add bitmap_remove in sidebars.json (#26523)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 38a14c3325e [docs](fix) add bitmap_remove in sidebars.json (#26523)
38a14c3325e is described below
commit 38a14c3325e3796cff9f2dc26aa4dc4e139c5243
Author: Liqf <109049295+lemonlit...@users.noreply.github.com>
AuthorDate: Tue Nov 7 19:01:27 2023 +0800
[docs](fix) add bitmap_remove in sidebars.json (#26523)
---
docs/sidebars.json | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/sidebars.json b/docs/sidebars.json
index c3e73573c08..9fd9cf97554 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -608,7 +608,8 @@
"sql-manual/sql-functions/bitmap-functions/orthogonal-bitmap-intersect-count",
"sql-manual/sql-functions/bitmap-functions/orthogonal-bitmap-expr-calculate",
"sql-manual/sql-functions/bitmap-functions/orthogonal-bitmap-expr-calculate-count",
-
"sql-manual/sql-functions/bitmap-functions/bitmap-hash64"
+
"sql-manual/sql-functions/bitmap-functions/bitmap-hash64",
+
"sql-manual/sql-functions/bitmap-functions/bitmap-remove"
]
},
{
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: Revert "[Chore](ci)Temporarily cancel the mandatory restrictions of ShellCheck (#26553)" (#26565)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new f8f3bc6a670 Revert "[Chore](ci)Temporarily cancel the mandatory restrictions of ShellCheck (#26553)" (#26565) f8f3bc6a670 is described below commit f8f3bc6a6701fad5d2e94fc6d54c7ded4b8912c9 Author: Calvin Kirs AuthorDate: Wed Nov 8 11:52:08 2023 +0800 Revert "[Chore](ci)Temporarily cancel the mandatory restrictions of ShellCheck (#26553)" (#26565) This reverts commit b7c81bc73625b26df746fc2213980c16b9d8f1a0. --- .asf.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.asf.yaml b/.asf.yaml index 59f6fd20e75..f8b7e0913cf 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -59,6 +59,7 @@ github: - BE UT (Doris BE UT) - Build Broker - Build Documents + - ShellCheck - clickbench-new (clickbench) - Build Third Party Libraries (Linux) - Build Third Party Libraries (macOS) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](jdbc catalog) fix mysql zero date (#26569)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5bcf6bfd468 [fix](jdbc catalog) fix mysql zero date (#26569)
5bcf6bfd468 is described below
commit 5bcf6bfd468f699af1b817a1a6301675b59638de
Author: zy-kkk
AuthorDate: Wed Nov 8 21:41:56 2023 +0800
[fix](jdbc catalog) fix mysql zero date (#26569)
---
.../docker-compose/mysql/init/03-create-table.sql | 5 +
.../docker-compose/mysql/init/04-insert.sql| 5 +
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 132 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 52
.../datasource/jdbc/client/JdbcMySQLClient.java| 3 +
.../jdbc/test_mysql_jdbc_catalog.out | 4 +
.../jdbc/test_mysql_jdbc_catalog.groovy| 2 +
7 files changed, 96 insertions(+), 107 deletions(-)
diff --git a/docker/thirdparties/docker-compose/mysql/init/03-create-table.sql
b/docker/thirdparties/docker-compose/mysql/init/03-create-table.sql
index 4663ea2cf6e..ee44e1c87a2 100644
--- a/docker/thirdparties/docker-compose/mysql/init/03-create-table.sql
+++ b/docker/thirdparties/docker-compose/mysql/init/03-create-table.sql
@@ -321,3 +321,8 @@ CREATE TABLE show_test_do_not_modify.ex_tb2 (
id int,
count_value varchar(20)
);
+
+CREATE TABLE doris_test.test_zd (
+`id` int(10) unsigned NOT NULL,
+`d_z` date NOT NULL
+);
diff --git a/docker/thirdparties/docker-compose/mysql/init/04-insert.sql
b/docker/thirdparties/docker-compose/mysql/init/04-insert.sql
index 93ae8c9b63e..3ff51a9a123 100644
--- a/docker/thirdparties/docker-compose/mysql/init/04-insert.sql
+++ b/docker/thirdparties/docker-compose/mysql/init/04-insert.sql
@@ -1152,3 +1152,8 @@ VALUES ('2023-06-17 10:00:00', '2023-06-17 10:00:01.1',
'2023-06-17 10:00:02.22'
SET SESSION sql_mode=(SELECT REPLACE(@@sql_mode,'STRICT_TRANS_TABLES',''));
INSERT INTO doris_test.dt_null
VALUES ('2023-06-17 10:00:00'),('-00-00 00:00:00');
+
+SET SESSION sql_mode=(SELECT REPLACE(@@sql_mode,'NO_ZERO_DATE',''));
+SET SESSION sql_mode=(SELECT REPLACE(@@sql_mode,'NO_ZERO_IN_DATE',''));
+
+insert into doris_test.test_zd (id,d_z) VALUES
(1,'-00-00'),(2,'2022-01-01');
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index d6294846062..db41b033a49 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -695,60 +695,59 @@ DROP CATALOG ;
SET NAMES utf8mb4
```
-3. Why does the error message "CAUSED BY: DataReadException: Zero date value
prohibited" pop up when DateTime=":00:00 00:00:00" while reading MySQL
external tables?
+3. Exception occurs when reading MySQL date/datetime type
- This error occurs because of an illegal DateTime. It can be fixed by
modifying the `zeroDateTimeBehavior` parameter.
-
- The options for this parameter include:
`EXCEPTION`,`CONVERT_TO_NULL`,`ROUND`. Respectively, they mean to report error,
convert to null, and round the DateTime to "0001-01-01 00:00:00" when
encountering an illegal DateTime.
-
- You can add
`"jdbc_url"="jdbc:mysql://IP:PORT/doris_test?zeroDateTimeBehavior=convertToNull"`
to the URL.
-
-4. Why do loading failures happen when reading MySQL or other external tables?
+```
+ERROR 1105 (HY000): errCode = 2, detailMessage =
(10.16.10.6)[INTERNAL_ERROR]UdfRuntimeException: get next block failed:
+CAUSED BY: SQLException: Zero date value prohibited
+CAUSED BY: DataReadException: Zero date value prohibited
+```
- For example:
+This is because the default handling of illegal Date/DateTime in JDBC is
to throw an exception, and this behavior can be controlled through the
parameter `zeroDateTimeBehavior`.
- ```
- failed to load driver class com.mysql.jdbc.driver in either of hikariconfig
class loader
- ```
+The optional parameters are: `EXCEPTION`, `CONVERT_TO_NULL`, `ROUND`,
respectively: exception error reporting, converted to NULL value, converted to
"0001-01-01 00:00:00";
- Such errors occur because the `driver_class` has been wrongly put when
creating the catalog. The problem with the above example is the letter case. It
should be corrected as `"driver_class" = "com.mysql.jdbc.Driver"`.
+You need to add `zeroDateTimeBehavior=convertToNull` to the end of the
JDBC connection string when creating the Catalog `jdbc_url`, such as
`"jdbc_url" =
"jdbc:mysql://127.0.0.1:3306/test?zeroDateTimeBehavior=convertToNull"`
+In this case, JDBC will convert -00-00 or -00-00 00:00:00 into
null, and then Doris will
(doris) branch master updated (30805a2d79b -> 84343893588)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 30805a2d79b [fix](Nereids): skip OptimizeGroupExpressionJob if groupExpression is unused (#26641) add 84343893588 [fix](jdbc) fix clickhouse catalog arr nullable and add case (#26639) No new revisions were added by this update. Summary of changes: .../clickhouse/init/03-create-table.sql| 141 + .../docker-compose/clickhouse/init/04-insert.sql | 13 ++ .../java/org/apache/doris/jdbc/JdbcExecutor.java | 38 +- .../jdbc/test_clickhouse_jdbc_catalog.out | Bin 2809 -> 4896 bytes .../jdbc/test_clickhouse_jdbc_catalog.groovy | 8 ++ 5 files changed, 195 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](jdbc) add doris and sqlserver jdbc catalog test case (#26656)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7ce746654a1 [test](jdbc) add doris and sqlserver jdbc catalog test
case (#26656)
7ce746654a1 is described below
commit 7ce746654a1321a9210ce99d2d1d4129c80298ed
Author: zy-kkk
AuthorDate: Fri Nov 10 10:32:09 2023 +0800
[test](jdbc) add doris and sqlserver jdbc catalog test case (#26656)
---
.../sqlserver/init/03-create-table.sql | 33 ++
.../docker-compose/sqlserver/init/04-insert.sql| 34 ++
.../jdbc/test_doris_jdbc_catalog.out | 53 ++
.../jdbc/test_sqlserver_jdbc_catalog.out | 43 ++
.../jdbc/test_doris_jdbc_catalog.groovy| 12 +
.../jdbc/test_sqlserver_jdbc_catalog.groovy| 6 ++-
6 files changed, 180 insertions(+), 1 deletion(-)
diff --git
a/docker/thirdparties/docker-compose/sqlserver/init/03-create-table.sql
b/docker/thirdparties/docker-compose/sqlserver/init/03-create-table.sql
index a491647e785..cddd38f7863 100644
--- a/docker/thirdparties/docker-compose/sqlserver/init/03-create-table.sql
+++ b/docker/thirdparties/docker-compose/sqlserver/init/03-create-table.sql
@@ -101,3 +101,36 @@ CREATE TABLE dbo.t_id (
Name nvarchar(100)
);
+CREATE TABLE dbo.all_type (
+id int PRIMARY KEY NOT NULL,
+name varchar(10) NULL,
+age int NULL,
+tinyint_value tinyint NULL,
+smallint_value smallint NULL,
+bigint_value bigint NULL,
+real_value real NULL,
+float_value float NULL,
+floatn_value float(5) NULL,
+decimal_value decimal(38,0) NULL,
+numeric_value numeric(38,0) NULL,
+decimal_value2 decimal(38,10) NULL,
+numeric_value2 numeric(38,10) NULL,
+char_value char(20) NULL,
+varchar_value varchar(20) NULL,
+varcharmax_value varchar(max) NULL,
+nchar_value nchar(20) NULL,
+nvarchar_value nvarchar(20) NULL,
+nvarcharmax_value nvarchar(max) NULL,
+date_value date NULL,
+time_value time NULL,
+datetime_value datetime NULL,
+datetime2_value datetime2 NULL,
+smalldatetime_value smalldatetime NULL,
+datetimeoffset_value datetimeoffset NULL,
+text_value text NULL,
+ntext_value ntext NULL,
+money_value money NULL,
+smallmoney_value smallmoney NULL,
+bit_value bit NULL
+);
+
diff --git a/docker/thirdparties/docker-compose/sqlserver/init/04-insert.sql
b/docker/thirdparties/docker-compose/sqlserver/init/04-insert.sql
index f4f67523dfb..c18f629707d 100644
--- a/docker/thirdparties/docker-compose/sqlserver/init/04-insert.sql
+++ b/docker/thirdparties/docker-compose/sqlserver/init/04-insert.sql
@@ -63,3 +63,37 @@ VALUES (
INSERT INTO dbo.t_id (ID, Name) VALUES (NEWID(), 'Data 1');
INSERT INTO dbo.t_id (ID, Name) VALUES (NEWID(), 'Data 2');
+Insert into dbo.all_type values
+(
+1,
+'doris',
+18,
+0,
+1,
+1,
+123.123,
+123.123,
+123.123,
+12345678901234567890123456789012345678,
+12345678901234567890123456789012345678,
+1234567890123456789012345678.0123456789,
+1234567890123456789012345678.0123456789,
+'Make Doris Great!',
+'Make Doris Great!',
+'Make Doris Great!',
+'Make Doris Great!',
+'Make Doris Great!',
+'Make Doris Great!',
+'2023-01-17',
+'16:49:05.1234567',
+'2023-01-17 16:49:05',
+'2023-01-17 16:49:05.1234567',
+'2023-01-17 16:49:05',
+'2023-01-17 16:49:05+08:00',
+'Make Doris Great!',
+'Make Doris Great!',
+922337203685477.5807,
+214748.3647,
+0
+),
+(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
diff --git
a/regression-test/data/external_table_p0/jdbc/test_doris_jdbc_catalog.out
b/regression-test/data/external_table_p0/jdbc/test_doris_jdbc_catalog.out
index 011a1f2a4b5..2e686056d40 100644
--- a/regression-test/data/external_table_p0/jdbc/test_doris_jdbc_catalog.out
+++ b/regression-test/data/external_table_p0/jdbc/test_doris_jdbc_catalog.out
@@ -28,10 +28,12 @@ doris_jdbc_catalog
6 doris6
-- !base1 --
+\N \N \N \N \N \N \N \N \N \N
\N \N \N \N \N
true 1 1 1 1 1 1.0 1.0 1.0
1.002021-01-01 2021-01-01T00:00a a {"a":1}
-- !arr1 --
1 [1] [1] [1] [1] [1] [1] [1] [1]
[1.0] [1.00] ["2021-01-01"] ["2021-01-01 00:00:00.000"]
["a"] ["a"] ["a"]
+2 \N \N \N \N \N \N \N \N \N
\N \N \N \N \N \N
(doris) branch branch-2.0 updated: [docs](docs) Update Files of Branch-2.0 (#26737)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 54111934778 [docs](docs) Update Files of Branch-2.0 (#26737)
54111934778 is described below
commit 5411193477863d67f538cd631c4008c0fa2023c5
Author: KassieZ <139741991+kass...@users.noreply.github.com>
AuthorDate: Fri Nov 10 14:24:02 2023 +0800
[docs](docs) Update Files of Branch-2.0 (#26737)
---
.../maint-monitor/tablet-restore-tool.md | 136 ++
.../ecosystem/udf/native-user-defined-function.md | 271
.../lakehouse/multi-catalog/faq-multi-catalog.md | 235 +
.../docs/lakehouse/multi-catalog/multi-catalog.md | 4 +-
.../enable-feature.md | 38 +++
docs/sidebars.json | 16 +-
.../maint-monitor/tablet-restore-tool.md | 142 +++
.../docs/ecosystem/native-user-defined-function.md | 278 +
.../lakehouse/multi-catalog/faq-multi-catalog.md | 238 ++
.../window-functions/window-function-avg.md| 2 +-
.../window-functions/window-function-max.md| 2 +-
.../window-functions/window-function-ntile.md | 2 +-
.../enable-feature.md | 38 +++
13 files changed, 1392 insertions(+), 10 deletions(-)
diff --git a/docs/en/docs/admin-manual/maint-monitor/tablet-restore-tool.md
b/docs/en/docs/admin-manual/maint-monitor/tablet-restore-tool.md
new file mode 100644
index 000..a33c1dc5ca5
--- /dev/null
+++ b/docs/en/docs/admin-manual/maint-monitor/tablet-restore-tool.md
@@ -0,0 +1,136 @@
+---
+{
+"title": "Tablet Restore Tool",
+"language": "en"
+}
+---
+
+
+
+# Tablet Restore Tool
+
+## Restore data from BE Recycle Bin
+
+During the user's use of Doris, some valid tablets (including metadata and
data) may be deleted due to some misoperations or online bugs. In order to
prevent data loss in these abnormal situations, Doris provides a recycle bin
mechanism to protect user data. Tablet data deleted by users will not be
deleted directly, but will be stored in the recycle bin for a period of time.
After a period of time, there will be a regular cleaning mechanism to delete
expired data. The data in the recycle [...]
+
+```
+/root_path/trash/time_label/tablet_id/schema_hash/
+```
+
+* `root_path`: a data root directory corresponding to the BE node.
+* `trash`: The directory of the recycle bin.
+* `time_label`: Time label, for the uniqueness of the data directory in the
recycle bin, while recording the data time, use the time label as a
subdirectory.
+
+When a user finds that online data has been deleted by mistake, he needs to
recover the deleted tablet from the recycle bin. This tablet data recovery
function is needed.
+
+BE provides http interface and `restore_tablet_tool.sh` script to achieve this
function, and supports single tablet operation (single mode) and batch
operation mode (batch mode).
+
+* In single mode, data recovery of a single tablet is supported.
+* In batch mode, support batch tablet data recovery.
+
+### Operation
+
+ single mode
+
+1. http request method
+
+BE provides an http interface for single tablet data recovery, the
interface is as follows:
+
+```
+curl -X POST
"http://be_host:be_webserver_port/api/restore_tablet?tablet_id=1\&schema_hash=12345";
+```
+
+The successful results are as follows:
+
+```
+{"status": "Success", "msg": "OK"}
+```
+
+If it fails, the corresponding failure reason will be returned. One
possible result is as follows:
+
+```
+{"status": "Failed", "msg": "create link path failed"}
+```
+
+2. Script mode
+
+`restore_tablet_tool.sh` can be used to realize the function of single
tablet data recovery.
+
+```
+sh tools/restore_tablet_tool.sh -b "http://127.0.0.1:8040"; -t 12345 -s
1
+sh tools/restore_tablet_tool.sh --backend "http://127.0.0.1:8040";
--tablet_id 12345 --schema_hash 1
+```
+
+ batch mode
+
+The batch recovery mode is used to realize the function of recovering multiple
tablet data.
+
+When using, you need to put the restored tablet id and schema hash in a file
in a comma-separated format in advance, one tablet per line.
+
+The format is as follows:
+
+```
+12345,1
+12346,1
+12347,1
+```
+
+Then perform the recovery with the following command (assuming the file name
is: `tablets.txt`):
+
+```
+sh restore_tablet_tool.sh -b "http://127.0.0.1:8040"; -f tablets.txt
+sh restore_tablet_tool.sh --backend "http://127.0.0.1:8040"; --file tablets.txt
+```
+
(doris) branch master updated (7e36ab838f1 -> db29850e1c3)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7e36ab838f1 [regression](partial update) Add cases when the deleted rows have non nullable columns without default value (#26776) add db29850e1c3 [bug](user login)fix PASSWORD_LOCK_TIME setting UNBOUNDED does not take effect (#26585) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/mysql/privilege/PasswordPolicy.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated (e1574ed9f1e -> 2f7483234d1)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from e1574ed9f1e [compile](gcc) fix gcc compile error #26863 add 2f7483234d1 [test](jdbc) pick some jdbc test from branch master (#26860) No new revisions were added by this update. Summary of changes: .../clickhouse/init/03-create-table.sql| 141 + .../docker-compose/clickhouse/init/04-insert.sql | 13 ++ .../sqlserver/init/03-create-table.sql | 33 + .../docker-compose/sqlserver/init/04-insert.sql| 34 + .../java/org/apache/doris/jdbc/JdbcExecutor.java | 19 ++- .../jdbc/client/JdbcClickHouseClient.java | 2 + regression-test/conf/regression-conf.groovy| 10 +- .../jdbc/test_clickhouse_jdbc_catalog.out | Bin 2801 -> 4887 bytes .../jdbc/test_doris_jdbc_catalog.out | 56 +++- .../jdbc/test_sqlserver_jdbc_catalog.out | 43 +++ .../jdbc/test_clickhouse_jdbc_catalog.groovy | 8 ++ .../jdbc/test_doris_jdbc_catalog.groovy| 12 ++ .../jdbc/test_sqlserver_jdbc_catalog.groovy| 6 +- 13 files changed, 368 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [docs](readme)Update README.md (#26844)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 761fa68ab23 [docs](readme)Update README.md (#26844) 761fa68ab23 is described below commit 761fa68ab2392cd050b03f20d0e97f9f3d2ab36d Author: Luzhijing <82810928+luzhij...@users.noreply.github.com> AuthorDate: Mon Nov 13 14:29:39 2023 +0800 [docs](readme)Update README.md (#26844) --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index cecece03282..d14ad11deb9 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,6 @@ Apache Doris is an easy-to-use, high-performance and real-time analytical databa All this makes Apache Doris an ideal tool for scenarios including report analysis, ad-hoc query, unified data warehouse, and data lake query acceleration. On Apache Doris, users can build various applications, such as user behavior analysis, AB test platform, log retrieval analysis, user portrait analysis, and order analysis. -Doris Summit Asia 2023 is coming and warmly invite you to join! Click Now 🔗[doris-summit.org.cn](https://doris-summit.org.cn/?utm_source=website&utm_medium=readme&utm_campaign=2023&utm_id=2023) - 🎉 Version 2.0.2 version released now. The 2.0.2 version has achieved over 10x performance improvements on standard Benchmark, comprehensive enhancement in log analysis and lakehouse scenarios, more efficient and stable data update and write efficiency, support for more comprehensive multi-tenant and resource isolation mechanisms, and take a new step in the direction of resource elasticity and storage computing separation. It has also been added a series of usability features for enterpri [...] 🎉 Version 1.2.7 released now! It is fully evolved release and all users are encouraged to upgrade to this release. Check out the 🔗[Release Notes](https://doris.apache.org/docs/dev/releasenotes/release-1.2.7) here. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](doc)modify error link description in fe-config-template (#24676)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new a6d1da0db90 [typo](doc)modify error link description in fe-config-template (#24676) a6d1da0db90 is described below commit a6d1da0db90db76eaadab1dacaa8df23151cd416 Author: ZhenchaoXu <49646212+i...@users.noreply.github.com> AuthorDate: Wed Sep 27 15:51:47 2023 +0800 [typo](doc)modify error link description in fe-config-template (#24676) --- docs/en/docs/admin-manual/config/fe-config-template.md| 2 +- docs/zh-CN/docs/admin-manual/config/fe-config-template.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/admin-manual/config/fe-config-template.md b/docs/en/docs/admin-manual/config/fe-config-template.md index e73cf1ec9f2..2ae517ac243 100644 --- a/docs/en/docs/admin-manual/config/fe-config-template.md +++ b/docs/en/docs/admin-manual/config/fe-config-template.md @@ -52,7 +52,7 @@ There are two ways to view the configuration items of FE: 1. FE web page -Open the FE web page `http://fe_host:fe_http_port/variable` in the browser. You can see the currently effective FE configuration items in `Configure Info`. +Open the FE web page `http://fe_host:fe_http_port/Configure` in the browser. You can see the currently effective FE configuration items in `Configure Info`. 2. View by command diff --git a/docs/zh-CN/docs/admin-manual/config/fe-config-template.md b/docs/zh-CN/docs/admin-manual/config/fe-config-template.md index 08a711aadb0..a0fb6a62778 100644 --- a/docs/zh-CN/docs/admin-manual/config/fe-config-template.md +++ b/docs/zh-CN/docs/admin-manual/config/fe-config-template.md @@ -50,7 +50,7 @@ FE 的配置项有两种方式进行查看： 1. FE 前端页面查看 - 在浏览器中打开 FE 前端页面 `http://fe_host:fe_http_port/variable`。在 `Configure Info` 中可以看到当前生效的 FE 配置项。 + 在浏览器中打开 FE 前端页面 `http://fe_host:fe_http_port/Configure`。在 `Configure Info` 中可以看到当前生效的 FE 配置项。 2. 通过命令查看 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a6d1da0db90 -> 53025ce3fcd)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a6d1da0db90 [typo](doc)modify error link description in fe-config-template (#24676) add 53025ce3fcd [typo](doc)modify error description of GROUP_CONCAT fiction. (#24619) No new revisions were added by this update. Summary of changes: .../docs/sql-manual/sql-functions/aggregate-functions/group-concat.md | 2 +- .../docs/sql-manual/sql-functions/aggregate-functions/group-concat.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](doc)modify the wrong parameter in the example. (#24307)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 41dfbdac14f [typo](doc)modify the wrong parameter in the example. (#24307) 41dfbdac14f is described below commit 41dfbdac14f31684bee71fdb571b7e8bedd44e0c Author: ZhenchaoXu <49646212+i...@users.noreply.github.com> AuthorDate: Wed Sep 27 16:04:07 2023 +0800 [typo](doc)modify the wrong parameter in the example. (#24307) --- .../Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md| 2 +- .../Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md index 12cbbfc62f2..f9484d63c56 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md @@ -428,7 +428,7 @@ Assuming that we need to import data from Kafka into tables "test1" and "test2" "max_batch_interval" = "20", "max_batch_rows" = "30", "max_batch_size" = "209715200", - "strict_mode" = "false" + "strict_mode" = "true" ) FROM KAFKA ( diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md index 2f6db96c167..4e9b55d49c6 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Load/CREATE-ROUTINE-LOAD.md @@ -430,7 +430,7 @@ FROM data_source [data_source_properties] "max_batch_interval" = "20", "max_batch_rows" = "30", "max_batch_size" = "209715200", - "strict_mode" = "false" + "strict_mode" = "true" ) FROM KAFKA ( - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](doc)modify error result of explode_split function. (#24185)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 83f5ff7b225 [typo](doc)modify error result of explode_split function. (#24185) 83f5ff7b225 is described below commit 83f5ff7b225d64d1322dd222d42cd7990bbbc0af Author: ZhenchaoXu <49646212+i...@users.noreply.github.com> AuthorDate: Wed Sep 27 16:04:18 2023 +0800 [typo](doc)modify error result of explode_split function. (#24185) --- .../sql-manual/sql-functions/table-functions/explode-split.md| 9 ++--- .../sql-manual/sql-functions/table-functions/explode-split.md| 3 +-- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-functions/table-functions/explode-split.md b/docs/en/docs/sql-manual/sql-functions/table-functions/explode-split.md index ae9af43b196..1ccf91c40aa 100644 --- a/docs/en/docs/sql-manual/sql-functions/table-functions/explode-split.md +++ b/docs/en/docs/sql-manual/sql-functions/table-functions/explode-split.md @@ -70,18 +70,13 @@ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e +--+--+ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 2 order by k1, e1; -+--+--+ -| k1 | e1 | -+--+--+ -|2 | NULL | -+--+--+ +Empty set mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 3 order by k1, e1; +--+--+ | k1 | e1 | +--+--+ |3 | | -|3 | | +--+--+ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 4 order by k1, e1; @@ -95,9 +90,9 @@ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e +--+--+ | k1 | e1 | +--+--+ -|5 | 1| |5 | 2| |5 | 3| +|5 | 1| +--+--+ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 6 order by k1, e1; diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/explode-split.md b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/explode-split.md index faaae733870..64f5b1eee72 100644 --- a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/explode-split.md +++ b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/explode-split.md @@ -71,7 +71,6 @@ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e | k1 | e1 | +--+--+ |3 | | -|3 | | +--+--+ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 4 order by k1, e1; @@ -85,9 +84,9 @@ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e +--+--+ | k1 | e1 | +--+--+ -|5 | 1| |5 | 2| |5 | 3| +|5 | 1| +--+--+ mysql> select k1, e1 from example1 lateral view explode_split(k2, ',') tmp1 as e1 where k1 = 6 order by k1, e1; - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (0f6ea412207 -> 47694c5b36f)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0f6ea412207 [bug][auth]Show grant causes role errors in the memory. #24783 (#24841) add 47694c5b36f [fix](jdbc catalog )fix jdbc catalog current_timestamp default (#25016) No new revisions were added by this update. Summary of changes: .../doris/datasource/jdbc/client/JdbcMySQLClient.java | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (1405f1efd24 -> 59261174d55)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1405f1efd24 [refactor](nereids) unify withSel/updateRowCountOnly/withRowCount (#24997) add 59261174d55 [chore](unused) Remove unused variable CPU_HARD_LIMIT in task_group.cc (#25076) No new revisions were added by this update. Summary of changes: be/src/runtime/task_group/task_group.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (9d0f4c00943 -> 5130a6c006e)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 9d0f4c00943 [minor](be) set fd number check to 6 for BE start script (#25078) add 5130a6c006e [improvement](jdbc catalog)Adjustment to JDBC External Table Configuration Based on Internal Table Settings (#25059) No new revisions were added by this update. Summary of changes: docs/en/docs/lakehouse/multi-catalog/jdbc.md | 4 +++- docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 4 +++- .../java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java | 7 +++ 3 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (b380b8b0b52 -> f3e95608cb3)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from b380b8b0b52 [bugfix](multi-catalog) Esexternalcatalog is missing LastUpdateTime. (#24559) add f3e95608cb3 (Fix)(RoutineLoad)Query the transaction status NPE when the task has not yet started scheduling (#25074) No new revisions were added by this update. Summary of changes: .../Show-Statements/SHOW-ROUTINE-LOAD-TASK.md | 2 +- .../Show-Statements/SHOW-ROUTINE-LOAD-TASK.md | 2 +- .../doris/load/routineload/RoutineLoadJob.java | 22 -- .../load/routineload/RoutineLoadTaskInfo.java | 9 +++-- 4 files changed, 25 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (07f9f27fa9 -> 96eb363b01)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 07f9f27fa9 [improvement](start script) start script can not set http proxy (#25086) add 96eb363b01 [fix](help-module)fix use regex match replaceAll may cause backtracking (#24918) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/doris/common/MarkDownParser.java | 15 ++- 1 file changed, 14 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](auth)fix use regex verify mysql password may cause backtracking (#24900)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 737124aaef9 [fix](auth)fix use regex verify mysql password may cause
backtracking (#24900)
737124aaef9 is described below
commit 737124aaef9eb4e183c1863ff8ccc752f85d3027
Author: DongLiang-0 <46414265+donglian...@users.noreply.github.com>
AuthorDate: Sun Oct 8 12:18:20 2023 +0800
[fix](auth)fix use regex verify mysql password may cause backtracking
(#24900)
---
.../java/org/apache/doris/mysql/MysqlPassword.java | 22 --
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlPassword.java
b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlPassword.java
index 10fcefcb710..7861e99e92c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlPassword.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlPassword.java
@@ -31,6 +31,8 @@ import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.util.Random;
+import java.util.Set;
+import java.util.stream.Collectors;
// this is stolen from MySQL
//
@@ -83,6 +85,12 @@ public class MysqlPassword {
private static final byte[] DIG_VEC_UPPER = {'0', '1', '2', '3', '4', '5',
'6', '7',
'8', '9', 'A', 'B', 'C', 'D', 'E', 'F'};
private static final Random random = new SecureRandom();
+private static final Set complexCharSet;
+public static final int MIN_PASSWORD_LEN = 8;
+
+static {
+complexCharSet = "~!@#$%^&*()_+|<>,.?/:;'[]{}".chars().mapToObj(c ->
(char) c).collect(Collectors.toSet());
+}
public static byte[] createRandomString(int len) {
byte[] bytes = new byte[len];
@@ -281,12 +289,6 @@ public class MysqlPassword {
return passwd;
}
-public static final String REG_NUMBER = ".*\\d+.*";
-public static final String REG_UPPERCASE = ".*[A-Z]+.*";
-public static final String REG_LOWERCASE = ".*[a-z]+.*";
-public static final String REG_SYMBOL =
".*[~!@#$%^&*()_+|<>,.?/:;'\\[\\]{}\"]+.*";
-public static final int MIN_PASSWORD_LEN = 8;
-
public static void validatePlainPassword(long validaPolicy, String text)
throws AnalysisException {
if (validaPolicy == GlobalVariable.VALIDATE_PASSWORD_POLICY_STRONG) {
if (Strings.isNullOrEmpty(text) || text.length() <
MIN_PASSWORD_LEN) {
@@ -295,16 +297,16 @@ public class MysqlPassword {
}
int i = 0;
-if (text.matches(REG_NUMBER)) {
+if (text.chars().anyMatch(Character::isDigit)) {
i++;
}
-if (text.matches(REG_LOWERCASE)) {
+if (text.chars().anyMatch(Character::isLowerCase)) {
i++;
}
-if (text.matches(REG_UPPERCASE)) {
+if (text.chars().anyMatch(Character::isUpperCase)) {
i++;
}
-if (text.matches(REG_SYMBOL)) {
+if (text.chars().anyMatch(c -> complexCharSet.contains((char) c)))
{
i++;
}
if (i < 3) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch master updated: [improve] add auto redirect and uniq default open 2pc (#202)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 8aaaded [improve] add auto redirect and uniq default open 2pc (#202)
8aaaded is described below
commit 8aaadedaf3fb3147a134e333603cb1af3d7530cf
Author: wudi <676366...@qq.com>
AuthorDate: Sun Oct 8 14:31:37 2023 +0800
[improve] add auto redirect and uniq default open 2pc (#202)
---
.../doris/flink/cfg/DorisConnectionOptions.java| 27 +++---
.../doris/flink/cfg/DorisExecutionOptions.java | 27 +++---
.../org/apache/doris/flink/cfg/DorisOptions.java | 12 +++---
.../flink/exception/ConnectedFailedException.java | 4 ++--
.../org/apache/doris/flink/rest/RestService.java | 20
.../apache/doris/flink/rest/models/BackendV2.java | 8 +++
.../org/apache/doris/flink/sink/DorisSink.java | 16 +
.../doris/flink/table/DorisConfigOptions.java | 4 ++--
.../flink/table/DorisDynamicTableFactory.java | 9 ++--
.../apache/doris/flink/tools/cdc/DatabaseSync.java | 10 ++--
10 files changed, 120 insertions(+), 17 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisConnectionOptions.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisConnectionOptions.java
index 1382dde..541e4e5 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisConnectionOptions.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisConnectionOptions.java
@@ -32,6 +32,11 @@ public class DorisConnectionOptions implements Serializable {
protected final String password;
protected String jdbcUrl;
protected String benodes;
+/**
+ * Used to enable automatic redirection of fe,
+ * When it is not enabled, it will actively request the be list, and the
polling will initiate a streamload request to be.
+ */
+protected boolean autoRedirect;
public DorisConnectionOptions(String fenodes, String username, String
password) {
this.fenodes = Preconditions.checkNotNull(fenodes, "fenodes is
empty");
@@ -45,10 +50,11 @@ public class DorisConnectionOptions implements Serializable
{
}
public DorisConnectionOptions(String fenodes, String benodes, String
username, String password,
-String jdbcUrl) {
+String jdbcUrl, boolean autoRedirect) {
this(fenodes, username, password);
this.benodes = benodes;
this.jdbcUrl = jdbcUrl;
+this.autoRedirect = autoRedirect;
}
public String getFenodes() {
@@ -71,21 +77,31 @@ public class DorisConnectionOptions implements Serializable
{
return jdbcUrl;
}
+public boolean isAutoRedirect() {
+return autoRedirect;
+}
+
/**
* Builder for {@link DorisConnectionOptions}.
*/
public static class DorisConnectionOptionsBuilder {
private String fenodes;
+private String benodes;
private String username;
private String password;
-
private String jdbcUrl;
+private boolean autoRedirect;
public DorisConnectionOptionsBuilder withFenodes(String fenodes) {
this.fenodes = fenodes;
return this;
}
+public DorisConnectionOptionsBuilder withBenodes(String benodes) {
+this.benodes = benodes;
+return this;
+}
+
public DorisConnectionOptionsBuilder withUsername(String username) {
this.username = username;
return this;
@@ -101,8 +117,13 @@ public class DorisConnectionOptions implements
Serializable {
return this;
}
+public DorisConnectionOptionsBuilder withAutoRedirect(boolean
autoRedirect) {
+this.autoRedirect = autoRedirect;
+return this;
+}
+
public DorisConnectionOptions build() {
-return new DorisConnectionOptions(fenodes, username, password,
jdbcUrl);
+return new DorisConnectionOptions(fenodes, benodes, username,
password, jdbcUrl, autoRedirect);
}
}
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
index 2422df8..8f7022d 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/cfg/DorisExecutionOptions.java
@@ -49,7 +49,8 @@ public class DorisExecutionOptions implements Serializable {
*/
private final Properties streamLoadProp;
private final Boolean enableDelete;
-private final Boolean enable2PC;
+
[doris-flink-connector] branch master updated: fix buffer memory grow bug (#203)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new c94c331 fix buffer memory grow bug (#203)
c94c331 is described below
commit c94c3311e5f0c4e4b80300e808c4a1209e24efba
Author: wudi <676366...@qq.com>
AuthorDate: Sun Oct 8 14:32:01 2023 +0800
fix buffer memory grow bug (#203)
Co-authored-by: wudi <>
---
.../doris/flink/sink/batch/BatchRecordBuffer.java | 20
.../flink/sink/batch/TestBatchRecordBuffer.java| 27 ++
2 files changed, 33 insertions(+), 14 deletions(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
index 99876bb..5fa601d 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java
@@ -17,7 +17,6 @@
package org.apache.doris.flink.sink.batch;
-import org.apache.doris.flink.sink.writer.RecordBuffer;
import org.apache.flink.annotation.VisibleForTesting;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -60,22 +59,25 @@ public class BatchRecordBuffer {
@VisibleForTesting
public void ensureCapacity(int length) {
-if(buffer.remaining() >= length){
+int lineDelimiterSize = this.lineDelimiter.length;
+if(buffer.remaining() - lineDelimiterSize >= length){
return;
}
int currentRemain = buffer.remaining();
int currentCapacity = buffer.capacity();
-
-int target = buffer.remaining() + length;
-int capacity = buffer.capacity();
-//grow 512kb each time
-target = Math.max(target, Math.min(capacity + 512 * 1024, capacity *
2));
-ByteBuffer tmp = ByteBuffer.allocate(target);
+// add lineDelimiter length
+int needed = length - buffer.remaining() + lineDelimiterSize;
+// grow at least 1MB
+long grow = Math.max(needed, 1024 * 1024);
+// grow at least 50% of the current size
+grow = Math.max(buffer.capacity() / 2, grow);
+int newCapacity = (int) Math.min(Integer.MAX_VALUE, buffer.capacity()
+ grow);
+ByteBuffer tmp = ByteBuffer.allocate(newCapacity);
buffer.flip();
tmp.put(buffer);
buffer.clear();
buffer = tmp;
-LOG.info("record length {},buffer remain {} ,grow capacity {} to {}",
length, currentRemain, currentCapacity, target);
+LOG.info("record length {},buffer remain {} ,grow capacity {} to {}",
length, currentRemain, currentCapacity, newCapacity);
}
public String getLabelName() {
diff --git
a/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestBatchRecordBuffer.java
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestBatchRecordBuffer.java
index 18cb79a..1139358 100644
---
a/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestBatchRecordBuffer.java
+++
b/flink-doris-connector/src/test/java/org/apache/doris/flink/sink/batch/TestBatchRecordBuffer.java
@@ -55,21 +55,38 @@ public class TestBatchRecordBuffer {
BatchRecordBuffer recordBuffer = new
BatchRecordBuffer("\n".getBytes(StandardCharsets.UTF_8),1);
recordBuffer.ensureCapacity(10);
-Assert.assertEquals(recordBuffer.getBuffer().capacity(), 10 + 1);
+//grow at least 1MB
+Assert.assertEquals(recordBuffer.getBuffer().capacity(), 1024 * 1024 +
1);
recordBuffer.ensureCapacity(100);
-Assert.assertEquals(recordBuffer.getBuffer().capacity(), 100 + 11);
+Assert.assertEquals(recordBuffer.getBuffer().capacity(), 1024 * 1024 +
1);
recordBuffer.ensureCapacity(1024);
-Assert.assertEquals(recordBuffer.getBuffer().capacity(), 1024 + 111);
+Assert.assertEquals(recordBuffer.getBuffer().capacity(), 1024 * 1024 +
1);
+//not need grow
recordBuffer = new
BatchRecordBuffer("\n".getBytes(StandardCharsets.UTF_8),16);
-recordBuffer.ensureCapacity(16);
+recordBuffer.ensureCapacity(15);
Assert.assertEquals(recordBuffer.getBuffer().capacity(), 16);
recordBuffer.insert("1234567890".getBytes(StandardCharsets.UTF_8));
recordBuffer.ensureCapacity(8);
-Assert.assertEquals(recordBuffer.getBuffer().capacity(), 32);
+Assert.assertEquals(recordBuffer.getBuffer().capacity(), 1024 * 1024 +
16);
+
+recordBuffer = new
BatchRecordBuffer("\n".getBytes(StandardCharsets.UTF_8),10);
+recordBuffer.insert("1234567".getBytes
[doris] branch master updated (f41b6a5fc3d -> 6d8391e8b52)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f41b6a5fc3d [minor](doc) update the doc for docker env and custom_lib dir (#25088) add 6d8391e8b52 [fix](load)fix use regex split partition may cause backtracking (#24903) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/qe/MultiLoadMgr.java | 41 -- .../java/org/apache/doris/task/StreamLoadTask.java | 10 +++--- 2 files changed, 27 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6d8391e8b52 -> 683546f71fc)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6d8391e8b52 [fix](load)fix use regex split partition may cause backtracking (#24903) add 683546f71fc [fix](catalog)fix use regex parse partition may cause backtracking (#24876) No new revisions were added by this update. Summary of changes: .../glue/catalog/converters/PartitionNameParser.java| 13 + 1 file changed, 5 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [doc](data-model) update data-model doc (#24941)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new aa1704c50b5 [doc](data-model) update data-model doc (#24941) aa1704c50b5 is described below commit aa1704c50b5077b00aee6ac0652f2d5c09c76c05 Author: Guangdong Liu AuthorDate: Mon Oct 9 10:08:16 2023 +0800 [doc](data-model) update data-model doc (#24941) --- docs/en/docs/data-table/data-model.md| 357 +++ docs/zh-CN/docs/data-table/data-model.md | 399 +-- 2 files changed, 430 insertions(+), 326 deletions(-) diff --git a/docs/en/docs/data-table/data-model.md b/docs/en/docs/data-table/data-model.md index 77c7c771606..4e6cae8c668 100644 --- a/docs/en/docs/data-table/data-model.md +++ b/docs/en/docs/data-table/data-model.md @@ -50,22 +50,24 @@ We illustrate what aggregation model is and how to use it correctly with practic Assume that the business has the following data table schema: -|ColumnName|Type|AggregationType|Comment| -|---|---|---|---| -| userid | LARGEINT | | user id| -| date | DATE | | date of data filling| -| City | VARCHAR (20) | | User City| -| age | SMALLINT | | User age| -| sex | TINYINT | | User gender| -| Last_visit_date | DATETIME | REPLACE | Last user access time| -| Cost | BIGINT | SUM | Total User Consumption| -| max dwell time | INT | MAX | Maximum user residence time| -| min dwell time | INT | MIN | User minimum residence time| - -The corresponding CREATE TABLE statement would be as follows (omitting the Partition and Distribution information): +| ColumnName | Type | AggregationType | Comment | +|-|--|-|-| +| userid | LARGEINT | | user id | +| date| DATE | | date of data filling | +| City| VARCHAR (20) | | User City | +| age | SMALLINT | | User age | +| sex | TINYINT | | User gender | +| Last_visit_date | DATETIME | REPLACE | Last user access time | +| Cost| BIGINT | SUM | Total User Consumption | +| max dwell time | INT | MAX | Maximum user residence time | +| min dwell time | INT | MIN | User minimum residence time | + +The corresponding to CREATE TABLE statement would be as follows (omitting the Partition and Distribution information): ``` -CREATE TABLE IF NOT EXISTS example_db.example_tbl +CREATE DATABASE IF NOT EXISTS example_db; + +CREATE TABLE IF NOT EXISTS example_db.example_tbl_agg1 ( `user_id` LARGEINT NOT NULL COMMENT "user id", `date` DATE NOT NULL COMMENT "data import time", @@ -102,21 +104,21 @@ If these aggregation methods cannot meet the requirements, you can choose to use Suppose that you have the following import data (raw data): -|user\_id|date|city|age|sex|last\_visit\_date|cost|max\_dwell\_time|min\_dwell\_time| -|---|---|---|---|---|---|---|---|---| -| 1 | 2017-10-01 | Beijing | 20 | 0 | 2017-10-01 06:00 | 20 | 10 | 10| -| 1 | 2017-10-01 | Beijing | 20 | 0 | 2017-10-01 07:00 | 15 | 2 | 2| -| 10001 | 2017-10-01 | Beijing | 30 | 1 | 2017-10-01 17:05:45 | 2 | 22 | 22| -| 10002 | 2017-10-02 | Shanghai | 20 | 1 | 2017-10-02 12:59:12 | 200 | 5 | 5| -| 10003 | 2017-10-02 | Guangzhou | 32 | 0 | 2017-10-02 11:20:00 | 30 | 11 | 11| -| 10004 | 2017-10-01 | Shenzhen | 35 | 0 | 2017-10-01 10:00:15 | 100 | 3 | 3| -| 10004 | 2017-10-03 | Shenzhen | 35 | 0 | 2017-10-03 10:20:22 | 11 | 6 | 6| +| user\_id | date | city | age | sex | last\_visit\_date | cost | max\_dwell\_time | min\_dwell\_time | +|--||---|-|-|-|--|--|--| +| 1| 2017-10-01 | Beijing | 20 | 0 | 2017-10-01 06:00| 20 | 10 | 10 | +| 1| 2017-10-01 | Beijing | 20 | 0 | 2017-10-01 07:00| 15 | 2| 2| +| 10001| 2017-10-01 | Beijing | 30 | 1 | 2017-10-01 17:05:45 | 2| 22 | 22 | +| 10002| 2017-10-02 | Shanghai | 20 | 1 | 2017-10-02 12:59:12 | 200 | 5| 5| +| 10003| 2017-10-02 | Guangzhou | 32 | 0 | 2017-10-02 11:20:00 | 30 | 11 | 11 | +| 10004| 2017-10-01 | Shenzhen | 35 | 0 | 2017-10-01 10:00:15 | 100 | 3| 3| +| 10004| 2017-10-03 | Shenzhen | 35 | 0 | 2017-10-03 10:20:22 | 11 | 6| 6| And you can import data wi
[doris] branch master updated: [Enhancement](log) Improve Safety and Robustness of Log4j Configuration (#24861)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 400b9f2f97b [Enhancement](log) Improve Safety and Robustness of Log4j
Configuration (#24861)
400b9f2f97b is described below
commit 400b9f2f97badd458c324dbc4aa4f57ff2219e4b
Author: zy-kkk
AuthorDate: Mon Oct 9 06:44:37 2023 -0500
[Enhancement](log) Improve Safety and Robustness of Log4j Configuration
(#24861)
---
.../java/org/apache/doris/common/Log4jConfig.java | 19 ++-
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
index 019886ec648..3c72048e00b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
@@ -29,6 +29,7 @@ import org.apache.logging.log4j.core.lookup.StrSubstitutor;
import java.io.ByteArrayInputStream;
import java.io.IOException;
+import java.nio.charset.StandardCharsets;
import java.util.Map;
//
@@ -228,12 +229,20 @@ public class Log4jConfig extends XmlConfiguration {
SpringLog4j2Config.writeSpringLogConf(customConfDir);
// new SimpleLog4jConfiguration with xmlConfTemplate
-ByteArrayInputStream bis = new
ByteArrayInputStream(newXmlConfTemplate.getBytes("UTF-8"));
-ConfigurationSource source = new ConfigurationSource(bis);
-Log4jConfig config = new Log4jConfig(source);
+if (newXmlConfTemplate == null || newXmlConfTemplate.isEmpty()) {
+throw new IOException("The configuration template is empty!");
+}
+
+Log4jConfig config;
+try (ByteArrayInputStream bis = new
ByteArrayInputStream(newXmlConfTemplate.getBytes(StandardCharsets.UTF_8))) {
+ConfigurationSource source = new ConfigurationSource(bis);
+config = new Log4jConfig(source);
-LoggerContext context = (LoggerContext) LogManager.getContext(false);
-context.start(config);
+LoggerContext context = (LoggerContext)
LogManager.getContext(LogManager.class.getClassLoader(), false);
+context.start(config);
+} catch (Exception e) {
+throw new IOException("Error occurred while configuring Log4j", e);
+}
}
public static String getLogXmlConfTemplate() {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (5f95e97c56 -> 643f7cad0e)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5f95e97c56 [fix](function) array distance should return null when result is nan (#25214) add 643f7cad0e [typo](docs) Delete wrong schema change memory parameters (#25234) No new revisions were added by this update. Summary of changes: docs/en/docs/admin-manual/config/be-config.md| 7 +-- docs/zh-CN/docs/admin-manual/config/be-config.md | 7 +-- 2 files changed, 2 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (643f7cad0e -> 691889419f)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 643f7cad0e [typo](docs) Delete wrong schema change memory parameters (#25234) add 691889419f [fix](dbt) fix dbt doris bug that appeared in annotated models and table column comment bug (#24986) No new revisions were added by this update. Summary of changes: .../dbt-doris/dbt/adapters/doris/__version__.py| 2 +- .../dbt/adapters/doris/doris_column_item.py| 54 ++ extension/dbt-doris/dbt/adapters/doris/impl.py | 16 ++- .../dbt-doris/dbt/include/doris/dbt_project.yml| 2 +- .../dbt/include/doris/macros/adapters/columns.sql | 13 -- .../materializations/table/create_table_as.sql | 4 +- extension/dbt-doris/setup.py | 2 +- 7 files changed, 74 insertions(+), 19 deletions(-) create mode 100644 extension/dbt-doris/dbt/adapters/doris/doris_column_item.py - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (8b56ca84c7 -> 67ddfb1abc)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8b56ca84c7 [fix](Nereids) support AnyDataType in function signature (#25173) add 67ddfb1abc [fix](httpserver) creating this cookie without the "secure" flag and enabling cross-origin resource safe (#25107) No new revisions were added by this update. Summary of changes: fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 7 +++ .../main/java/org/apache/doris/httpv2/config/WebConfigurer.java| 2 +- .../java/org/apache/doris/httpv2/controller/BaseController.java| 1 + 3 files changed, 9 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (a9b84ae6eed -> 193ef79df2e)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a9b84ae6eed [test](nereids)add more case in PushdownFilterThroughAggregationTest (#24927) add 193ef79df2e [chore](fix) Update .asf.yaml (#25300) No new revisions were added by this update. Summary of changes: .asf.yaml | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [typo](docs) add 'order by' when use 'limit m,n' (#24236)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 004d3264a6c [typo](docs) add 'order by' when use 'limit m,n' (#24236) 004d3264a6c is described below commit 004d3264a6cc03ee3863967e03e7ae4bf105ec1d Author: lsy3993 <110876560+lsy3...@users.noreply.github.com> AuthorDate: Wed Oct 11 16:15:33 2023 +0800 [typo](docs) add 'order by' when use 'limit m,n' (#24236) --- .../sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md | 2 +- .../sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md index 67b80fc91db..f3d5df71842 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md @@ -77,7 +77,7 @@ SELECT If you need to return the top N sorted results, you need to use the LIMIT clause; in order to limit memory usage, if the user does not specify the LIMIT clause, the first 65535 sorted results are returned by default. - 9. `Limit n`: limit the number of lines in the output result, `limit m,n` means output n records starting from the mth line. + 9. `Limit n`: limit the number of lines in the output result, `limit m,n` means output n records starting from the mth line.You should use `order by` before you use `limit m,n`, otherwise the data may be inconsistent each time it is executed. 10. The `Having` clause does not filter the row data in the table, but filters the results produced by the aggregate function. diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md index 262a1dd9222..7c11caf93b7 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/SELECT.md @@ -77,7 +77,7 @@ SELECT 如果需要返回前 N 个排序结果，需要使用 LIMIT 从句；为了限制内存的使用，如果用户没有指定 LIMIT 从句，则默认返回前 65535 个排序结果。 -9. `Limit n`: 限制输出结果中的行数，`limit m,n` 表示从第m行开始输出n条记录。 +9. `Limit n`: 限制输出结果中的行数，`limit m,n` 表示从第m行开始输出n条记录，**使用limit m,n的时候要加上order by才有意义，否则每次执行的数据可能会不一致** 10. `Having` 从句不是过滤表中的行数据，而是过滤聚合函数产出的结果。 - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (004d3264a6c -> df7724d6cb9)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 004d3264a6c [typo](docs) add 'order by' when use 'limit m,n' (#24236) add df7724d6cb9 [typo](docs)delete wrong description of from_unixtime (#23897) No new revisions were added by this update. Summary of changes: .../docs/sql-manual/sql-functions/date-time-functions/from-unixtime.md | 3 --- 1 file changed, 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (bdb64eab73b -> c6824ce1ae1)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from bdb64eab73b [feature](meta) queries as table valued function (#25052) (#25052) add c6824ce1ae1 [test](fix) unstable case test_jdbc_query_mysql (#25279) No new revisions were added by this update. Summary of changes: .../external_table_p0/jdbc/test_jdbc_query_mysql.out | 20 ++-- .../jdbc/test_jdbc_query_mysql.groovy| 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-flink-connector] branch master updated: [improve] add ckptid in label when 2pc false (#207)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 84b9bda [improve] add ckptid in label when 2pc false (#207)
84b9bda is described below
commit 84b9bdac2b1c6231796daf714030cfc86ff3bca5
Author: wudi <676366...@qq.com>
AuthorDate: Fri Oct 13 11:56:35 2023 +0800
[improve] add ckptid in label when 2pc false (#207)
Currently, when 2pc is closed, the label will only splice the uuid without
adding the ckid, which is inconvenient to locate the problem.
---
.../main/java/org/apache/doris/flink/sink/writer/LabelGenerator.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/LabelGenerator.java
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/LabelGenerator.java
index 35edae9..235b553 100644
---
a/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/LabelGenerator.java
+++
b/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/writer/LabelGenerator.java
@@ -31,7 +31,8 @@ public class LabelGenerator {
}
public String generateLabel(long chkId) {
-return enable2PC ? labelPrefix + "_" + chkId : labelPrefix + "_" +
UUID.randomUUID();
+String label = labelPrefix + "_" + chkId;
+return enable2PC ? label : label + "_" + UUID.randomUUID();
}
public String generateBatchLabel() {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [ecosystem](doc) mysql synchronization example add mysql-conf port (#24666)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 6298f903470 [ecosystem](doc) mysql synchronization example add mysql-conf port (#24666) 6298f903470 is described below commit 6298f9034700a0e3ac2e6e80631f258f1d7d42dc Author: 小可耐 <46134044+sdh...@users.noreply.github.com> AuthorDate: Fri Oct 13 14:36:26 2023 +0800 [ecosystem](doc) mysql synchronization example add mysql-conf port (#24666) --- docs/en/docs/ecosystem/flink-doris-connector.md| 1 + docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/en/docs/ecosystem/flink-doris-connector.md b/docs/en/docs/ecosystem/flink-doris-connector.md index 5de90e2fbe6..87b47a78d08 100644 --- a/docs/en/docs/ecosystem/flink-doris-connector.md +++ b/docs/en/docs/ecosystem/flink-doris-connector.md @@ -464,6 +464,7 @@ insert into doris_sink select id,name from cdc_mysql_source; mysql-sync-database\ --database test_db \ --mysql-conf hostname=127.0.0.1 \ + --mysql-conf port=3306 \ --mysql-conf username=root \ --mysql-conf password=123456 \ --mysql-conf database-name=mysql_db \ diff --git a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md index ad8c14eb575..f70aeb79ed5 100644 --- a/docs/zh-CN/docs/ecosystem/flink-doris-connector.md +++ b/docs/zh-CN/docs/ecosystem/flink-doris-connector.md @@ -466,6 +466,7 @@ insert into doris_sink select id,name from cdc_mysql_source; mysql-sync-database \ --database test_db \ --mysql-conf hostname=127.0.0.1 \ +--mysql-conf port=3306 \ --mysql-conf username=root \ --mysql-conf password=123456 \ --mysql-conf database-name=mysql_db \ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [chore](format) Refactor BaseTablet _full_name by using fmt replacing stringstream (#25400)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 789210bc384 [chore](format) Refactor BaseTablet _full_name by using
fmt replacing stringstream (#25400)
789210bc384 is described below
commit 789210bc38496c3c1379dee55661374ce531aae5
Author: Jack Drogon
AuthorDate: Fri Oct 13 03:59:03 2023 -0500
[chore](format) Refactor BaseTablet _full_name by using fmt replacing
stringstream (#25400)
Signed-off-by: Jack Drogon
---
be/src/olap/base_tablet.cpp | 7 ++-
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/be/src/olap/base_tablet.cpp b/be/src/olap/base_tablet.cpp
index b65d373449a..928333c2cba 100644
--- a/be/src/olap/base_tablet.cpp
+++ b/be/src/olap/base_tablet.cpp
@@ -20,7 +20,6 @@
#include
#include
-#include
#include
#include "gutil/strings/substitute.h"
@@ -42,10 +41,8 @@ BaseTablet::BaseTablet(TabletMetaSharedPtr tablet_meta,
DataDir* data_dir)
_schema =
TabletSchemaCache::instance()->insert(_tablet_meta->tablet_schema()->to_key());
_gen_tablet_path();
-std::stringstream ss;
-ss << _tablet_meta->tablet_id() << "." << _tablet_meta->schema_hash() <<
"."
- << _tablet_meta->tablet_uid().to_string();
-_full_name = ss.str();
+_full_name = fmt::format("{}.{}.{}", _tablet_meta->tablet_id(),
_tablet_meta->schema_hash(),
+ _tablet_meta->tablet_uid().to_string());
_metric_entity =
DorisMetrics::instance()->metric_registry()->register_entity(
strings::Substitute("Tablet.$0", tablet_id()),
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Imporve](UNIX_TIMESTAMP) UNIX_TIMESTAMP func support 'yyyy-MM-dd HH:mm:ss' format (#24561)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c9f5142420 [Imporve](UNIX_TIMESTAMP) UNIX_TIMESTAMP func support
'-MM-dd HH:mm:ss' format (#24561)
c9f5142420 is described below
commit c9f5142420eabf2ebda77a942cc9984484ddd982
Author: Liqf <109049295+lemonlit...@users.noreply.github.com>
AuthorDate: Tue Sep 19 18:41:59 2023 +0800
[Imporve](UNIX_TIMESTAMP) UNIX_TIMESTAMP func support '-MM-dd
HH:mm:ss' format (#24561)
UNIX_TIMESTAMP function data format parameter supports '-MM-dd HH:mm:ss'
The implementation is the same as the date_format function
before:
```sql
mysql> select UNIX_TIMESTAMP('2023-09-18 00:00:00','-MM-dd HH:mm:ss');
+--+
| unix_timestamp('2023-09-18 00:00:00', '-MM-dd HH:mm:ss') |
+--+
| NULL |
+--+
1 row in set (0.04 sec)
```
now:
```sql
mysql> select UNIX_TIMESTAMP('2023-09-18 00:00:00','-MM-dd HH:mm:ss');
++
| 1694966400 |
++
| 1694966400 |
++
1 row in set (0.01 sec)
```
---
.../java/org/apache/doris/analysis/FunctionCallExpr.java | 3 ++-
.../rules/expression/rules/SupportJavaDateFormatter.java | 14 ++
.../datetime_functions/test_date_function.out | 3 +++
.../datetime_functions/test_date_function.groovy | 9 +
4 files changed, 28 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index 6217eef61c..fa3786294b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -1581,7 +1581,8 @@ public class FunctionCallExpr extends Expr {
}
if (fnName.getFunction().equalsIgnoreCase("from_unixtime")
-|| fnName.getFunction().equalsIgnoreCase("date_format")) {
+|| fnName.getFunction().equalsIgnoreCase("date_format")
+|| fnName.getFunction().equalsIgnoreCase("unix_timestamp")) {
// if has only one child, it has default time format: -MM-dd
HH:mm:ss.SS
if (children.size() > 1) {
final StringLiteral fmtLiteral = (StringLiteral)
children.get(1);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SupportJavaDateFormatter.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SupportJavaDateFormatter.java
index bf3a698887..17f4b7d239 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SupportJavaDateFormatter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SupportJavaDateFormatter.java
@@ -22,6 +22,7 @@ import
org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.scalar.DateFormat;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.FromUnixtime;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.UnixTimestamp;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral;
@@ -59,6 +60,19 @@ public class SupportJavaDateFormatter extends
AbstractExpressionRewriteRule {
return fromUnixtime;
}
+@Override
+public Expression visitUnixTimestamp(UnixTimestamp unixTimestamp,
ExpressionRewriteContext context) {
+Expression expr = super.visitUnixTimestamp(unixTimestamp, context);
+if (!(expr instanceof UnixTimestamp)) {
+return expr;
+}
+unixTimestamp = (UnixTimestamp) expr;
+if (unixTimestamp.arity() > 1) {
+return translateJavaFormatter(unixTimestamp, 1);
+}
+return unixTimestamp;
+}
+
private Expression translateJavaFormatter(Expression function, int
formatterIndex) {
Expression formatterExpr = function.getArgument(formatterIndex);
Expression newFormatterExpr = translateJavaFormatter(formatterExpr);
diff --git
a/regression-test/data/query_p0/sql_functions/datetime_functions/test_date_function.out
b/regression-test/data/query_p0/sql_functions
[doris] branch master updated (1a553f7e14 -> 4f215a7dc3)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1a553f7e14 [Improve](start-shell)Optimize fe&be startup (#24556) add 4f215a7dc3 [Improve](Fe)Ensure that only one FE process uses the metedata file (#24442) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/DorisFE.java| 61 +- .../apache/doris/utframe/TestWithFeService.java| 6 +++ .../org/apache/doris/utframe/UtFrameUtils.java | 6 +++ 3 files changed, 72 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](jdbc catalog) Extend conjunctExprToString to Support both 'AND' and 'OR' with Optimized DateLiteral Handling (#24537)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 527b284e90 [improvement](jdbc catalog) Extend conjunctExprToString to
Support both 'AND' and 'OR' with Optimized DateLiteral Handling (#24537)
527b284e90 is described below
commit 527b284e90280a8b0ef37d6047f3f96c0b2c2ef5
Author: zy-kkk
AuthorDate: Tue Sep 19 23:11:44 2023 +0800
[improvement](jdbc catalog) Extend conjunctExprToString to Support both
'AND' and 'OR' with Optimized DateLiteral Handling (#24537)
---
.../doris/planner/external/jdbc/JdbcScanNode.java | 70 +-
.../jdbc/test_oracle_jdbc_catalog.out | 21 +++
.../jdbc/test_mysql_jdbc_catalog.groovy| 2 +-
.../jdbc/test_oracle_jdbc_catalog.groovy | 6 ++
4 files changed, 71 insertions(+), 28 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
index ccba1a165f..1fefd749d4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcScanNode.java
@@ -20,6 +20,7 @@ package org.apache.doris.planner.external.jdbc;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.BinaryPredicate;
import org.apache.doris.analysis.BoolLiteral;
+import org.apache.doris.analysis.CompoundPredicate;
import org.apache.doris.analysis.DateLiteral;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.ExprSubstitutionMap;
@@ -325,43 +326,58 @@ public class JdbcScanNode extends ExternalScanNode {
}
public static String conjunctExprToString(TOdbcTableType tableType, Expr
expr) {
-if (tableType.equals(TOdbcTableType.ORACLE) &&
expr.contains(DateLiteral.class)
-&& (expr instanceof BinaryPredicate)) {
-ArrayList children = expr.getChildren();
-// k1 OP '2022-12-10 20:55:59' changTo ---> k1 OP
to_date('{}','-mm-dd hh24:mi:ss')
-// oracle datetime push down is different:
https://github.com/apache/doris/discussions/15069
-if (children.get(1).isConstant() &&
(children.get(1).getType().equals(Type.DATETIME) || children
-.get(1).getType().equals(Type.DATETIMEV2))) {
-String filter = children.get(0).toMySql();
-filter += ((BinaryPredicate) expr).getOp().toString();
-filter += "to_date('" + children.get(1).getStringValue() +
"','-mm-dd hh24:mi:ss')";
-return filter;
+if (expr instanceof CompoundPredicate) {
+StringBuilder result = new StringBuilder();
+CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
+for (Expr child : compoundPredicate.getChildren()) {
+result.append(conjunctExprToString(tableType, child));
+result.append("
").append(compoundPredicate.getOp().toString()).append(" ");
}
+// Remove the last operator
+result.setLength(result.length() -
compoundPredicate.getOp().toString().length() - 2);
+return result.toString();
}
-if ((tableType.equals(TOdbcTableType.TRINO) ||
tableType.equals(TOdbcTableType.PRESTO))
-&& expr.contains(DateLiteral.class) && (expr instanceof
BinaryPredicate)) {
+
+if (expr.contains(DateLiteral.class) && expr instanceof
BinaryPredicate) {
ArrayList children = expr.getChildren();
-if (children.get(1).isConstant() &&
(children.get(1).getType().isDate()) || children
-.get(1).getType().isDateV2()) {
-String filter = children.get(0).toMySql();
-filter += ((BinaryPredicate) expr).getOp().toString();
-filter += "date '" + children.get(1).getStringValue() + "'";
-return filter;
-}
-if (children.get(1).isConstant() &&
(children.get(1).getType().isDatetime() || children
-.get(1).getType().isDatetimeV2())) {
-String filter = children.get(0).toMySql();
-filter += ((BinaryPredicate) expr).getOp().toString();
-filter += "timestamp '" + children.get(1).getStringValue() +
"'";
-return filter;
+String filter = children.get(0).toMySql();
+filter += " " + ((BinaryPredicate) expr).getOp().toString() + &
[doris] branch master updated (1405b7ca82 -> 5eb8fe3d6e)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 1405b7ca82 [improve](scan) support lower the thread priority of scan thread (#24526) add 5eb8fe3d6e [improvement](type) modify the inner type display of the Array/Map/Struct type (#24459) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/catalog/ArrayType.java| 2 +- .../src/main/java/org/apache/doris/catalog/MapType.java | 3 ++- .../src/main/java/org/apache/doris/catalog/StructField.java | 12 .../src/main/java/org/apache/doris/catalog/StructType.java | 6 +- .../data/schema_change_p0/test_alter_table_column.out| 6 +++--- 5 files changed, 23 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (4d2a92e47c -> a946f99b8c)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4d2a92e47c [community](collaborator) add collaborator wm1581066 for issue and pr label #24500 add a946f99b8c [Fix](regression-test) fix regression-test of export parquet file format (#24450) No new revisions were added by this update. Summary of changes: .../data/export_p0/test_export_parquet.out | 218 ++--- .../data/export_p0/test_export_view.out| 8 +- .../pipeline/p0/conf/regression-conf.groovy| 2 +- .../suites/export_p0/test_export_parquet.groovy| 144 +++--- .../suites/export_p0/test_export_view.groovy | 106 +- .../export/test_export_external_table.groovy | 4 +- 6 files changed, 200 insertions(+), 282 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (df66922bc0 -> aa9f2260ea)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from df66922bc0 [Chore](sonar)sonar (C++) configuration file name error (#24662) add aa9f2260ea [fix](multi-catalog)Es catalog needs to verify whether it is a valid configuration. (#24309) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/datasource/EsExternalCatalog.java | 5 + .../java/org/apache/doris/external/elasticsearch/EsRestClient.java | 5 + 2 files changed, 10 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (8c8c563c3c -> 2a260be10c)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8c8c563c3c [Chore](status) adjust some error status print log (#24660) add 2a260be10c [improvement](jdbc catalog) when `lower_case_table_names` of jdbc catalog properties is set to `true`, use the real table name to query the jdbc data source (#24520) No new revisions were added by this update. Summary of changes: .../docker-compose/oracle/init/03-create-table.sql | 9 ++- .../docker-compose/oracle/init/04-insert.sql | 6 ++ docs/en/docs/lakehouse/multi-catalog/jdbc.md | 32 ++--- docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md| 24 +-- .../java/org/apache/doris/catalog/JdbcTable.java | 57 +++- .../doris/catalog/external/JdbcExternalTable.java | 3 + .../doris/datasource/jdbc/client/JdbcClient.java | 76 ++ .../datasource/jdbc/client/JdbcOracleClient.java | 12 +--- .../doris/planner/external/jdbc/JdbcScanNode.java | 4 +- .../doris/planner/external/jdbc/JdbcTableSink.java | 2 +- .../jdbc/test_oracle_jdbc_catalog.out | 6 ++ .../jdbc/test_oracle_jdbc_catalog.groovy | 3 + 12 files changed, 177 insertions(+), 57 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
