[GitHub] [incubator-doris] yangzhg merged pull request #6933: [Doc] Use Flink CDC to realize real-time MySQL data into Apache Doris
yangzhg merged pull request #6933: URL: https://github.com/apache/incubator-doris/pull/6933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: [Doc] Use Flink CDC to realize real-time MySQL data into Apache Doris (#6933)
This is an automated email from the ASF dual-hosted git repository. yangzhg pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git The following commit(s) were added to refs/heads/master by this push: new 31f3eb4 [Doc] Use Flink CDC to realize real-time MySQL data into Apache Doris (#6933) 31f3eb4 is described below commit 31f3eb4a3cf9e5c8183c8fe8773ec9db41cccb35 Author: jiafeng.zhang AuthorDate: Sat Nov 6 16:18:19 2021 +0800 [Doc] Use Flink CDC to realize real-time MySQL data into Apache Doris (#6933) * Best Practices ，Use Flink CDC to realize real-time MySQL data into Apache Doris --- .../public/images/cdc/image-20211025162831632.png | Bin 0 -> 143949 bytes .../public/images/cdc/image-20211025165547903.png | Bin 0 -> 39487 bytes .../public/images/cdc/image-20211025170642628.png | Bin 0 -> 49165 bytes .../public/images/cdc/image-20211025182341086.png | Bin 0 -> 21677 bytes .../public/images/cdc/image-20211025182435827.png | Bin 0 -> 48534 bytes .../public/images/cdc/image-20211026095513892.png | Bin 0 -> 92955 bytes .../public/images/cdc/image-20211026100505972.png | Bin 0 -> 33173 bytes .../public/images/cdc/image-20211026100804091.png | Bin 0 -> 10829 bytes .../public/images/cdc/image-20211026100943474.png | Bin 0 -> 284282 bytes .../public/images/cdc/image-20211026101004547.png | Bin 0 -> 21854 bytes .../public/images/cdc/image-20211026101203629.png | Bin 0 -> 43195 bytes docs/.vuepress/sidebar/en.js | 3 +- docs/.vuepress/sidebar/zh-CN.js| 3 +- docs/en/best-practices/datax-doris-writer.md | 6 +- docs/en/best-practices/flink-cdc-to-doris.md | 367 + docs/zh-CN/best-practices/flink-cdc-to-doris.md| 342 +++ 16 files changed, 716 insertions(+), 5 deletions(-) diff --git a/docs/.vuepress/public/images/cdc/image-20211025162831632.png b/docs/.vuepress/public/images/cdc/image-20211025162831632.png new file mode 100644 index 000..3760970 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211025162831632.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211025165547903.png b/docs/.vuepress/public/images/cdc/image-20211025165547903.png new file mode 100644 index 000..0e61dda Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211025165547903.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211025170642628.png b/docs/.vuepress/public/images/cdc/image-20211025170642628.png new file mode 100644 index 000..5381d95 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211025170642628.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211025182341086.png b/docs/.vuepress/public/images/cdc/image-20211025182341086.png new file mode 100644 index 000..800c63a Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211025182341086.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211025182435827.png b/docs/.vuepress/public/images/cdc/image-20211025182435827.png new file mode 100644 index 000..fa97de2 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211025182435827.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026095513892.png b/docs/.vuepress/public/images/cdc/image-20211026095513892.png new file mode 100644 index 000..8f139bd Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026095513892.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026100505972.png b/docs/.vuepress/public/images/cdc/image-20211026100505972.png new file mode 100644 index 000..e92f187 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026100505972.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026100804091.png b/docs/.vuepress/public/images/cdc/image-20211026100804091.png new file mode 100644 index 000..7c1dd62 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026100804091.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026100943474.png b/docs/.vuepress/public/images/cdc/image-20211026100943474.png new file mode 100644 index 000..bfc3d7b Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026100943474.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026101004547.png b/docs/.vuepress/public/images/cdc/image-20211026101004547.png new file mode 100644 index 000..e458645 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026101004547.png differ diff --git a/docs/.vuepress/public/images/cdc/image-20211026101203629.png b/docs/.vuepress/public/images/cdc/image-20211026101203629.png new file mode 100644 index 000..fcb5ea4 Binary files /dev/null and b/docs/.vuepress/public/images/cdc/image-20211026101203629.png differ diff --git a/docs/.vuepress
[GitHub] [incubator-doris] JNSimba commented on pull request #7010: [HTTP][API] add backend by rest api
JNSimba commented on pull request #7010: URL: https://github.com/apache/incubator-doris/pull/7010#issuecomment-962417750 > > > > Because there is no be, there is no way to add be through the jdbc connection > > > > > > > > > I think this should be resolved by this PR #6150 ? > > > > > > In which version is this pr? Does that mean that this previous version was not available for doris manager? Or do we qualify the supported version? > > #6150 is included in 0.15, you can try it. Thank you very much, I test with version 0.15.1-rc03 just now, it's works to use jdbc to execute alter system add backend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] renyuankun opened a new issue #7027: [Bug]
renyuankun opened a new issue #7027: URL: https://github.com/apache/incubator-doris/issues/7027 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 0.14.12.4 ### What's Wrong? According to the official document`Colocation Join` function description，Created a table `test_table` Table building statement： ``` CREATE TABLE `test_table` ( `id` bigint(20) NULL COMMENT "Continuously increase from 1 without repeating", `user_id` bigint(20) NULL COMMENT "Will repeat", INDEX id_index (`id`) COMMENT '', INDEX user_id_index (`user_id`) COMMENT '' ) ENGINE=OLAP COMMENT "" DISTRIBUTED BY HASH(`user_id`) BUCKETS 10 PROPERTIES ( "replication_num" = "3", "colocate_with" = "group_test", "in_memory" = "false", "storage_format" = "V2" ); ``` After the table is created, do the following things： 1. 10 million pieces of data with continuous self-increment and non-repetitive IDs are inserted into the table 2. Execute statement： ``` select * from test_table where id BETWEEN 1 and 100 ``` The result of the query is only 88, and the user_id of these 88 data is the same（This user_id only has 88 pieces of data） 3. Change the sentence to： ``` select * from test_table where id BETWEEN 89 and 100 ``` At this time, the remaining 12 pieces of data have been found out, and the user_id of these 12 pieces of data has another value. If the first step is to limit the number of inserted data, for example, limit to 10,000, then there will be no problems when you perform step 2。Or when creating the table, change `DISTRIBUTED BY HASH(`user_id`)` to `DISTRIBUTED BY HASH(`id`)` it is normal Remove the `"colocate_with" = "group_test",` in the table creation statement, no effect ### What You Expected? Performing the single-table query in step two should return 100 data instead of 88 ### How to Reproduce? _No response_ ### Anything Else? The explain results are as follows： ``` PLAN FRAGMENT 0 OUTPUT EXPRS:`default_cluster:test_data.test_table`.`id` | `default_cluster:test_data.test_table`.`user_id` PARTITION: UNPARTITIONED RESULT SINK 1:EXCHANGE PLAN FRAGMENT 1 OUTPUT EXPRS: PARTITION: HASH_PARTITIONED: `default_cluster:test_data`.`test_table`.`user_id` STREAM DATA SINK EXCHANGE ID: 01 UNPARTITIONED 0:OlapScanNode TABLE: test_table PREAGGREGATION: ON PREDICATES: `id` >= 1, `id` <= 100 partitions=1/1 rollup: test_table tabletRatio=10/10 tabletList=70118,70122,70126,70130,70134,70138,70142,70146,70150,70154 cardinality=0 avgRowSize=16.0 numNodes=1 ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] KKtwo opened a new issue #7028: [Bug] Memory consumption of JDBC MySQL Driver
KKtwo opened a new issue #7028: URL: https://github.com/apache/incubator-doris/issues/7028 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 0.14.0 ### What's Wrong? Describe the bug My project deployed in docker has been restarted When I need to load lots of data (3000W rows) in MySQL table use JDBC Driver.  As far as I know, this problem was fixed in version 0.14.0 #5323 ### What You Expected? fixed ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] KKtwo commented on issue #7028: [Bug] Memory consumption of JDBC MySQL Driver
KKtwo commented on issue #7028: URL: https://github.com/apache/incubator-doris/issues/7028#issuecomment-962454399 modify -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] KKtwo removed a comment on issue #7028: [Bug] Memory consumption of JDBC MySQL Driver
KKtwo removed a comment on issue #7028: URL: https://github.com/apache/incubator-doris/issues/7028#issuecomment-962454399 modify -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WindyGao commented on issue #6716: palo_be: rdkafka_broker.c:5321: rd_kafka_broker_destroy_final: Assertion `thrd_is_current(rkb->rkb_thread)' failed.
WindyGao commented on issue #6716: URL: https://github.com/apache/incubator-doris/issues/6716#issuecomment-962550481 we also encountered this issue be was crashed by following Error msg ``` start time: Wed Oct 13 14:38:37 CST 2021 start time: Fri Oct 29 14:49:34 CST 2021 palo_be: rdkafka_broker.c:5321: rd_kafka_broker_destroy_final: Assertion `thrd_is_current(rkb->rkb_thread)' failed. *** Aborted at 1636152931 (unix time) try "date -d @1636152931" if you are using GNU date *** PC: @ 0x7f113a9452c7 __GI_raise *** SIGABRT (@0xd6b000210bd) received by PID 135357 (TID 0x7f0f0080f700) from PID 135357; stack trace: *** @ 0x1fd12e2 google::(anonymous namespace)::FailureSignalHandler() @ 0x7f113afed5d0 (unknown) @ 0x7f113a9452c7 __GI_raise @ 0x7f113a9469b8 __GI_abort @ 0x7f113a93e0e6 __assert_fail_base @ 0x7f113a93e192 __GI___assert_fail @ 0x1d7c767 rd_kafka_broker_destroy_final @ 0x1dfccb8 rd_kafka_metadata_refresh_topics @ 0x1dfd04a rd_kafka_metadata_refresh_known_topics @ 0x1d774c2 rd_kafka_broker_fail @ 0x1d87bd4 rd_kafka_broker_op_serve @ 0x1d89463 rd_kafka_broker_ops_io_serve @ 0x1d89968 rd_kafka_broker_consumer_serve @ 0x1d8b485 rd_kafka_broker_thread_main @ 0x1df43a7 _thrd_wrapper_function @ 0x7f113afe5dd5 start_thread @ 0x7f113aa0d02d __clone ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
