This is an automated email from the ASF dual-hosted git repository. luzhijing pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push: new 46e2b4b591 [Improve]Add best practices for doris-kafka-connector (#724) 46e2b4b591 is described below commit 46e2b4b591e0125f063d3f8c5c46775caf97e830 Author: wudongliang <46414265+donglian...@users.noreply.github.com> AuthorDate: Thu Jun 20 19:00:52 2024 +0800 [Improve]Add best practices for doris-kafka-connector (#724) --- docs/ecosystem/doris-kafka-connector.md | 83 +++++++++++++++++++++- .../current/ecosystem/doris-kafka-connector.md | 80 +++++++++++++++++++++ .../version-2.0/ecosystem/doris-kafka-connector.md | 80 +++++++++++++++++++++ .../version-2.1/ecosystem/doris-kafka-connector.md | 80 +++++++++++++++++++++ .../version-2.0/ecosystem/doris-kafka-connector.md | 83 +++++++++++++++++++++- .../version-2.1/ecosystem/doris-kafka-connector.md | 83 +++++++++++++++++++++- 6 files changed, 486 insertions(+), 3 deletions(-) diff --git a/docs/ecosystem/doris-kafka-connector.md b/docs/ecosystem/doris-kafka-connector.md index 120f089eec..c3d254f797 100644 --- a/docs/ecosystem/doris-kafka-connector.md +++ b/docs/ecosystem/doris-kafka-connector.md @@ -246,4 +246,85 @@ Doris-kafka-connector uses logical or primitive type mapping to resolve the colu | io.debezium.time.MicroTimestamp | DATETIME | | io.debezium.time.NanoTimestamp | DATETIME | | io.debezium.time.ZonedTimestamp | DATETIME | -| io.debezium.data.VariableScaleDecimal | DOUBLE | \ No newline at end of file +| io.debezium.data.VariableScaleDecimal | DOUBLE | + + +## Best Practices +### Load Json serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### Load Avro serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### Load Protobuf serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector.md index 319c393356..7eb53a5215 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/ecosystem/doris-kafka-connector.md @@ -248,3 +248,83 @@ Doris-kafka-connector 使用逻辑或原始类型映射来解析列的数据类 | io.debezium.time.ZonedTimestamp | DATETIME | | io.debezium.data.VariableScaleDecimal | DOUBLE | + +## 最佳实践 +### 同步 Json 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### 同步 Avro 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### 同步 Protobuf 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/doris-kafka-connector.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/doris-kafka-connector.md index 319c393356..7eb53a5215 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/doris-kafka-connector.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/ecosystem/doris-kafka-connector.md @@ -248,3 +248,83 @@ Doris-kafka-connector 使用逻辑或原始类型映射来解析列的数据类 | io.debezium.time.ZonedTimestamp | DATETIME | | io.debezium.data.VariableScaleDecimal | DOUBLE | + +## 最佳实践 +### 同步 Json 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### 同步 Avro 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### 同步 Protobuf 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/doris-kafka-connector.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/doris-kafka-connector.md index 319c393356..7eb53a5215 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/doris-kafka-connector.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/ecosystem/doris-kafka-connector.md @@ -248,3 +248,83 @@ Doris-kafka-connector 使用逻辑或原始类型映射来解析列的数据类 | io.debezium.time.ZonedTimestamp | DATETIME | | io.debezium.data.VariableScaleDecimal | DOUBLE | + +## 最佳实践 +### 同步 Json 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### 同步 Avro 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### 同步 Protobuf 序列化数据 +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file diff --git a/versioned_docs/version-2.0/ecosystem/doris-kafka-connector.md b/versioned_docs/version-2.0/ecosystem/doris-kafka-connector.md index 120f089eec..c3d254f797 100644 --- a/versioned_docs/version-2.0/ecosystem/doris-kafka-connector.md +++ b/versioned_docs/version-2.0/ecosystem/doris-kafka-connector.md @@ -246,4 +246,85 @@ Doris-kafka-connector uses logical or primitive type mapping to resolve the colu | io.debezium.time.MicroTimestamp | DATETIME | | io.debezium.time.NanoTimestamp | DATETIME | | io.debezium.time.ZonedTimestamp | DATETIME | -| io.debezium.data.VariableScaleDecimal | DOUBLE | \ No newline at end of file +| io.debezium.data.VariableScaleDecimal | DOUBLE | + + +## Best Practices +### Load Json serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### Load Avro serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### Load Protobuf serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file diff --git a/versioned_docs/version-2.1/ecosystem/doris-kafka-connector.md b/versioned_docs/version-2.1/ecosystem/doris-kafka-connector.md index 120f089eec..c3d254f797 100644 --- a/versioned_docs/version-2.1/ecosystem/doris-kafka-connector.md +++ b/versioned_docs/version-2.1/ecosystem/doris-kafka-connector.md @@ -246,4 +246,85 @@ Doris-kafka-connector uses logical or primitive type mapping to resolve the colu | io.debezium.time.MicroTimestamp | DATETIME | | io.debezium.time.NanoTimestamp | DATETIME | | io.debezium.time.ZonedTimestamp | DATETIME | -| io.debezium.data.VariableScaleDecimal | DOUBLE | \ No newline at end of file +| io.debezium.data.VariableScaleDecimal | DOUBLE | + + +## Best Practices +### Load Json serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-json-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"json_topic", + "tasks.max":"10", + "doris.topic2table.map": "json_topic:json_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"org.apache.kafka.connect.json.JsonConverter", + "value.converter":"org.apache.kafka.connect.json.JsonConverter" + } +}' +``` + +### Load Avro serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-avro-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"avro_topic", + "tasks.max":"10", + "doris.topic2table.map": "avro_topic:avro_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.avro.AvroConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.avro.AvroConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` + +### Load Protobuf serialized data +``` +curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ + "name":"doris-protobuf-test", + "config":{ + "connector.class":"org.apache.doris.kafka.connector.DorisSinkConnector", + "topics":"proto_topic", + "tasks.max":"10", + "doris.topic2table.map": "proto_topic:proto_tab", + "buffer.count.records":"100000", + "buffer.flush.time":"120", + "buffer.size.bytes":"10000000", + "doris.urls":"127.0.0.1", + "doris.user":"root", + "doris.password":"", + "doris.http.port":"8030", + "doris.query.port":"9030", + "doris.database":"test", + "load.model":"stream_load", + "key.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "key.converter.schema.registry.url":"http://127.0.0.1:8081", + "value.converter":"io.confluent.connect.protobuf.ProtobufConverter", + "value.converter.schema.registry.url":"http://127.0.0.1:8081" + } +}' +``` \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org