This is an automated email from the ASF dual-hosted git repository. dockerzhang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push: new a501ded234c [INLONG-990][Doc] Add quick start guide for offline data sync from Pulsar to MySQL (#991) a501ded234c is described below commit a501ded234c7700d7b9358fa686c668a383c300d Author: AloysZhang <lofterzh...@gmail.com> AuthorDate: Fri Sep 27 22:50:05 2024 +0800 [INLONG-990][Doc] Add quick start guide for offline data sync from Pulsar to MySQL (#991) Co-authored-by: Aloys Zhang <aloyszh...@apche.org> --- docs/quick_start/offline_data_sync/_category_.json | 4 + .../offline_data_sync/img/pulsar_mysql/approve.png | Bin 0 -> 37521 bytes .../img/pulsar_mysql/cluster_tag.png | Bin 0 -> 28397 bytes .../img/pulsar_mysql/conventional_schedule.png | Bin 0 -> 55128 bytes .../img/pulsar_mysql/create_offline_group.png | Bin 0 -> 44131 bytes .../img/pulsar_mysql/create_pulsar_cluster.png | Bin 0 -> 36568 bytes .../img/pulsar_mysql/cron_schedule.png | Bin 0 -> 21244 bytes .../img/pulsar_mysql/flink_batch_job.png | Bin 0 -> 215724 bytes .../img/pulsar_mysql/mysql_sink.png | Bin 0 -> 476668 bytes .../offline_data_sync/img/pulsar_mysql/sink.png | Bin 0 -> 38261 bytes .../img/pulsar_mysql/sink_fields.png | Bin 0 -> 38312 bytes .../offline_data_sync/img/pulsar_mysql/source.png | Bin 0 -> 51243 bytes .../img/pulsar_mysql/source_fields.png | Bin 0 -> 29766 bytes .../offline_data_sync/pulsar_mysql_example.md | 151 +++++++++++++++++++++ .../offline_data_sync/img/pulsar_mysql/approve.png | Bin 0 -> 24000 bytes .../img/pulsar_mysql/cluster_tag.png | Bin 0 -> 14019 bytes .../img/pulsar_mysql/create_offline_group.png | Bin 0 -> 44936 bytes .../img/pulsar_mysql/cron_schedule.png | Bin 0 -> 16984 bytes .../img/pulsar_mysql/flink_batch_job.png | Bin 0 -> 215724 bytes .../offline_data_sync/img/pulsar_mysql/group.png | Bin 0 -> 37242 bytes .../img/pulsar_mysql/mysql_sink.png | Bin 0 -> 539660 bytes .../img/pulsar_mysql/normal_schedule.png | Bin 0 -> 70008 bytes .../offline_data_sync/img/pulsar_mysql/pulsar.png | Bin 0 -> 28881 bytes .../offline_data_sync/img/pulsar_mysql/sink.png | Bin 0 -> 37364 bytes .../img/pulsar_mysql/sink_field.png | Bin 0 -> 36128 bytes .../offline_data_sync/img/pulsar_mysql/source.png | Bin 0 -> 49903 bytes .../img/pulsar_mysql/source_field.png | Bin 0 -> 29224 bytes .../offline_data_sync/pulsar_mysql_example.md | 149 ++++++++++++++++++++ 28 files changed, 304 insertions(+) diff --git a/docs/quick_start/offline_data_sync/_category_.json b/docs/quick_start/offline_data_sync/_category_.json new file mode 100644 index 00000000000..5afffc682cc --- /dev/null +++ b/docs/quick_start/offline_data_sync/_category_.json @@ -0,0 +1,4 @@ +{ + "label": "Data Offline Synchronization", + "position": 4 +} \ No newline at end of file diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/approve.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/approve.png new file mode 100644 index 00000000000..ea066ee6928 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/approve.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png new file mode 100644 index 00000000000..bc1fc8a16d3 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/conventional_schedule.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/conventional_schedule.png new file mode 100644 index 00000000000..8ed5b1a421e Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/conventional_schedule.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png new file mode 100644 index 00000000000..7ea229ca343 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_pulsar_cluster.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_pulsar_cluster.png new file mode 100644 index 00000000000..1cc548f34a4 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/create_pulsar_cluster.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png new file mode 100644 index 00000000000..9728d4b7e98 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png new file mode 100644 index 00000000000..cee8337048f Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png new file mode 100644 index 00000000000..04f85daf284 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink.png new file mode 100644 index 00000000000..62062b12751 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink_fields.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink_fields.png new file mode 100644 index 00000000000..ccd89b6d76f Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/sink_fields.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/source.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/source.png new file mode 100644 index 00000000000..4c2d0eb9908 Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/source.png differ diff --git a/docs/quick_start/offline_data_sync/img/pulsar_mysql/source_fields.png b/docs/quick_start/offline_data_sync/img/pulsar_mysql/source_fields.png new file mode 100644 index 00000000000..409204ef9bf Binary files /dev/null and b/docs/quick_start/offline_data_sync/img/pulsar_mysql/source_fields.png differ diff --git a/docs/quick_start/offline_data_sync/pulsar_mysql_example.md b/docs/quick_start/offline_data_sync/pulsar_mysql_example.md new file mode 100644 index 00000000000..c215543c456 --- /dev/null +++ b/docs/quick_start/offline_data_sync/pulsar_mysql_example.md @@ -0,0 +1,151 @@ +--- +title: Pulsar to MySQL Example +sidebar_position: 1 +--- + +In the following content, we will introduce how to use Apache InLong to create offline data synchronization from Pulsar to MySQL through a complete example. + +## Deployment +### Install InLong + +Before we begin, we need to install InLong. Here we provide two ways: +- [Docker Deployment](deployment/docker.md) (Recommended) +- [Bare Metal Deployment](deployment/bare_metal.md) + +### Add Connectors + +Download the [connectors](https://inlong.apache.org/downloads/) corresponding to Flink version, and after decompression, place `sort-connector-jdbc-[version]-SNAPSHOT.jar` in `/inlong-sort/connectors/` directory. +> Currently, Apache InLong's offline data synchronization capability only supports Flink-1.18, so please download the 1.18 version of connectors. + +### Install Pulsar +First, download the binary release package of Pulsar (the version can be modified as needed): + +```shell + wget https://archive.apache.org/dist/pulsar/pulsar-3.3.1/apache-pulsar-3.3.1-bin.tar.gz +``` +Then, unzip it: +```shell + tar xvfz apache-pulsar-3.3.1-bin.tar.gz +``` +Navigate to the Pulsar directory: +```shell + cd apache-pulsar-3.3.1 +``` +Quickly start a Pulsar cluster in Standalone mode: +```shell + sh bin/pulsar-daemon start standalone +``` +After starting, the Pulsar cluster will have: +- `adminUrl` as http://localhost:8080 +- `serviceUrl` as pulsar://localhost:6650 + +You can manage the cluster through `adminUrl` and produce and consume data via `serviceUrl`. + +## Cluster Initialize +When all containers are successfully started, you can access the InLong dashboard address http://localhost, and use the following default account to log in. +``` +User: admin +Password: inlong +``` + +### Create Cluster Tag +Click [Clusters] -> [ClusterTags] -> [Create] on the page to specify the cluster label name and person in charge. + + + +**caution: `default_cluster` is the default ClusterTags for each component. If you decide to use a different name, make sure to update the corresponding tag configuration accordingly.** + +### Register Pulsar Cluster + + + +**You can refer to the screenshot information to fill in details such as cluster name, associated tag, and Pulsar cluster address.** + +## Task Creation +### Create Data Stream Group +Click on [Synchronization]→[Create], fill in the Group ID, and ensure the [Sync Type] is checked as "Offline". + + + + +### Configuration Scheduling Rules +After selecting "offline" for [Sync Type], you can configure the [Scheduling Rules] for offline tasks. Currently, two types are supported: Conventional and Crontab. + +Conventional Scheduling Configuration requires the following parameters: +- Scheduling Unit: Supports minutes, hours, days, months, years, and single execution (single execution means it will run only once). +- Scheduling Interval: Indicates the time interval between two task schedules. +- Delay Time: Indicates the delay time for task startup. +- Valid Time: Includes start time and end time; the scheduled task will only execute within this time range. + + + +Crontab Scheduling requires the following parameters: +- Valid Time: Includes start time and end time; the scheduled task will only execute within this time range. +- Crontab Expression: Indicates the task cycle, e.g., `0 */5 * * * ?` + + + +### Create Data Source +In the data source section, click [Create] → [Pulsar], and configure the data source name, Pulsar tenant, namespace, topic, admin URL, service URL, data format, and other parameters. + + + +Note: The Pulsar topic needs to be created in the Pulsar cluster in advance (or enable the automatic topic creation feature in the Pulsar cluster). + +### Create Data Sink + +Create the target MySQL table; the example SQL is as follows: + +```sql +CREATE TABLE sink_table ( + id INT AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(255) NOT NULL, + create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +``` + +In the data sink section, click [Create] → [MySQL], and configure the data sink name, database name, and table name (test.source_table), among other information. + + + +### Configure source and sink fields + +Configure schema mapping information in the [Source fields] and [Sink fields] sections, and click [Submit] for approval. + + + + + +### Approval data flow + +On the page, click [Approval] -> [My Approvals] -> [Approve] → [OK]. + + + +Return to the [Synchronization] page and wait for the task configuration to succeed. Once configured successfully, the Manager will periodically submit Flink Batch Jobs to the Flink cluster. + + + +## Test Data +### Sending Data + +Use the Pulsar SDK to produce data into the Pulsar topic. An example is as follows: +```java + // Create Pulsar client and producer + PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(); + Producer<byte[]> producer = pulsarClient.newProducer().topic("public/default/test").create(); + + // Send messages + for (int i = 0; i < 10000; i++) { + // Field separator is | + String msgStr = i + "|msg-" + i; + MessageId msgId = producer.send(msgStr.getBytes(StandardCharsets.UTF_8)); + System.out.println("Send msg : " + msgStr + " with msgId: " + msgId); + } +``` + +### Data Validation + +Then enter MySQL to check the data in the table: + + diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/approve.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/approve.png new file mode 100644 index 00000000000..c1376e2f274 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/approve.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png new file mode 100644 index 00000000000..aa4b45fdc89 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cluster_tag.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png new file mode 100644 index 00000000000..5c56d74737d Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/create_offline_group.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png new file mode 100644 index 00000000000..87fe37a9519 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/cron_schedule.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png new file mode 100644 index 00000000000..cee8337048f Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/flink_batch_job.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/group.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/group.png new file mode 100644 index 00000000000..b4bd8ea56a3 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/group.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png new file mode 100644 index 00000000000..12c871375bd Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/mysql_sink.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/normal_schedule.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/normal_schedule.png new file mode 100644 index 00000000000..248412606ea Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/normal_schedule.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/pulsar.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/pulsar.png new file mode 100644 index 00000000000..fb75c178aa3 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/pulsar.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink.png new file mode 100644 index 00000000000..07650ac5abb Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink_field.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink_field.png new file mode 100644 index 00000000000..92fc35a1e39 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/sink_field.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source.png new file mode 100644 index 00000000000..7dffd59f6f3 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source_field.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source_field.png new file mode 100644 index 00000000000..a15481de839 Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/img/pulsar_mysql/source_field.png differ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/pulsar_mysql_example.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/pulsar_mysql_example.md new file mode 100644 index 00000000000..5c674b45990 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/quick_start/offline_data_sync/pulsar_mysql_example.md @@ -0,0 +1,149 @@ +--- +title: Pulsar 到 MySQL 示例 +sidebar_position: 1 +--- + +在下面的内容中,我们将通过一个完整的示例介绍如何使用 Apache InLong 创建 Pulsar -> MySQL 的离线数据同步。 + +## 环境部署 +### 安装 InLong + +在开始之前,我们需要安装 InLong 的全部组件,这里提供两种方式: +- [Docker 部署](deployment/docker.md)(推荐) +- [Bare Metal 部署](deployment/bare_metal.md) + +### 添加 Connectors + +下载与 Flink 版本对应的 [connectors](https://inlong.apache.org/zh-CN/downloads),解压后将 `sort-connector-jdbc-[version]-SNAPSHOT.jar` 放在 `/inlong-sort/connectors/` 目录下。 +> 当前 Apache InLong 的离线数据同步能力只支持 Flink-1.18 版本,所以请下载 1.18 版本的 connectors。 + +### 安装 Pulsar + +首先下载 Pulsar 的二进制发布包(版本可以根据需要修改): +```shell + wget https://archive.apache.org/dist/pulsar/pulsar-3.3.1/apache-pulsar-3.3.1-bin.tar.gz +``` +然后,解压: +```shell + tar xvfz apache-pulsar-3.3.1-bin.tar.gz +``` +进入 Pulsar 目录: +```shell + cd apache-pulsar-3.3.1 +``` +通过 Standalone 的方式快速启动一个 Pulsar 集群: +```shell + sh bin/pulsar-daemon start standalone +``` +启动之后的 Pulsar 集群: +- `adminUrl` 为 `http://localhost:8080` +- `serviceUrl` 为 `pulsar://localhost:6650` + +可以通过 `adminUrl` 来管理集群,通过 `serviceUrl` 进行数据的生产和消费。 + +## 集群初始化 +InLong 服务启动后,可以访问 InLong Dashboard 地址 http://localhost,并使用以下默认账号登录: +``` +User: admin +Password: inlong +``` +### 创建集群标签 +页面点击 【集群管理】→【标签管理】→【新建】 + + + +**注意:default_cluster 是各个组件默认的集群标签,如果使用其它名称,确认对应标签配置已修改。** + +### 注册 Pulsar 集群 + + + +**可以参考截图信息填写,包括集群名称、所属标签、Pulsar 集群地址等。** + +## 任务创建 +### 新建数据流组 +页面点击【数据同步】→【新建数据同步】,填写 数据流组 ID,注意同步类型勾选为“离线”。 + + + +### 配置调度规则 +在同步类型勾选为“离线”之后,就可以配置离线任务的调度规则,目前支持两种:常规和 crontab。 + +常规调度配置需要设置以下参数: +- 调度单位:支持分钟、小时、天、月、年以及单次,单次表示只执行一次 +- 调度周期:表示两次任务调度之间的时间间隔 +- 延迟时间:表示任务启动的延迟时间 +- 有效时间:包括起始时间和结束时间,调度任务只会在这个时间范围内执行 + + + +crontab调度需要设置以下参数: +- 有效时间:包括起始时间和结束时间,调度任务只会在这个时间范围内执行 +- crontab 表达式:表示任务的周期,比如 0 */5 * * * ? + + + +### 新建数据源 + +数据来源中 点击 【新建】→【Pulsar】,配置数据源名称、Pulsar tenant、namespace、topic、admin url、service url、数据格式等参数。 + + + +注:Pulsar 的 topic 需要预先在 Pulsar 集群创建(或者在 Pulsar 集群开启自动创建 topic 功能)。 + +### 新建数据目标 + +创建目标 MySQL 表,示例 SQL 如下: +```sql +CREATE TABLE sink_table ( + id INT AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(255) NOT NULL, + create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +``` + +数据目标中 点击 【新建】→【MySQL】,配置数据源名称、库名和表名(test.source_table)等信息信息等。 + + + +### 配置字段信息 + +分别在 【源字段】 和 【目标字段】中配置 Schema 映射信息,完成后点击 【提交审批】 + + + + + +### 审批数据流 + +页面点击【审批管理】->【我的审批】->【详情】->【通过】 + + + +返回 【数据同步】页面,等待任务配置成功,配置成功后,Manager 会周期提交 Flink Batch Job 到 Flink 集群。 + + + +## 测试数据 +### 发送数据 + +通过 Pulsar SDK 生产数据写入的 Pulsar topic 中,示例如下: +```java + // 创建 pulsar client 和 producer + PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build(); + Producer<byte[]> producer = pulsarClient.newProducer().topic("public/default/test").create(); + + // 发送消息 + for (int i = 0; i < 10000; i++) { + // 字段分隔符为 | + String msgStr = i + "|msg-" + i; + MessageId msgId = producer.send(msgStr.getBytes(StandardCharsets.UTF_8)); + System.out.println("Send msg : " + msgStr + " with msgId: " + msgId); + } +``` + +### 数据验证 + +然后进入 Mysql,查看库表数据,可以看到数据已经同步到 MySQL 中。 + +