saLeox commented on code in PR #287: URL: https://github.com/apache/incubator-streampark-website/pull/287#discussion_r1389567056
########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -1,139 +1,139 @@ --- slug: streampark-usercase-bondex-with-paimon -title: 海程邦达基于 Apache Paimon + StreamPark 的流式数仓实践 -tags: [StreamPark, 生产实践, paimon, streaming-warehouse] +title: Based on Apache Paimon + StreamPark's Streaming Data Warehouse Practice by Bondex +tags: [StreamPark, Production Practice, paimon, streaming-warehouse] ---  -**导读：**本文主要介绍作为供应链物流服务商海程邦达在数字化转型过程中采用 Paimon + StreamPark 平台实现流式数仓的落地方案。我们以 Apache StreamPark 流批一体平台提供了一个易于上手的生产操作手册，以帮助用户提交 Flink 任务并迅速掌握 Paimon 的使用方法。 +**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. -- 公司业务情况介绍 -- 大数据技术痛点以及选型 -- 生产实践 -- 问题排查分析 -- 未来规划 +- Introduction to Company Business +- Pain Points in Big Data Technology and Selection Review Comment: Pain Points and Selection in Big Data Technology ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -182,24 +182,24 @@ export PATH=$PATH:$FLINK_HOME/bin source /etc/profile ``` -在 StreamPark 添加 Flink conf: +In StreamPark, add Flink configuration: Review Comment: In StreamPark, add Flink conf: ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -234,127 +234,127 @@ docker tag flink-table-store:v1.16.0 registry-vpc.cn-zhangjiakou.aliyuncs.com/xx docker push registry-vpc.cn-zhangjiakou.aliyuncs.com/xxxxx/flink-table-store:v1.16.0 ``` -接下来准备 Paimon jar 包，可以在 Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon) 下载对应版本，需要注意的是要和 flink 大版本保持一致 +Next, prepare the Paimon jar package. You can download the corresponding version from the Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon). It's important to note that it should be consistent with the major version of Flink. -### **使用 StreamPark 管理作业** +### **Managing Jobs with StreamPark** -**前提条件：** +**Prerequisites:** -- Kubernetes 客户端连接配置 -- Kubernetes RBAC 配置 -- 容器镜像仓库配置 （案例中使用的是阿里云镜像免费版） -- 创建挂载 checkpoint/savepoint 的 pvc 资源 +- Kubernetes client connection configuration +- Kubernetes RBAC configuration +- Container image repository configuration (the free version of Alibaba Cloud image is used in this case) +- Create a PVC resource to mount checkpoints/savepoints -**Kubernetes 客户端连接配置：** +**Kubernetes Client Connection Configuration:** -将 k8s master节点~/.kube/config 配置直接拷贝到 StreamPark 服务器的目录，之后在 StreamPark 服务器执行以下命令显示 k8s 集群 running 代表权限和网络验证成功。 +Copy the k8s master node's `~/.kube/config` configuration directly to the directory on the StreamPark server, then execute the following command on the StreamPark server to display the k8s cluster as running, which indicates successful permission and network verification. ```shell kubectl cluster-info ``` -Kubernetes RBAC 配置,创建 streamx 命名空间: +Kubernetes RBAC Configuration, create the streampark namespace: ```shell -kubectl create ns streamx +kubectl create ns streampark ``` -使用 default 账户创建 clusterrolebinding 资源: +Use the default account to create the clusterrolebinding resource: ```shell -kubectl create secret docker-registry streamparksecret ---docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com ---docker-username=xxxxxx +kubectl create secret docker-registry streamparksecret +--docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com +--docker-username=xxxxxx --docker-password=xxxxxx -n streamx``` ``` -**容器镜像仓库配置:** +**Container Image Registry Configuration:** -案例中使用阿里云容器镜像服务ACR，也可以使用自建镜像服务harbor代替。 +In this case, Alibaba Cloud's Container Registry Service (ACR) is used, but you can also substitute it with a self-hosted image service such as Harbor. -创建命名空间 StreamPark (安全设置需要设置为私有) +Create a namespace named StreamPark (set the security setting to private).  -在 StreamPark 配置镜像仓库，任务构建镜像会推送到镜像仓库 +Configure the image repository in StreamPark; task build images will be pushed to the repository. Review Comment: build task images will be pushed to the repository. ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -234,127 +234,127 @@ docker tag flink-table-store:v1.16.0 registry-vpc.cn-zhangjiakou.aliyuncs.com/xx docker push registry-vpc.cn-zhangjiakou.aliyuncs.com/xxxxx/flink-table-store:v1.16.0 ``` -接下来准备 Paimon jar 包，可以在 Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon) 下载对应版本，需要注意的是要和 flink 大版本保持一致 +Next, prepare the Paimon jar package. You can download the corresponding version from the Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon). It's important to note that it should be consistent with the major version of Flink. -### **使用 StreamPark 管理作业** +### **Managing Jobs with StreamPark** -**前提条件：** +**Prerequisites:** -- Kubernetes 客户端连接配置 -- Kubernetes RBAC 配置 -- 容器镜像仓库配置 （案例中使用的是阿里云镜像免费版） -- 创建挂载 checkpoint/savepoint 的 pvc 资源 +- Kubernetes client connection configuration +- Kubernetes RBAC configuration +- Container image repository configuration (the free version of Alibaba Cloud image is used in this case) +- Create a PVC resource to mount checkpoints/savepoints -**Kubernetes 客户端连接配置：** +**Kubernetes Client Connection Configuration:** -将 k8s master节点~/.kube/config 配置直接拷贝到 StreamPark 服务器的目录，之后在 StreamPark 服务器执行以下命令显示 k8s 集群 running 代表权限和网络验证成功。 +Copy the k8s master node's `~/.kube/config` configuration directly to the directory on the StreamPark server, then execute the following command on the StreamPark server to display the k8s cluster as running, which indicates successful permission and network verification. ```shell kubectl cluster-info ``` -Kubernetes RBAC 配置,创建 streamx 命名空间: +Kubernetes RBAC Configuration, create the streampark namespace: ```shell -kubectl create ns streamx +kubectl create ns streampark ``` -使用 default 账户创建 clusterrolebinding 资源: +Use the default account to create the clusterrolebinding resource: ```shell -kubectl create secret docker-registry streamparksecret ---docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com ---docker-username=xxxxxx +kubectl create secret docker-registry streamparksecret +--docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com +--docker-username=xxxxxx --docker-password=xxxxxx -n streamx``` ``` -**容器镜像仓库配置:** +**Container Image Registry Configuration:** -案例中使用阿里云容器镜像服务ACR，也可以使用自建镜像服务harbor代替。 +In this case, Alibaba Cloud's Container Registry Service (ACR) is used, but you can also substitute it with a self-hosted image service such as Harbor. -创建命名空间 StreamPark (安全设置需要设置为私有) +Create a namespace named StreamPark (set the security setting to private).  -在 StreamPark 配置镜像仓库，任务构建镜像会推送到镜像仓库 +Configure the image repository in StreamPark; task build images will be pushed to the repository.  -创建 k8s secret 密钥用来拉取 ACR 中的镜像 streamparksecret 为密钥名称 自定义 +Create a k8s secret key to pull images from ACR; streamparksecret is the name of the secret, customizable. ```shell -kubectl create secret docker-registry streamparksecret ---docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com ---docker-username=xxxxxx +kubectl create secret docker-registry streamparksecret +--docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com +--docker-username=xxxxxx --docker-password=xxxxxx -n streamx ``` -创建挂载 checkpoint/savepoint 的 pvc 资源，基于阿里云的对象存储OSS做K8S的持久化 +**Creation of PVC Resources for Checkpoints/Savepoints, Utilizing Alibaba Cloud's OSS for K8S Persistence** -**OSS CSI 插件：** +**OSS CSI Plugin:** -可以使用 OSS CSI 插件来帮助简化存储管理。您可以使用 csi 配置创建 pv，并且 pvc、pod 像往常一样定义，yaml 文件参考： +The OSS CSI plugin can be used to help simplify storage management. You can use the CSI configuration to create a PV, and define PVCs and pods as usual. For the YAML file reference, visit: https://bondextest.oss-cn-zhangjiakou.aliyuncs.com/ossyaml.zip -**配置要求：** +**Configuration Requirements:** -\- 创建具有所需 RBAC 权限的服务帐户，参考： +\- Create a service account with the necessary RBAC permissions, reference: Review Comment: Create a service account with the necessary RBAC permissions, please refer as below: ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -1,139 +1,139 @@ --- slug: streampark-usercase-bondex-with-paimon -title: 海程邦达基于 Apache Paimon + StreamPark 的流式数仓实践 -tags: [StreamPark, 生产实践, paimon, streaming-warehouse] +title: Based on Apache Paimon + StreamPark's Streaming Data Warehouse Practice by Bondex +tags: [StreamPark, Production Practice, paimon, streaming-warehouse] ---  -**导读：**本文主要介绍作为供应链物流服务商海程邦达在数字化转型过程中采用 Paimon + StreamPark 平台实现流式数仓的落地方案。我们以 Apache StreamPark 流批一体平台提供了一个易于上手的生产操作手册，以帮助用户提交 Flink 任务并迅速掌握 Paimon 的使用方法。 +**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. -- 公司业务情况介绍 -- 大数据技术痛点以及选型 -- 生产实践 -- 问题排查分析 -- 未来规划 +- Introduction to Company Business +- Pain Points in Big Data Technology and Selection +- Production Practice +- Troubleshooting Analysis +- Future Planning <!-- truncate --> -## 01 公司业务情况介绍 +## 01 Introduction to Company Business -海程邦达集团一直专注于供应链物流领域，通过打造优秀的国际化物流平台，为客户提供端到端一站式智慧型供应链物流服务。集团现有员工 2000 余人，年营业额逾 120 亿人民币，网络遍及全球 200 余个港口，在海内外有超 80 家分、子公司，助力中国企业与世界互联互通。 +Bondex Group has always focused on the field of supply chain logistics. By creating an excellent international logistics platform, it provides customers with end-to-end one-stop intelligent supply chain logistics services. The group currently has over 2,000 employees, an annual turnover of more than 12 billion RMB, a network covering over 200 ports globally, and more than 80 branches and subsidiaries at home and abroad, aiding Chinese enterprises to connect with the world. -### **业务背景** +### **Business Background** -随着公司规模的不断扩大和业务复杂性的增加，为了更好地实现资源优化和流程改进，公司运营与流程管理部需要实时监控公司的业务运转情况，以确保业务流程的稳定性和高效性。 +As the company continues to expand and the complexity of its business increases, in order to better achieve resource optimization and process improvement, the Operations and Process Management Department needs to monitor the company's business operations in real time to ensure the stability and efficiency of business processes. -公司运营与流程管理部负责监督公司各类业务流程的执行，包括海运、空运、铁运各个大区和事业部的订单量，大客户的订单量，航线订单量，关务、仓储、陆运各个操作站点的委托量，公司当天各个大区和事业部实际收入和支出情况等。通过对这些流程的监控和分析，公司能够识别出潜在的问题和瓶颈，提出改进措施和建议，以优化公司运营效率。 +The Operations and Process Management Department is responsible for overseeing the execution of various business processes within the company, including the volume of orders for sea, air, and rail transport across different regions and business divisions, large customer order volumes, route order volumes, the amount of business entrusted to each operation site for customs, warehousing, and land transportation, as well as the actual revenue and expenses of each region and business division on the day. Through monitoring and analysis of these processes, the company can identify potential issues and bottlenecks, propose measures for improvement and suggestions, in order to optimize operational efficiency. -**实时数仓架构：** +**Real-Time Data Warehouse Architecture:**  -当前系统要求直接从生产系统收集实时数据，但存在多个数据源需要进行关联查询，而帆软报表在处理多个数据源时展示不够友好，且无法再次聚合多个数据源。定时查询生产系统会给生产系统数据库带来压力，影响生产系统的稳定运行。因此，我们需要引入一个可以通过 [Flink CDC](https://github.com/ververica/flink-cdc-connectors) 技术实现流式处理的数仓，以解决实时数据处理的问题。这个数仓需要能够从多个数据源收集实时数据并在此基础上实现复杂的关联 SQL 查询、机器学习等操作，并且可以避免不定时查询生产系统，从而减轻生产系统的压力，保障生产系统的稳定运行。 +The current system requires direct collection of real-time data from the production system, but there are multiple data sources that need to be associated for queries. The Fanruan report is not user-friendly when dealing with multiple data sources and cannot re-aggregate multiple data sources. Scheduled queries to the production system database can put pressure on it, affecting the stable operation of the production system. Therefore, we need to introduce a warehouse that can handle real-time data through [Flink CDC](https://github.com/ververica/flink-cdc-connectors) technology for stream processing. This data warehouse needs to be able to collect real-time data from multiple data sources and on this basis, perform complex associated SQL queries, machine learning, etc., and avoid unscheduled queries to the production system, thereby reducing the load on the production system and ensuring its stable operation. -## 02 大数据技术痛点以及选型 +## 02 Big Data Technology Pain Points and Selection -海程邦达大数据团队建立以来一直以高效运维工具或平台来实现对人员的高效配置，优化重复劳动，手工作业。 +Since its establishment, the Bondex big data team has always focused on using efficient operational tools or platforms to achieve effective staffing arrangements, optimize repetitive labor, and reduce manual operations. -在离线批数处理已能够支持集团基础驾驶舱和管理报表的情况下，集团运管部门提出了业务要实时统计订单数量，操作单量的需求，财务部门有现金流实时展示的需求，在这样的背景下，基于大数据的流批一体方案势在必行。 +While offline batch data processing has been able to support the group's basic cockpit and management reporting, the Transportation Management Department of the group has proposed the need for real-time statistics on order quantities and operational volumes. The finance department has expressed the need for a real-time display of cash flow. In this context, a big data-based integrated stream-batch solution became imperative. -虽然大数据部门已经使用了 [Apache Doris](https://github.com/apache/doris) 来实现湖仓一体的存储和计算，此前已在 Doris 社区发表湖仓一体建设的文章，但是有些问题有待解决，流式数据存储无法复用、中间层数据不可查、做不到实时聚合计算问题。 +Although the big data department has already utilized [Apache Doris](https://github.com/apache/doris) for integrated storage and computing of lakehouse architecture and has published articles on lakehouse construction in the Doris community, there are some issues that need to be addressed, such as the inability to reuse streaming data storage, the inaccessibility of intermediate layer data, and the inability to perform real-time aggregation calculations. -按照架构演进时间排序，近几年通用的架构解决方案如下： +Sorted by the evolution of architecture over recent years, the common architectural solutions are as follows: -### **hadoop架构** +### **Hadoop Architecture** -传统数仓和互联网数仓的分界点，在互联网早期的时候，大家对于数据分析的要求也不高，主要是做实时性不高的报表、支撑决策，对应的离线数据分析方案就产生了。 +The demarcation point between traditional data warehouses and internet data warehouses dates back to the early days of the internet when the requirements for data analysis were not high, mainly focusing on reports with low real-time needs to support decision-making. This gave rise to offline data analysis solutions. -**优点：**数据类型支持丰富，支持海量运算，机器配置要求低，时效性低，容错 +**Advantages: **Supports a rich variety of data types, capable of massive computations, low requirements for machine configurations, low timeliness, fault-tolerant. -**缺点：**不支持实时；运维复杂；查询优化器不如 MPP，响应慢 +**Disadvantages: **Does not support real-time; complex to maintain; the query optimizer is not as good as MPP, slow response. -选型依据：不支持实时；运维复杂，不符合人员精简配置原则；性能差 +Selection Basis: Does not support real-time; maintenance is complex, which does not conform to the principle of streamlined staffing; poor performance. -### **lambda架构** +### **Lambda Architecture** -Lambda 架构是由 Storm 的作者 Nathan Marz 提出的一个实时大数据处理框架。Marz 在 Twitter 工作期间开发了著名的实时大数据处理框架 [Apache Storm](https://github.com/apache/storm) ，Lambda 架构是其根据多年进行分布式大数据系统的经验总结提炼而成。 +Lambda Architecture is a real-time big data processing framework proposed by Nathan Marz, the author of Storm. Marz developed the famous real-time big data processing framework [Apache Storm](https://github.com/apache/storm) while working at Twitter, and the Lambda Architecture is the culmination of his years of experience in distributed big data systems.  -数据流处理分为 ServingLayer、SpeedLayer、BatchLayer 三层： +Data stream processing is divided into three layers: Serving Layer, Speed Layer, and Batch Layer: -- Batch层: 对离线数据进行处理，最后提供 view 服务给到业务； -- Speed层: 对实时增量数据进行处理，最后提供 view 服务给到业务； -- Serving层: 响应用户的请求，实现离线和增量数据的聚合计算，并最终提供服务； +- Batch Layer: Processes offline data and eventually provides a view service to the business; +- Speed Layer: Processes real-time incremental data and eventually provides a view service to the business; +- Serving Layer: Responds to user requests, performs aggregation calculations on offline and incremental data, and ultimately provides the service; -优点是：离线和实时分开计算，使用两套框架，架构稳定 +The advantage is that offline and real-time computations are separated, using two sets of frameworks, which makes the architecture stable. -缺点是：离线和实时数据很难保持一致性，运维人员需要维护两套框架三层架构，开发人员需要写三套代码 +The disadvantage is that it is difficult to maintain consistency between offline and real-time data, and operational staff need to maintain two sets of frameworks and three layers of architecture. Developers need to write three sets of code. -选型依据：数据一致性不可控；运维、开发工作量大，不符合人员精简配置的原则； +Selection Basis: Data consistency is uncontrollable; operations and development require significant workload, which does not conform to the principle of streamlined staffing. -### **kappa架构** +### **Kappa Architecture** -kappa 架构只用一套数据流处理架构来解决离线和实时数据，用实时流来解决所有问题，旨在提供快速可靠的查询访问结果。它非常适合各种数据处理工作负载，包括连续数据管道、实时数据处理、机器学习模型和实时数据分析、物联网系统以及许多其他具有单一技术堆栈的用例。 +The Kappa Architecture uses a single stream-processing framework to address both offline and real-time data, solving all problems with a real-time stream, with the goal of providing fast and reliable query access to results. It is highly suitable for various data processing workloads, including continuous data pipelines, real-time data processing, machine learning models and real-time analytics, IoT systems, and many other use cases with a single technology stack. -它通常使用流处理引擎实现，例如Apache Flink、Apache Storm、Apache Kinesis、 Apache Kafka，旨在处理大量数据流并提供快速可靠的查询访问结果。 +It is typically implemented using a streaming processing engine such as Apache Flink, Apache Storm, Apache Kinesis, Apache Kafka, designed to handle large data streams and provide fast and reliable query results.  -**优点是：**单数据流处理框架 +**Advantages: **Single data stream processing framework. -**缺点是：**虽然它的架构相对 lamabda 架构简单，但是流式处理框架的设置和维护相对复杂，不具备真正意义上的离线数据处理能力；流平台中存储大数据成本高昂 +**Disadvantages: **Although its architecture is simpler compared to Lambda Architecture, the setup and maintenance of the streaming processing framework are relatively complex, and it lacks true offline data processing capabilities; storing large amounts of data in streaming platforms can be costly. -选型依据：离线数据处理能力需要保留，控制成本 +Selection Basis: The capability for offline data processing needs to be retained, and costs controlled. ### **Iceberg** -为此我们也调研了 [Apache Iceberg](https://github.com/apache/Iceberg) ，它的快照功能一定程度上能够实现流批一体，但是它的问题是基于 kafka 做的实时表中间层不可查或者无法复用已经存在的表，对 kafka 有强依赖，需要利用 kafka 将中间结果写到 iceberg 表，增加了系统的复杂度和可维护性。 +Therefore, we also researched [Apache Iceberg](https://github.com/apache/iceberg), whose snapshot feature can to some extent achieve streaming-batch integration. However, the issue with it is that the real-time table layer based on Kafka is either not queryable or cannot reuse existing tables, with a strong dependency on Kafka. It requires the use of Kafka to write intermediate results to Iceberg tables, increasing system complexity and maintainability. -选型依据：无 kafka 实时架构已落地，中间数据无法实现可查可复用 +Selection Basis: A Kafka-free real-time architecture has been implemented, intermediate data cannot be made queryable or reusable. -### **流式数仓（kappa架构的延续)** +### **Streaming Data Warehouse (Continuation of the Kappa Architecture)** -海程邦达大数据团队自 FTS0.3.0 版本开始参与流式数仓建设，旨在进一步降低数据处理框架的复杂度和人员的精简配置，前期的宗旨是既然是趋势就要参与进来，过程中不断学习精进，向最前沿的技术靠拢，团队一致认为有坑就踩坑，摸着石头也要过河，好在经过几个版本的迭代，在社区的高效配合下，最开始出现的问题也慢慢得以解决 +Since the FTS0.3.0 version, the BonDex big data team has participated in the construction of a streaming data warehouse, aiming to further reduce the complexity of the data processing framework and streamline personnel configuration. The initial principle was to get involved with the trend, to continuously learn and improve, and to move closer to cutting-edge technology. The team unanimously believes that it is essential to embrace challenges and "cross the river by feeling the stones." Fortunately, after several iterations, the problems that initially arose have been gradually resolved with the efficient cooperation of the community. -**流式数仓架构如下：** +**The architecture of the streaming data warehouse is as follows:**  -延续了 kappa 架构的特点，一套流处理架构，好处在与，底层 Paimon 的技术支撑使得数据在全链路可查，数仓分层架构得以复用，同时兼顾了离线和实时的处理能力，减少存储和计算的浪费 +Continuing the characteristics of the Kappa architecture with a single stream processing framework, the advantage lies in the fact that the underlying Paimon technology support makes the data traceable throughout the entire chain. The data warehouse layer architecture can be reused, while also considering the processing capabilities of both offline and real-time data, reducing the waste of storage and computing resources. -## 03 生 产 实 践 +## 03 Production Practices -本方案采用 Flink Application On K8s 集群，Flink CDC 实时摄取业务系统关系型数据库数据，通过 StreamPark 任务平台提交 Flink + Paimon Streaming Data Warehouse 任务， 最后采用 Trino 引擎接入 Finereport 提供服务和开发人员的查询。Paimon 底层存储支持 S3 协议，因为公司大数据服务依赖于阿里云所以使用对象存储OSS作为数据文件系统。 +This solution adopts Flink Application on K8s clusters, with Flink CDC for real-time ingestion of relational database data from business systems. Tasks for Flink + Paimon Streaming Data Warehouse are submitted through the StreamPark task platform, with the Trino engine ultimately used to access Finereport for service provision and developer queries. Paimon's underlying storage supports the S3 protocol, and as the company's big data services rely on Alibaba Cloud, Object Storage Service (OSS) is used as the data filesystem. -[StreamPark](https://github.com/apache/incubator-streampark) 是一个实时计算平台，与 [Paimon](https://github.com/apache/incubator-paimon) 结合使用其强大功能来处理实时数据流。此平台提供以下主要功能： +[StreamPark](https://github.com/apache/incubator-streampark) is a real-time computing platform that leverages the powerful capabilities of [Paimon](https://github.com/apache/incubator-paimon) to process real-time data streams. This platform offers the following key features: -**实时数据处理：**StreamPark 支持提交实时数据流任务，能够实时获取、转换、过滤和分析数据。这对于需要快速响应实时数据的应用非常重要，例如实时监控、实时推荐和实时风控等领域。 +**Real-time Data Processing: **StreamPark supports the submission of real-time data stream tasks, capable of real-time acquisition, transformation, filtering, and analysis of data. This is extremely important for applications that require rapid response to real-time data, such as real-time monitoring, real-time recommendations, and real-time risk control. -**可扩展性：**可以高效处理大规模实时数据，具备良好的可扩展性。可以在分布式计算环境中运行，并能够自动处理并行化、故障恢复和负载均衡等问题，以确保高效且可靠地处理数据。 +**Scalability: **Capable of efficiently processing large-scale real-time data with good scalability. It can operate in a distributed computing environment, automatically handling parallelization, fault recovery, and load balancing to ensure efficient and reliable data processing. -**Flink 集成：**基于 [Apache Flink](https://github.com/apache/flink) 构建，利用 Flink 的强大流处理引擎来实现高性能和鲁棒性。用户可以充分利用 Flink 的特性和生态系统，如广泛的连接器、状态管理和事件时间处理等。 +**Flink Integration: **Built on [Apache Flink](https://github.com/apache/flink), it harnesses Flink’s powerful stream processing engine for high performance and robustness. Users can fully utilize the features and ecosystem of Flink, such as its extensive connectors, state management, and event-time processing. Review Comment: it leverages Flink’s powerful stream processing engine ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -171,7 +171,7 @@ rest.port: 8081 rest.address: localhost ``` -建议可以在本地添加 FLINK_HOME 方便在上 k8s 之前本地排查问题使用 +It is advisable to add FLINK_HOME locally for convenient troubleshooting before deploying on k8s. Review Comment: It is suggested to add FLINK_HOME locally for convenient troubleshooting before deploying on k8s. ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -234,127 +234,127 @@ docker tag flink-table-store:v1.16.0 registry-vpc.cn-zhangjiakou.aliyuncs.com/xx docker push registry-vpc.cn-zhangjiakou.aliyuncs.com/xxxxx/flink-table-store:v1.16.0 ``` -接下来准备 Paimon jar 包，可以在 Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon) 下载对应版本，需要注意的是要和 flink 大版本保持一致 +Next, prepare the Paimon jar package. You can download the corresponding version from the Apache [Repository](https://repository.apache.org/content/groups/snapshots/org/apache/paimon). It's important to note that it should be consistent with the major version of Flink. -### **使用 StreamPark 管理作业** +### **Managing Jobs with StreamPark** -**前提条件：** +**Prerequisites:** -- Kubernetes 客户端连接配置 -- Kubernetes RBAC 配置 -- 容器镜像仓库配置 （案例中使用的是阿里云镜像免费版） -- 创建挂载 checkpoint/savepoint 的 pvc 资源 +- Kubernetes client connection configuration +- Kubernetes RBAC configuration +- Container image repository configuration (the free version of Alibaba Cloud image is used in this case) +- Create a PVC resource to mount checkpoints/savepoints -**Kubernetes 客户端连接配置：** +**Kubernetes Client Connection Configuration:** -将 k8s master节点~/.kube/config 配置直接拷贝到 StreamPark 服务器的目录，之后在 StreamPark 服务器执行以下命令显示 k8s 集群 running 代表权限和网络验证成功。 +Copy the k8s master node's `~/.kube/config` configuration directly to the directory on the StreamPark server, then execute the following command on the StreamPark server to display the k8s cluster as running, which indicates successful permission and network verification. ```shell kubectl cluster-info ``` -Kubernetes RBAC 配置,创建 streamx 命名空间: +Kubernetes RBAC Configuration, create the streampark namespace: ```shell -kubectl create ns streamx +kubectl create ns streampark ``` -使用 default 账户创建 clusterrolebinding 资源: +Use the default account to create the clusterrolebinding resource: ```shell -kubectl create secret docker-registry streamparksecret ---docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com ---docker-username=xxxxxx +kubectl create secret docker-registry streamparksecret +--docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com +--docker-username=xxxxxx --docker-password=xxxxxx -n streamx``` ``` -**容器镜像仓库配置:** +**Container Image Registry Configuration:** -案例中使用阿里云容器镜像服务ACR，也可以使用自建镜像服务harbor代替。 +In this case, Alibaba Cloud's Container Registry Service (ACR) is used, but you can also substitute it with a self-hosted image service such as Harbor. -创建命名空间 StreamPark (安全设置需要设置为私有) +Create a namespace named StreamPark (set the security setting to private).  -在 StreamPark 配置镜像仓库，任务构建镜像会推送到镜像仓库 +Configure the image repository in StreamPark; task build images will be pushed to the repository.  -创建 k8s secret 密钥用来拉取 ACR 中的镜像 streamparksecret 为密钥名称 自定义 +Create a k8s secret key to pull images from ACR; streamparksecret is the name of the secret, customizable. ```shell -kubectl create secret docker-registry streamparksecret ---docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com ---docker-username=xxxxxx +kubectl create secret docker-registry streamparksecret +--docker-server=registry-vpc.cn-zhangjiakou.aliyuncs.com +--docker-username=xxxxxx --docker-password=xxxxxx -n streamx ``` -创建挂载 checkpoint/savepoint 的 pvc 资源，基于阿里云的对象存储OSS做K8S的持久化 +**Creation of PVC Resources for Checkpoints/Savepoints, Utilizing Alibaba Cloud's OSS for K8S Persistence** Review Comment: No ** ** ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -1,139 +1,139 @@ --- slug: streampark-usercase-bondex-with-paimon -title: 海程邦达基于 Apache Paimon + StreamPark 的流式数仓实践 -tags: [StreamPark, 生产实践, paimon, streaming-warehouse] +title: Based on Apache Paimon + StreamPark's Streaming Data Warehouse Practice by Bondex +tags: [StreamPark, Production Practice, paimon, streaming-warehouse] ---  -**导读：**本文主要介绍作为供应链物流服务商海程邦达在数字化转型过程中采用 Paimon + StreamPark 平台实现流式数仓的落地方案。我们以 Apache StreamPark 流批一体平台提供了一个易于上手的生产操作手册，以帮助用户提交 Flink 任务并迅速掌握 Paimon 的使用方法。 +**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. -- 公司业务情况介绍 -- 大数据技术痛点以及选型 -- 生产实践 -- 问题排查分析 -- 未来规划 +- Introduction to Company Business +- Pain Points in Big Data Technology and Selection +- Production Practice +- Troubleshooting Analysis +- Future Planning <!-- truncate --> -## 01 公司业务情况介绍 +## 01 Introduction to Company Business -海程邦达集团一直专注于供应链物流领域，通过打造优秀的国际化物流平台，为客户提供端到端一站式智慧型供应链物流服务。集团现有员工 2000 余人，年营业额逾 120 亿人民币，网络遍及全球 200 余个港口，在海内外有超 80 家分、子公司，助力中国企业与世界互联互通。 +Bondex Group has always focused on the field of supply chain logistics. By creating an excellent international logistics platform, it provides customers with end-to-end one-stop intelligent supply chain logistics services. The group currently has over 2,000 employees, an annual turnover of more than 12 billion RMB, a network covering over 200 ports globally, and more than 80 branches and subsidiaries at home and abroad, aiding Chinese enterprises to connect with the world. -### **业务背景** +### **Business Background** -随着公司规模的不断扩大和业务复杂性的增加，为了更好地实现资源优化和流程改进，公司运营与流程管理部需要实时监控公司的业务运转情况，以确保业务流程的稳定性和高效性。 +As the company continues to expand and the complexity of its business increases, in order to better achieve resource optimization and process improvement, the Operations and Process Management Department needs to monitor the company's business operations in real time to ensure the stability and efficiency of business processes. -公司运营与流程管理部负责监督公司各类业务流程的执行，包括海运、空运、铁运各个大区和事业部的订单量，大客户的订单量，航线订单量，关务、仓储、陆运各个操作站点的委托量，公司当天各个大区和事业部实际收入和支出情况等。通过对这些流程的监控和分析，公司能够识别出潜在的问题和瓶颈，提出改进措施和建议，以优化公司运营效率。 +The Operations and Process Management Department is responsible for overseeing the execution of various business processes within the company, including the volume of orders for sea, air, and rail transport across different regions and business divisions, large customer order volumes, route order volumes, the amount of business entrusted to each operation site for customs, warehousing, and land transportation, as well as the actual revenue and expenses of each region and business division on the day. Through monitoring and analysis of these processes, the company can identify potential issues and bottlenecks, propose measures for improvement and suggestions, in order to optimize operational efficiency. -**实时数仓架构：** +**Real-Time Data Warehouse Architecture:**  -当前系统要求直接从生产系统收集实时数据，但存在多个数据源需要进行关联查询，而帆软报表在处理多个数据源时展示不够友好，且无法再次聚合多个数据源。定时查询生产系统会给生产系统数据库带来压力，影响生产系统的稳定运行。因此，我们需要引入一个可以通过 [Flink CDC](https://github.com/ververica/flink-cdc-connectors) 技术实现流式处理的数仓，以解决实时数据处理的问题。这个数仓需要能够从多个数据源收集实时数据并在此基础上实现复杂的关联 SQL 查询、机器学习等操作，并且可以避免不定时查询生产系统，从而减轻生产系统的压力，保障生产系统的稳定运行。 +The current system requires direct collection of real-time data from the production system, but there are multiple data sources that need to be associated for queries. The Fanruan report is not user-friendly when dealing with multiple data sources and cannot re-aggregate multiple data sources. Scheduled queries to the production system database can put pressure on it, affecting the stable operation of the production system. Therefore, we need to introduce a warehouse that can handle real-time data through [Flink CDC](https://github.com/ververica/flink-cdc-connectors) technology for stream processing. This data warehouse needs to be able to collect real-time data from multiple data sources and on this basis, perform complex associated SQL queries, machine learning, etc., and avoid unscheduled queries to the production system, thereby reducing the load on the production system and ensuring its stable operation. -## 02 大数据技术痛点以及选型 +## 02 Big Data Technology Pain Points and Selection -海程邦达大数据团队建立以来一直以高效运维工具或平台来实现对人员的高效配置，优化重复劳动，手工作业。 +Since its establishment, the Bondex big data team has always focused on using efficient operational tools or platforms to achieve effective staffing arrangements, optimize repetitive labor, and reduce manual operations. -在离线批数处理已能够支持集团基础驾驶舱和管理报表的情况下，集团运管部门提出了业务要实时统计订单数量，操作单量的需求，财务部门有现金流实时展示的需求，在这样的背景下，基于大数据的流批一体方案势在必行。 +While offline batch data processing has been able to support the group's basic cockpit and management reporting, the Transportation Management Department of the group has proposed the need for real-time statistics on order quantities and operational volumes. The finance department has expressed the need for a real-time display of cash flow. In this context, a big data-based integrated stream-batch solution became imperative. -虽然大数据部门已经使用了 [Apache Doris](https://github.com/apache/doris) 来实现湖仓一体的存储和计算，此前已在 Doris 社区发表湖仓一体建设的文章，但是有些问题有待解决，流式数据存储无法复用、中间层数据不可查、做不到实时聚合计算问题。 +Although the big data department has already utilized [Apache Doris](https://github.com/apache/doris) for integrated storage and computing of lakehouse architecture and has published articles on lakehouse construction in the Doris community, there are some issues that need to be addressed, such as the inability to reuse streaming data storage, the inaccessibility of intermediate layer data, and the inability to perform real-time aggregation calculations. -按照架构演进时间排序，近几年通用的架构解决方案如下： +Sorted by the evolution of architecture over recent years, the common architectural solutions are as follows: -### **hadoop架构** +### **Hadoop Architecture** -传统数仓和互联网数仓的分界点，在互联网早期的时候，大家对于数据分析的要求也不高，主要是做实时性不高的报表、支撑决策，对应的离线数据分析方案就产生了。 +The demarcation point between traditional data warehouses and internet data warehouses dates back to the early days of the internet when the requirements for data analysis were not high, mainly focusing on reports with low real-time needs to support decision-making. This gave rise to offline data analysis solutions. -**优点：**数据类型支持丰富，支持海量运算，机器配置要求低，时效性低，容错 +**Advantages: **Supports a rich variety of data types, capable of massive computations, low requirements for machine configurations, low timeliness, fault-tolerant. -**缺点：**不支持实时；运维复杂；查询优化器不如 MPP，响应慢 +**Disadvantages: **Does not support real-time; complex to maintain; the query optimizer is not as good as MPP, slow response. -选型依据：不支持实时；运维复杂，不符合人员精简配置原则；性能差 +Selection Basis: Does not support real-time; maintenance is complex, which does not conform to the principle of streamlined staffing; poor performance. -### **lambda架构** +### **Lambda Architecture** -Lambda 架构是由 Storm 的作者 Nathan Marz 提出的一个实时大数据处理框架。Marz 在 Twitter 工作期间开发了著名的实时大数据处理框架 [Apache Storm](https://github.com/apache/storm) ，Lambda 架构是其根据多年进行分布式大数据系统的经验总结提炼而成。 +Lambda Architecture is a real-time big data processing framework proposed by Nathan Marz, the author of Storm. Marz developed the famous real-time big data processing framework [Apache Storm](https://github.com/apache/storm) while working at Twitter, and the Lambda Architecture is the culmination of his years of experience in distributed big data systems.  -数据流处理分为 ServingLayer、SpeedLayer、BatchLayer 三层： +Data stream processing is divided into three layers: Serving Layer, Speed Layer, and Batch Layer: -- Batch层: 对离线数据进行处理，最后提供 view 服务给到业务； -- Speed层: 对实时增量数据进行处理，最后提供 view 服务给到业务； -- Serving层: 响应用户的请求，实现离线和增量数据的聚合计算，并最终提供服务； +- Batch Layer: Processes offline data and eventually provides a view service to the business; +- Speed Layer: Processes real-time incremental data and eventually provides a view service to the business; +- Serving Layer: Responds to user requests, performs aggregation calculations on offline and incremental data, and ultimately provides the service; -优点是：离线和实时分开计算，使用两套框架，架构稳定 +The advantage is that offline and real-time computations are separated, using two sets of frameworks, which makes the architecture stable. -缺点是：离线和实时数据很难保持一致性，运维人员需要维护两套框架三层架构，开发人员需要写三套代码 +The disadvantage is that it is difficult to maintain consistency between offline and real-time data, and operational staff need to maintain two sets of frameworks and three layers of architecture. Developers need to write three sets of code. -选型依据：数据一致性不可控；运维、开发工作量大，不符合人员精简配置的原则； +Selection Basis: Data consistency is uncontrollable; operations and development require significant workload, which does not conform to the principle of streamlined staffing. -### **kappa架构** +### **Kappa Architecture** -kappa 架构只用一套数据流处理架构来解决离线和实时数据，用实时流来解决所有问题，旨在提供快速可靠的查询访问结果。它非常适合各种数据处理工作负载，包括连续数据管道、实时数据处理、机器学习模型和实时数据分析、物联网系统以及许多其他具有单一技术堆栈的用例。 +The Kappa Architecture uses a single stream-processing framework to address both offline and real-time data, solving all problems with a real-time stream, with the goal of providing fast and reliable query access to results. It is highly suitable for various data processing workloads, including continuous data pipelines, real-time data processing, machine learning models and real-time analytics, IoT systems, and many other use cases with a single technology stack. -它通常使用流处理引擎实现，例如Apache Flink、Apache Storm、Apache Kinesis、 Apache Kafka，旨在处理大量数据流并提供快速可靠的查询访问结果。 +It is typically implemented using a streaming processing engine such as Apache Flink, Apache Storm, Apache Kinesis, Apache Kafka, designed to handle large data streams and provide fast and reliable query results.  -**优点是：**单数据流处理框架 +**Advantages: **Single data stream processing framework. -**缺点是：**虽然它的架构相对 lamabda 架构简单，但是流式处理框架的设置和维护相对复杂，不具备真正意义上的离线数据处理能力；流平台中存储大数据成本高昂 +**Disadvantages: **Although its architecture is simpler compared to Lambda Architecture, the setup and maintenance of the streaming processing framework are relatively complex, and it lacks true offline data processing capabilities; storing large amounts of data in streaming platforms can be costly. -选型依据：离线数据处理能力需要保留，控制成本 +Selection Basis: The capability for offline data processing needs to be retained, and costs controlled. ### **Iceberg** -为此我们也调研了 [Apache Iceberg](https://github.com/apache/Iceberg) ，它的快照功能一定程度上能够实现流批一体，但是它的问题是基于 kafka 做的实时表中间层不可查或者无法复用已经存在的表，对 kafka 有强依赖，需要利用 kafka 将中间结果写到 iceberg 表，增加了系统的复杂度和可维护性。 +Therefore, we also researched [Apache Iceberg](https://github.com/apache/iceberg), whose snapshot feature can to some extent achieve streaming-batch integration. However, the issue with it is that the real-time table layer based on Kafka is either not queryable or cannot reuse existing tables, with a strong dependency on Kafka. It requires the use of Kafka to write intermediate results to Iceberg tables, increasing system complexity and maintainability. -选型依据：无 kafka 实时架构已落地，中间数据无法实现可查可复用 +Selection Basis: A Kafka-free real-time architecture has been implemented, intermediate data cannot be made queryable or reusable. -### **流式数仓（kappa架构的延续)** +### **Streaming Data Warehouse (Continuation of the Kappa Architecture)** -海程邦达大数据团队自 FTS0.3.0 版本开始参与流式数仓建设，旨在进一步降低数据处理框架的复杂度和人员的精简配置，前期的宗旨是既然是趋势就要参与进来，过程中不断学习精进，向最前沿的技术靠拢，团队一致认为有坑就踩坑，摸着石头也要过河，好在经过几个版本的迭代，在社区的高效配合下，最开始出现的问题也慢慢得以解决 +Since the FTS0.3.0 version, the BonDex big data team has participated in the construction of a streaming data warehouse, aiming to further reduce the complexity of the data processing framework and streamline personnel configuration. The initial principle was to get involved with the trend, to continuously learn and improve, and to move closer to cutting-edge technology. The team unanimously believes that it is essential to embrace challenges and "cross the river by feeling the stones." Fortunately, after several iterations, the problems that initially arose have been gradually resolved with the efficient cooperation of the community. -**流式数仓架构如下：** +**The architecture of the streaming data warehouse is as follows:**  -延续了 kappa 架构的特点，一套流处理架构，好处在与，底层 Paimon 的技术支撑使得数据在全链路可查，数仓分层架构得以复用，同时兼顾了离线和实时的处理能力，减少存储和计算的浪费 +Continuing the characteristics of the Kappa architecture with a single stream processing framework, the advantage lies in the fact that the underlying Paimon technology support makes the data traceable throughout the entire chain. The data warehouse layer architecture can be reused, while also considering the processing capabilities of both offline and real-time data, reducing the waste of storage and computing resources. -## 03 生 产 实 践 +## 03 Production Practices -本方案采用 Flink Application On K8s 集群，Flink CDC 实时摄取业务系统关系型数据库数据，通过 StreamPark 任务平台提交 Flink + Paimon Streaming Data Warehouse 任务， 最后采用 Trino 引擎接入 Finereport 提供服务和开发人员的查询。Paimon 底层存储支持 S3 协议，因为公司大数据服务依赖于阿里云所以使用对象存储OSS作为数据文件系统。 +This solution adopts Flink Application on K8s clusters, with Flink CDC for real-time ingestion of relational database data from business systems. Tasks for Flink + Paimon Streaming Data Warehouse are submitted through the StreamPark task platform, with the Trino engine ultimately used to access Finereport for service provision and developer queries. Paimon's underlying storage supports the S3 protocol, and as the company's big data services rely on Alibaba Cloud, Object Storage Service (OSS) is used as the data filesystem. -[StreamPark](https://github.com/apache/incubator-streampark) 是一个实时计算平台，与 [Paimon](https://github.com/apache/incubator-paimon) 结合使用其强大功能来处理实时数据流。此平台提供以下主要功能： +[StreamPark](https://github.com/apache/incubator-streampark) is a real-time computing platform that leverages the powerful capabilities of [Paimon](https://github.com/apache/incubator-paimon) to process real-time data streams. This platform offers the following key features: -**实时数据处理：**StreamPark 支持提交实时数据流任务，能够实时获取、转换、过滤和分析数据。这对于需要快速响应实时数据的应用非常重要，例如实时监控、实时推荐和实时风控等领域。 +**Real-time Data Processing: **StreamPark supports the submission of real-time data stream tasks, capable of real-time acquisition, transformation, filtering, and analysis of data. This is extremely important for applications that require rapid response to real-time data, such as real-time monitoring, real-time recommendations, and real-time risk control. -**可扩展性：**可以高效处理大规模实时数据，具备良好的可扩展性。可以在分布式计算环境中运行，并能够自动处理并行化、故障恢复和负载均衡等问题，以确保高效且可靠地处理数据。 +**Scalability: **Capable of efficiently processing large-scale real-time data with good scalability. It can operate in a distributed computing environment, automatically handling parallelization, fault recovery, and load balancing to ensure efficient and reliable data processing. -**Flink 集成：**基于 [Apache Flink](https://github.com/apache/flink) 构建，利用 Flink 的强大流处理引擎来实现高性能和鲁棒性。用户可以充分利用 Flink 的特性和生态系统，如广泛的连接器、状态管理和事件时间处理等。 +**Flink Integration: **Built on [Apache Flink](https://github.com/apache/flink), it harnesses Flink’s powerful stream processing engine for high performance and robustness. Users can fully utilize the features and ecosystem of Flink, such as its extensive connectors, state management, and event-time processing. Review Comment: it leverages Flink’s powerful stream processing engine ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -813,72 +813,72 @@ By specifying changelog-producer.compaction-interval table property (default val users can define the maximum interval between two full compactions to ensure latency. This table property does not affect normal compactions and they may still be performed once in a while by writers to reduce reader costs. -这样能解决上述问题。但是随之而来出现了新的问题。默认 changelog-producer.compaction-interval 是 30min，意味着 上游的改动到 ads 查询要间隔 30min，生产过程中发现将压缩间隔时间改成 1min 或者 2 分钟的情况下，又会出现上述 ADS 层聚合数据不准的情况。 +This approach can solve the aforementioned issue. However, it has led to a new problem. The default 'changelog-producer.compaction-interval' is 30 minutes, meaning that it takes 30 minutes for changes upstream to be reflected in the ads query. During production, it has been found that changing the compaction interval to 1 minute or 2 minutes can cause inaccuracies in the ADS layer aggregation data again. Review Comment: This approach can solve the above mentioned issue. The default changelog-producer.compaction-interval is 30 minutes. ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -813,72 +813,72 @@ By specifying changelog-producer.compaction-interval table property (default val users can define the maximum interval between two full compactions to ensure latency. This table property does not affect normal compactions and they may still be performed once in a while by writers to reduce reader costs. -这样能解决上述问题。但是随之而来出现了新的问题。默认 changelog-producer.compaction-interval 是 30min，意味着 上游的改动到 ads 查询要间隔 30min，生产过程中发现将压缩间隔时间改成 1min 或者 2 分钟的情况下，又会出现上述 ADS 层聚合数据不准的情况。 +This approach can solve the aforementioned issue. However, it has led to a new problem. The default 'changelog-producer.compaction-interval' is 30 minutes, meaning that it takes 30 minutes for changes upstream to be reflected in the ads query. During production, it has been found that changing the compaction interval to 1 minute or 2 minutes can cause inaccuracies in the ADS layer aggregation data again. ```sql 'changelog-producer.compaction-interval' = '2m' ``` -需要在写入 Flink Table Store 时需要配置 table.exec.sink.upsert-materialize= none，避免产生 Upsert 流，以保证 Flink Table Store 中能够保存完整的 changelog，为后续的流读操作做准备。 +When writing into the Flink Table Store, it is necessary to configure 'table.exec.sink.upsert-materialize' to 'none' to avoid generating an upsert stream, ensuring that the Flink Table Store can retain a complete changelog for subsequent stream read operations. ```sql set 'table.exec.sink.upsert-materialize' = 'none' ``` -**2. 相同 sequence.field 导致 dwd 明细宽表无法收到 update 数据更新** +**2. The same 'sequence.field' causes the dwd detailed wide table to not receive data updates** Review Comment: No '' in CN text. ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -813,72 +813,72 @@ By specifying changelog-producer.compaction-interval table property (default val users can define the maximum interval between two full compactions to ensure latency. This table property does not affect normal compactions and they may still be performed once in a while by writers to reduce reader costs. -这样能解决上述问题。但是随之而来出现了新的问题。默认 changelog-producer.compaction-interval 是 30min，意味着 上游的改动到 ads 查询要间隔 30min，生产过程中发现将压缩间隔时间改成 1min 或者 2 分钟的情况下，又会出现上述 ADS 层聚合数据不准的情况。 +This approach can solve the aforementioned issue. However, it has led to a new problem. The default 'changelog-producer.compaction-interval' is 30 minutes, meaning that it takes 30 minutes for changes upstream to be reflected in the ads query. During production, it has been found that changing the compaction interval to 1 minute or 2 minutes can cause inaccuracies in the ADS layer aggregation data again. ```sql 'changelog-producer.compaction-interval' = '2m' ``` -需要在写入 Flink Table Store 时需要配置 table.exec.sink.upsert-materialize= none，避免产生 Upsert 流，以保证 Flink Table Store 中能够保存完整的 changelog，为后续的流读操作做准备。 +When writing into the Flink Table Store, it is necessary to configure 'table.exec.sink.upsert-materialize' to 'none' to avoid generating an upsert stream, ensuring that the Flink Table Store can retain a complete changelog for subsequent stream read operations. Review Comment: No '' in CN text. ########## blog/3-streampark-usercase-bondex-paimon.md: ########## @@ -1,139 +1,139 @@ --- slug: streampark-usercase-bondex-with-paimon -title: 海程邦达基于 Apache Paimon + StreamPark 的流式数仓实践 -tags: [StreamPark, 生产实践, paimon, streaming-warehouse] +title: Based on Apache Paimon + StreamPark's Streaming Data Warehouse Practice by Bondex +tags: [StreamPark, Production Practice, paimon, streaming-warehouse] ---  -**导读：**本文主要介绍作为供应链物流服务商海程邦达在数字化转型过程中采用 Paimon + StreamPark 平台实现流式数仓的落地方案。我们以 Apache StreamPark 流批一体平台提供了一个易于上手的生产操作手册，以帮助用户提交 Flink 任务并迅速掌握 Paimon 的使用方法。 +**Foreword: **This article mainly introduces the implementation of a streaming data warehouse by Bondex, a supply chain logistics service provider, in the process of digital transformation using the Paimon + StreamPark platform. We provide an easy-to-follow operational manual with the Apache StreamPark integrated stream-batch platform to help users submit Flink tasks and quickly master the use of Paimon. -- 公司业务情况介绍 -- 大数据技术痛点以及选型 -- 生产实践 -- 问题排查分析 -- 未来规划 +- Introduction to Company Business +- Pain Points in Big Data Technology and Selection +- Production Practice +- Troubleshooting Analysis +- Future Planning <!-- truncate --> -## 01 公司业务情况介绍 +## 01 Introduction to Company Business -海程邦达集团一直专注于供应链物流领域，通过打造优秀的国际化物流平台，为客户提供端到端一站式智慧型供应链物流服务。集团现有员工 2000 余人，年营业额逾 120 亿人民币，网络遍及全球 200 余个港口，在海内外有超 80 家分、子公司，助力中国企业与世界互联互通。 +Bondex Group has always focused on the field of supply chain logistics. By creating an excellent international logistics platform, it provides customers with end-to-end one-stop intelligent supply chain logistics services. The group currently has over 2,000 employees, an annual turnover of more than 12 billion RMB, a network covering over 200 ports globally, and more than 80 branches and subsidiaries at home and abroad, aiding Chinese enterprises to connect with the world. -### **业务背景** +### **Business Background** -随着公司规模的不断扩大和业务复杂性的增加，为了更好地实现资源优化和流程改进，公司运营与流程管理部需要实时监控公司的业务运转情况，以确保业务流程的稳定性和高效性。 +As the company continues to expand and the complexity of its business increases, in order to better achieve resource optimization and process improvement, the Operations and Process Management Department needs to monitor the company's business operations in real time to ensure the stability and efficiency of business processes. -公司运营与流程管理部负责监督公司各类业务流程的执行，包括海运、空运、铁运各个大区和事业部的订单量，大客户的订单量，航线订单量，关务、仓储、陆运各个操作站点的委托量，公司当天各个大区和事业部实际收入和支出情况等。通过对这些流程的监控和分析，公司能够识别出潜在的问题和瓶颈，提出改进措施和建议，以优化公司运营效率。 +The Operations and Process Management Department is responsible for overseeing the execution of various business processes within the company, including the volume of orders for sea, air, and rail transport across different regions and business divisions, large customer order volumes, route order volumes, the amount of business entrusted to each operation site for customs, warehousing, and land transportation, as well as the actual revenue and expenses of each region and business division on the day. Through monitoring and analysis of these processes, the company can identify potential issues and bottlenecks, propose measures for improvement and suggestions, in order to optimize operational efficiency. -**实时数仓架构：** +**Real-Time Data Warehouse Architecture:**  -当前系统要求直接从生产系统收集实时数据，但存在多个数据源需要进行关联查询，而帆软报表在处理多个数据源时展示不够友好，且无法再次聚合多个数据源。定时查询生产系统会给生产系统数据库带来压力，影响生产系统的稳定运行。因此，我们需要引入一个可以通过 [Flink CDC](https://github.com/ververica/flink-cdc-connectors) 技术实现流式处理的数仓，以解决实时数据处理的问题。这个数仓需要能够从多个数据源收集实时数据并在此基础上实现复杂的关联 SQL 查询、机器学习等操作，并且可以避免不定时查询生产系统，从而减轻生产系统的压力，保障生产系统的稳定运行。 +The current system requires direct collection of real-time data from the production system, but there are multiple data sources that need to be associated for queries. The Fanruan report is not user-friendly when dealing with multiple data sources and cannot re-aggregate multiple data sources. Scheduled queries to the production system database can put pressure on it, affecting the stable operation of the production system. Therefore, we need to introduce a warehouse that can handle real-time data through [Flink CDC](https://github.com/ververica/flink-cdc-connectors) technology for stream processing. This data warehouse needs to be able to collect real-time data from multiple data sources and on this basis, perform complex associated SQL queries, machine learning, etc., and avoid unscheduled queries to the production system, thereby reducing the load on the production system and ensuring its stable operation. -## 02 大数据技术痛点以及选型 +## 02 Big Data Technology Pain Points and Selection -海程邦达大数据团队建立以来一直以高效运维工具或平台来实现对人员的高效配置，优化重复劳动，手工作业。 +Since its establishment, the Bondex big data team has always focused on using efficient operational tools or platforms to achieve effective staffing arrangements, optimize repetitive labor, and reduce manual operations. -在离线批数处理已能够支持集团基础驾驶舱和管理报表的情况下，集团运管部门提出了业务要实时统计订单数量，操作单量的需求，财务部门有现金流实时展示的需求，在这样的背景下，基于大数据的流批一体方案势在必行。 +While offline batch data processing has been able to support the group's basic cockpit and management reporting, the Transportation Management Department of the group has proposed the need for real-time statistics on order quantities and operational volumes. The finance department has expressed the need for a real-time display of cash flow. In this context, a big data-based integrated stream-batch solution became imperative. -虽然大数据部门已经使用了 [Apache Doris](https://github.com/apache/doris) 来实现湖仓一体的存储和计算，此前已在 Doris 社区发表湖仓一体建设的文章，但是有些问题有待解决，流式数据存储无法复用、中间层数据不可查、做不到实时聚合计算问题。 +Although the big data department has already utilized [Apache Doris](https://github.com/apache/doris) for integrated storage and computing of lakehouse architecture and has published articles on lakehouse construction in the Doris community, there are some issues that need to be addressed, such as the inability to reuse streaming data storage, the inaccessibility of intermediate layer data, and the inability to perform real-time aggregation calculations. -按照架构演进时间排序，近几年通用的架构解决方案如下： +Sorted by the evolution of architecture over recent years, the common architectural solutions are as follows: -### **hadoop架构** +### **Hadoop Architecture** -传统数仓和互联网数仓的分界点，在互联网早期的时候，大家对于数据分析的要求也不高，主要是做实时性不高的报表、支撑决策，对应的离线数据分析方案就产生了。 +The demarcation point between traditional data warehouses and internet data warehouses dates back to the early days of the internet when the requirements for data analysis were not high, mainly focusing on reports with low real-time needs to support decision-making. This gave rise to offline data analysis solutions. -**优点：**数据类型支持丰富，支持海量运算，机器配置要求低，时效性低，容错 +**Advantages: **Supports a rich variety of data types, capable of massive computations, low requirements for machine configurations, low timeliness, fault-tolerant. -**缺点：**不支持实时；运维复杂；查询优化器不如 MPP，响应慢 +**Disadvantages: **Does not support real-time; complex to maintain; the query optimizer is not as good as MPP, slow response. -选型依据：不支持实时；运维复杂，不符合人员精简配置原则；性能差 +Selection Basis: Does not support real-time; maintenance is complex, which does not conform to the principle of streamlined staffing; poor performance. -### **lambda架构** +### **Lambda Architecture** -Lambda 架构是由 Storm 的作者 Nathan Marz 提出的一个实时大数据处理框架。Marz 在 Twitter 工作期间开发了著名的实时大数据处理框架 [Apache Storm](https://github.com/apache/storm) ，Lambda 架构是其根据多年进行分布式大数据系统的经验总结提炼而成。 +Lambda Architecture is a real-time big data processing framework proposed by Nathan Marz, the author of Storm. Marz developed the famous real-time big data processing framework [Apache Storm](https://github.com/apache/storm) while working at Twitter, and the Lambda Architecture is the culmination of his years of experience in distributed big data systems.  -数据流处理分为 ServingLayer、SpeedLayer、BatchLayer 三层： +Data stream processing is divided into three layers: Serving Layer, Speed Layer, and Batch Layer: -- Batch层: 对离线数据进行处理，最后提供 view 服务给到业务； -- Speed层: 对实时增量数据进行处理，最后提供 view 服务给到业务； -- Serving层: 响应用户的请求，实现离线和增量数据的聚合计算，并最终提供服务； +- Batch Layer: Processes offline data and eventually provides a view service to the business; +- Speed Layer: Processes real-time incremental data and eventually provides a view service to the business; +- Serving Layer: Responds to user requests, performs aggregation calculations on offline and incremental data, and ultimately provides the service; -优点是：离线和实时分开计算，使用两套框架，架构稳定 +The advantage is that offline and real-time computations are separated, using two sets of frameworks, which makes the architecture stable. -缺点是：离线和实时数据很难保持一致性，运维人员需要维护两套框架三层架构，开发人员需要写三套代码 +The disadvantage is that it is difficult to maintain consistency between offline and real-time data, and operational staff need to maintain two sets of frameworks and three layers of architecture. Developers need to write three sets of code. -选型依据：数据一致性不可控；运维、开发工作量大，不符合人员精简配置的原则； +Selection Basis: Data consistency is uncontrollable; operations and development require significant workload, which does not conform to the principle of streamlined staffing. -### **kappa架构** +### **Kappa Architecture** -kappa 架构只用一套数据流处理架构来解决离线和实时数据，用实时流来解决所有问题，旨在提供快速可靠的查询访问结果。它非常适合各种数据处理工作负载，包括连续数据管道、实时数据处理、机器学习模型和实时数据分析、物联网系统以及许多其他具有单一技术堆栈的用例。 +The Kappa Architecture uses a single stream-processing framework to address both offline and real-time data, solving all problems with a real-time stream, with the goal of providing fast and reliable query access to results. It is highly suitable for various data processing workloads, including continuous data pipelines, real-time data processing, machine learning models and real-time analytics, IoT systems, and many other use cases with a single technology stack. -它通常使用流处理引擎实现，例如Apache Flink、Apache Storm、Apache Kinesis、 Apache Kafka，旨在处理大量数据流并提供快速可靠的查询访问结果。 +It is typically implemented using a streaming processing engine such as Apache Flink, Apache Storm, Apache Kinesis, Apache Kafka, designed to handle large data streams and provide fast and reliable query results.  -**优点是：**单数据流处理框架 +**Advantages: **Single data stream processing framework. -**缺点是：**虽然它的架构相对 lamabda 架构简单，但是流式处理框架的设置和维护相对复杂，不具备真正意义上的离线数据处理能力；流平台中存储大数据成本高昂 +**Disadvantages: **Although its architecture is simpler compared to Lambda Architecture, the setup and maintenance of the streaming processing framework are relatively complex, and it lacks true offline data processing capabilities; storing large amounts of data in streaming platforms can be costly. -选型依据：离线数据处理能力需要保留，控制成本 +Selection Basis: The capability for offline data processing needs to be retained, and costs controlled. ### **Iceberg** -为此我们也调研了 [Apache Iceberg](https://github.com/apache/Iceberg) ，它的快照功能一定程度上能够实现流批一体，但是它的问题是基于 kafka 做的实时表中间层不可查或者无法复用已经存在的表，对 kafka 有强依赖，需要利用 kafka 将中间结果写到 iceberg 表，增加了系统的复杂度和可维护性。 +Therefore, we also researched [Apache Iceberg](https://github.com/apache/iceberg), whose snapshot feature can to some extent achieve streaming-batch integration. However, the issue with it is that the real-time table layer based on Kafka is either not queryable or cannot reuse existing tables, with a strong dependency on Kafka. It requires the use of Kafka to write intermediate results to Iceberg tables, increasing system complexity and maintainability. -选型依据：无 kafka 实时架构已落地，中间数据无法实现可查可复用 +Selection Basis: A Kafka-free real-time architecture has been implemented, intermediate data cannot be made queryable or reusable. -### **流式数仓（kappa架构的延续)** +### **Streaming Data Warehouse (Continuation of the Kappa Architecture)** -海程邦达大数据团队自 FTS0.3.0 版本开始参与流式数仓建设，旨在进一步降低数据处理框架的复杂度和人员的精简配置，前期的宗旨是既然是趋势就要参与进来，过程中不断学习精进，向最前沿的技术靠拢，团队一致认为有坑就踩坑，摸着石头也要过河，好在经过几个版本的迭代，在社区的高效配合下，最开始出现的问题也慢慢得以解决 +Since the FTS0.3.0 version, the BonDex big data team has participated in the construction of a streaming data warehouse, aiming to further reduce the complexity of the data processing framework and streamline personnel configuration. The initial principle was to get involved with the trend, to continuously learn and improve, and to move closer to cutting-edge technology. The team unanimously believes that it is essential to embrace challenges and "cross the river by feeling the stones." Fortunately, after several iterations, the problems that initially arose have been gradually resolved with the efficient cooperation of the community. -**流式数仓架构如下：** +**The architecture of the streaming data warehouse is as follows:**  -延续了 kappa 架构的特点，一套流处理架构，好处在与，底层 Paimon 的技术支撑使得数据在全链路可查，数仓分层架构得以复用，同时兼顾了离线和实时的处理能力，减少存储和计算的浪费 +Continuing the characteristics of the Kappa architecture with a single stream processing framework, the advantage lies in the fact that the underlying Paimon technology support makes the data traceable throughout the entire chain. The data warehouse layer architecture can be reused, while also considering the processing capabilities of both offline and real-time data, reducing the waste of storage and computing resources. -## 03 生 产 实 践 +## 03 Production Practices -本方案采用 Flink Application On K8s 集群，Flink CDC 实时摄取业务系统关系型数据库数据，通过 StreamPark 任务平台提交 Flink + Paimon Streaming Data Warehouse 任务， 最后采用 Trino 引擎接入 Finereport 提供服务和开发人员的查询。Paimon 底层存储支持 S3 协议，因为公司大数据服务依赖于阿里云所以使用对象存储OSS作为数据文件系统。 +This solution adopts Flink Application on K8s clusters, with Flink CDC for real-time ingestion of relational database data from business systems. Tasks for Flink + Paimon Streaming Data Warehouse are submitted through the StreamPark task platform, with the Trino engine ultimately used to access Finereport for service provision and developer queries. Paimon's underlying storage supports the S3 protocol, and as the company's big data services rely on Alibaba Cloud, Object Storage Service (OSS) is used as the data filesystem. -[StreamPark](https://github.com/apache/incubator-streampark) 是一个实时计算平台，与 [Paimon](https://github.com/apache/incubator-paimon) 结合使用其强大功能来处理实时数据流。此平台提供以下主要功能： +[StreamPark](https://github.com/apache/incubator-streampark) is a real-time computing platform that leverages the powerful capabilities of [Paimon](https://github.com/apache/incubator-paimon) to process real-time data streams. This platform offers the following key features: -**实时数据处理：**StreamPark 支持提交实时数据流任务，能够实时获取、转换、过滤和分析数据。这对于需要快速响应实时数据的应用非常重要，例如实时监控、实时推荐和实时风控等领域。 +**Real-time Data Processing: **StreamPark supports the submission of real-time data stream tasks, capable of real-time acquisition, transformation, filtering, and analysis of data. This is extremely important for applications that require rapid response to real-time data, such as real-time monitoring, real-time recommendations, and real-time risk control. -**可扩展性：**可以高效处理大规模实时数据，具备良好的可扩展性。可以在分布式计算环境中运行，并能够自动处理并行化、故障恢复和负载均衡等问题，以确保高效且可靠地处理数据。 +**Scalability: **Capable of efficiently processing large-scale real-time data with good scalability. It can operate in a distributed computing environment, automatically handling parallelization, fault recovery, and load balancing to ensure efficient and reliable data processing. -**Flink 集成：**基于 [Apache Flink](https://github.com/apache/flink) 构建，利用 Flink 的强大流处理引擎来实现高性能和鲁棒性。用户可以充分利用 Flink 的特性和生态系统，如广泛的连接器、状态管理和事件时间处理等。 +**Flink Integration: **Built on [Apache Flink](https://github.com/apache/flink), it harnesses Flink’s powerful stream processing engine for high performance and robustness. Users can fully utilize the features and ecosystem of Flink, such as its extensive connectors, state management, and event-time processing. -**易用性：**提供了直观的图形界面和简化的 API，可以轻松地构建和部署数据处理任务，而无需深入了解底层技术细节。 +**Ease of Use: **Provides an intuitive graphical interface and simplified API, enabling easy construction and deployment of data processing tasks without needing to delve into underlying technical details. Review Comment: Provides a straightforward graphical interface -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
