2023-07-05 Minutes of the bi-weekly meeting of Apache Linkis
1. [Fixed topic] Apache Linkis 1.4.0 version progress. ——Guo Fei/Brother
Yue Chao/Brother Chenghui
a.The engine supports different version compatibility tests, the Hive
engine supports concurrent tasks, S3 has been tested and OSS, ECM stateless
tests have been completed, Linkis metadata support PG tests have been
completed, and Spark ETL multiple data sources have been tested. Multiple
Hadoop clusters are supported Not yet tested.
b.1.4.0 The document needs to be supplemented
2. [Fixed topic] Apache Linkis 1.5.0 version progress. ——Yue Chao/Cheng
Jie/Guo Fei
Version plan: https://github.com/apache/linkis/projects/27
a.On k8s progress: flink native k8s application mode and session mode
have been completed, spark k8s operator has been completed, flink k8s
operator is under development, and spark native submission k8s is being
developed by Chenghui and his classmates from Open Source Summer
b.Flink upgrade 1.16 progress: expected to be completed by the end of
July
c. Data source generation sql progress: JDBC DDLand spark DDL have
been completed, flink DDL was developed by Defu
Service Consolidation: PES Common Module Completed
3. [Fixed topic] WDS version synchronization —— Hao Jinfu
4. [Fixed topic] Synchronized progress of Open Source Summer —— Hao Jinfu
Support K8s cluster to submit task progress ——Zhao Lingxiang
a.ClusterLabel identification: create different resource types
according to the cluster type, you can implement ModifyLabel
b.Resource module: realize the resource type control of K8S
c. Spark task starts K8S: in progress
d.Basic Environment Availability Testing Progress ——Zhao Wenkai
a.The progress of the reconstruction of the management platform ——
Shen Yuyou
b.The design draft needs to be synchronized
c.Discussion on the use of BDP-Design——Mei Yonghao
5. [Fixed topic] The progress of Apache Linkis community operation is
synchronized. —— Li Wen
(1) Data growth: Issue / PR 80+, Star / Fork 35+, community users/sandbox
experience 60+, official account fans 40+;
(2) Operational matters:
●Content Release: "How Apache Linkis Became the Base of Data Application
Development", "[Activity Review] Exploration and Practice of Apache Linkis
Integration with OceanBase"
6. [Fixed topic] The host of the next regular meeting, welcome to claim ---
Brother Guo Fei
7. [Temporary topic] Version branch management, using Master for
management--Wang Heping
1. [Authentication Enhancement] Linkis permission authentication support is
handed over to upper application control Brother Wen Jun
Linkis in extension calls the authentication interface of the upper layer
application
2. [Enhancement of Data Source Permissions] User permission management of
data sources Brother Xiutao
3.【Enhancement of public capabilities】Addition of data lineage and sql
analysis functions, brother Tianzheng
4. [Python Enhancement] Python and Pyspark support direct execution of
Python files Brother Tianzheng
5. Orchestrator supports pluggable, Entrance supports consumption queue
storage Redis Brother Shupei
6. Once mode task supports recording task record information through
Entrance Brother Shupei
7. There is only one dependent package in the SDK merger, which will check
for leaks and fill in the gaps. Brother Shupei
8. How to start the test of Flink on k8s for integration test Jiexiong Cheng
9. It is planned to support the functions of some engine extra toolkits
(including package management, version management, package sharing, and
other parts. Such as third-party packages of Python, and extra jar packages
of spark) --Xia Chen
10. The linkis management console plans to add operation and maintenance
tools to support some simple scene-based operation and maintenance
operations (for example, administrators can configure and view other user
resource configurations, etc.) --Xia Chen
list LKIP
1. 【固定议题】Apache Linkis 1.4.0版本进展。 —— 郭飞兄/越超兄/呈辉兄
引擎支持不同的版本兼容测试，Hive引擎并发任务的支持，S3已经完成测试和OSS，ECM无状态测试完成，Linkis元数据支持PG测试完成，Spark
ETL 多个数据源已经完成测试，支持多Hadoop集群还未测试。
1.4.0 文档情况还需要补充下
2. 【固定议题】Apache Linkis 1.5.0版本进展。 —— 越超兄/程杰/郭飞
版本计划：
on k8s进度：flink native k8s application 模式和session模式已完成，spark k8s
operator已完成， flink k8s operator正在开发中，spark原生提交k8s由呈辉哥带着开源之夏的同学正在开发
flink升级1.16进度：预计七月底前完成
数据源生成sql进度：jdbc ddl和spark ddl已完成，flink ddl由得府同学开发
服务合并：PES Common模块已经完成
3. 【固定议题】WDS版本同步 ——郝金福
4. 【固定议题】开源之夏进展同步 ——郝金福
支持 K8s 集群提交任务进度 ——赵凌翔
ClusterLabel识别：根据集群类型创建不同的资源类型，可以通过实现ModifyLabel
资源模块: 实现K8S的资源类型管控
Spark任务启动K8S：正在进行中
基础环境可用性检测进度 ——赵文恺
管理台重构进度 ——沈俞佑
设计稿需要同步
BDP-Design使用讨论 ——梅永浩
5. 【固定议题】Apache Linkis 社区运营 进展同步。 —— 李文
（1）数据增长情况：Issue / PR 80+，Star / Fork 35+，社群用户/沙箱体验 60+，公众号粉丝 40+；
（2）运营事项：
● 内容发布：《Apache Linkis是如何成为数据应用开发基座的》、《【活动回顾】Apache Linkis集成OceanBase的探索与实践》
6. 【固定议题】下一场例会的主持人，欢迎认领 --- 郭飞兄
7. 【临时议题】版本分支管理，采用Master进行管理--王和平
1. 【认证增强】Linkis权限认证支持交给上层应用控制 文君兄
Linkis入扩展调用上层应用的认证接口
2. 【数据源权限增强】数据源的用户权限管理 秀涛兄
3. 【公共能力增强】数据血缘、sql解析功能的增加 天正兄
4. 【python增强】python和pyspark支持直接执行python文件 天正兄
5. Orchestrator支持可插拔、Entrance支持消费队列存储Redis 书培兄
6. Once模式任务支持经过Entrance记录任务记录信息 书培兄
7. SDK合并只有1个依赖包 会进行查漏补缺 书培兄
8. 集成测试如何调启Flink on k8s的测试 程杰兄
9. 计划支持某些引擎额外工具包的功能（包括包管理，版本管理，包共享等功能。如Python的第三方包，spark的额外的jar包） --夏晨
10. linkis 管理台计划新增运维工具来支持部分简单的场景化运维操作（如 管理员能配置查看其他用户资源配置 等） --夏晨
列出LKIP
