2022/05/25 Minutes of the bi-weekly meeting of Apache Linkis(incubating) 1. [Fixed topic] Apache Linkis incubation & version progress synchronization. —— Xu Jie https://docs.qq.com/sheet/DSFJyTld3Y0JGeU54?tab=uf5xax 2. [Fixed topic] Apache Linkis1.1.1 progress synchronization. —— Xia Chen https://docs.qq.com/sheet/DVWZLYlFVTWVrdmlr The voting is over, an email will be issued on 2022-05-25, and a soft article will be issued on the public account this week, which is basically completed. 3. [Fixed topic] Apache Linkis1.1.2 progress synchronization. - Tok Road 4. [Fixed topic] Apache Linkis1.2.0 progress synchronization. —— Wang Zhen Code has been merged, documentation and test cases are being prepared In addition, the 1.1.3 new feature code has been submitted, the unit test has been supplemented, and the documentation is being improved, hosted by Sun Shun 5. [Fixed topic] Apache Linkis community operation progress is synchronized. —— Li Wen The operating indicators of the open source community are growing normally, and the growth of core indicators such as committer and pr is in line with expectations. Judging from the tweets of the public account issued in the past two weeks, developers are more concerned about the technical level, as well as the components related to Linux or WDS. Some new editions are released, and some articles about other activities and news articles are of general interest. There are 28 contributors and 80 registrations in total, and further contact is required. The video on how to become a developer has been launched on websites such as Station B. It is recommended to start with a simpler way of documentation. Already 7 volunteers have started contributing to the community through documentation. Xingce community meetup, the number of viewers reached 4000+, but it did not bring traffic growth to the community. On the evening of Thursday, June 9th, a community meetup will be held, which will be postponed for a week until June 15th. 6. [Temporary Issue] Test Environment Whitelist Access Mechanism —— Wang Heping For security reasons, the existing test environment cannot be connected normally, and external resources such as Alibaba Cloud need to be purchased for redeployment, and it is necessary to further communicate resource application issues within Wezhong. 7. [Temporary topic] Apache Linkis 1.1.3 feature introduction (Prometheus monitoring) - Sun Shun It mainly introduces the monitoring architecture and deployment plan of Linkis Prometheus. The main monitoring data is in JVM and NETWORK. If necessary, Tianyi Cloud can contribute to the monitoring of indicators that are closer to the business, such as the number of successful and failed tasks. 8. [Temporary Issue] Linkis containerization starts - Tao Kelu Linis containerization starts to start development, and Tianyi Cloud can provide some codes and ideas to cooperate. 9. [Temporary Issue] Based on the linkis&dss 1.X version, the adaptation of peripheral components - Di Shuai 10. [Fixed topic] The host of the next regular meeting, welcome to claim it. Postponed for a week, held on June 15, hosted by Hui Ge 11. [Temporary Issue] Answers to developer questions https://docs.qq.com/sheet/DUlJIREJKaHlVVUVU?u=444994d9ce1841d3af5bb0d65efbe4a9&tab=ggfj6i For resource usage issues, it is recommended that users upgrade from 1.0.2 to 1.0.3 or later, and 1.1.2 will be the long-term support version. k8s deployment linkis exists. Due to the overlap network, the service ip and port are random. When restarting, linkis saves the eurka address in the local database, resulting in the address before the connection after the restart. It is recommended not to keep unnecessary information in the database. , The data source runs through the entire data governance. If it is configured in Linkis, it is not conducive to expansion and use. The main Linkis data source is for tool development and use, not the data source used by the company.
Configuration parameter change Whether it is necessary to add a md file of parameter change record in the source code library Recent developments merged 1. https://github.com/apache/incubator-linkis/pull/2168/files FileSource supports variable configuration for file types Jie Longping has been merged 2. https://github.com/apache/incubator-linkis/issues/2124 Optimize the result set path to be separated by date, to solve the problem of too many subdirectories in a single folder. The result set path of different dates is in the same folder, such as "/tmp /linkis/hadoop/linkis/20220516_210525/IDE/40099", which may result in too many files in a folder. The number of merged hdfs directories is limited. 3. https://github.com/apache/incubator-linkis/pull/2109 Add support for sqoop engine plugin Merged 4. https://github.com/apache/incubator-linkis/issues/2103 Fixed a bug where kerberos was not used, and the kinit thread was started when executing JDBC engine tasks. Merged 5. https://github.com/apache/incubator-linkis/issues/2110 Removed the binary file .mvn/wrapper/maven-wrapper.jar in the source code, and adjusted the LICENSE instructions related to .mvn/* 6. https://github.com/apache/incubator-linkis/pull/2113 Upgrade py4j-0.10.7-src.zip to py4j-0.10.9.5-src.zip 7. https://github.com/apache/incubator-linkis/pull/2116 linkis-storage module replaces cglib with spring built-in cglib 8. https://github.com/apache/incubator-linkis/pull/2131 Remove the introduction of pandas to solve the problem that the python engine fails to start due to lack of dependencies 9. https://github.com/apache/incubator-linkis/pull/2133 The temporary storage path of data source kafka and hive is added to check the function of automatically creating a directory 10. https://github.com/apache/incubator-linkis/pull/2142 Fix the problem that the JDBC Engine console configuration cannot take effect immediately after modification (the cache time is adjusted as a configuration item) 11. https://github.com/apache/incubator-linkis/pull/2160 The consumption queue for task submission supports the configuration of specific high-volume users 12. https://github.com/apache/incubator-linkis/pull/2161 Added support for automatic formatting parameters when exporting the result set to an excel file to be merged 1. https://github.com/apache/incubator-linkis/pull/2173 Add support for presto engine plugin 2. https://github.com/apache/incubator-linkis/pull/2164 entrance The parameter that supports the number of retries of the task - RetryCountLabel Asf header is missing 3. https://github.com/apache/incubator-linkis/pull/2163 Add task and execution EC records, and EC information is recorded in the task's Metrics field 4. https://github.com/apache/incubator-linkis/pull/2159 EC's log log supports scrolling by size and time 5. https://github.com/apache/incubator-linkis/pull/2150 The common and entry modules both have the logic of custom variable substitution, and the optimization is aggregated into the common module for processing 6. https://github.com/apache/incubator-linkis/pull/2147 The gson of dependabot is upgraded from 2.8.5 to 2.8.9 7. https://github.com/apache/incubator-linkis-website/pull/265 Supplementary engine implementation details document to be merged Build failed issue 1. https://github.com/apache/incubator-linkis/issues/2144 The problem of shell script authorization +x permission failure occurs when compiling different systems. Tao Kelu follow up 2. https://github.com/apache/incubator-linkis/issues/2141 Change dbcp in JDBC engine to dbcp2 --------------------- 2022/05/25 Apache Linkis(incubating) 双周例会会议纪要 1. 【固定议题】Apache Linkis 孵化&版本 进展同步。 —— 徐杰 https://docs.qq.com/sheet/DSFJyTld3Y0JGeU54?tab=uf5xax 2. 【固定议题】Apache Linkis1.1.1 进展同步。 —— 夏晨 https://docs.qq.com/sheet/DVWZLYlFVTWVrdmlr 投票结束，2022-05-25发版邮件，这周发公众号软文，基本完成。 3. 【固定议题】Apache Linkis1.1.2 进展同步。 —— 陶克路 测试完成，文档完成，等待1.1.1发版完成 4. 【固定议题】Apache Linkis1.2.0 进展同步。 —— 王震 代码已完成合并，文档与测试用例正在准备准备 另外1.1.3 新特性代码已提交，补充单元测试，文档在完善，孙顺主持 5. 【固定议题】Apache Linkis 社区运营 进展同步。 —— 李文 开源社区运行指标增长正常，committer与pr等核心指标增长符合预期，从最近两周发出的公众号的推文来看，开发者们是比较关注技术层面的，还有Linux或者WDS相关的组件的一些新版本的发布，而对于其他的活动类的些软文类的文章关注度一般。 贡献者证书和摆台登记人数28人，总数为80人，需要进一步联系。 已经在B站等网站推出如何成为开发者视频，建议是从文档较为简单的方式开始。已经有7个志愿者通过文档开始给社区做贡献。 星策社区meetup，观看人数达到4000+，但是没有带给社区带来流量增长。 6月9日周四晚上，开展社区meetup活动，双周会延期一周至6月15日举行。 6. 【临时议题】测试环境白名单访问机制 —— 王和平 安全原因，现有的测试环境不能正常接入，需要采购如阿里云等外部资源进行重新部署，需要微众内部进一步进行沟通资源申请问题。 7. 【临时议题】Apache Linkis1.1.3特性介绍(Prometheus监控) —— 孙顺 主要介绍了Linkis Prometheus监控架构涉及及部署方案，主要监控的数据在JVM、NETWORK，如有需要天翼云这边可贡献成功与失败任务数等更贴近业务的指标监控。 8. 【临时议题】Linkis容器化开始启动 —— 陶克路 Linis容器化开始启动开发，天翼云可提供一些代码与思路配合。 9. 【临时议题】基于linkis&dss 1.X版本的，周边组件适配情况—— 邸帅 强哥汇报了各个系统的版本规划情况，及各个版本之间的依赖关系。 10. 【固定议题】下一场例会的主持人，欢迎认领。 延期一周，6月15日举行，辉哥主持 11. 【临时议题】开发者问题答疑 https://docs.qq.com/sheet/DUlJIREJKaHlVVUVU?u=444994d9ce1841d3af5bb0d65efbe4a9&tab=ggfj6i 针对资源使用问题，建议用户从1.0.2升级至1.0.3以上版本，1.1.2将作为为长期支持版本。 k8s部署linkis存在，由于采用overlap网络，服务ip和端口是随机的，再重启的时候，linkis把eurka地址存在本地数据库中，导致重启后还是连接之前地址，建议不保留不必要的信息在数据库中， 数据源贯穿整个数据治理，如果在linkis中配置，不利于扩展和使用，主要Linkis数据源是针对工具开发使用，而非公司统一使用的数据源。 配置参数变化 是否需要在源码库中增加一个参数变化记录的md文档 近期动态 已合并 1. https://github.com/apache/incubator-linkis/pull/2168/files FileSource中文件类型支持变量配置 介龙平 已合并 2. https://github.com/apache/incubator-linkis/issues/2124 优化结果集路径以日期分隔，解决单个文件夹子目录过多问题 不同日期的resustset路径在同一个文件夹，如“/tmp/linkis/hadoop/linkis/20220516_210525/IDE/40099”，可能会导致一个文件夹下文件太多 已合并 hdfs目录个数有限制 3. https://github.com/apache/incubator-linkis/pull/2109 添加sqoop引擎插件的支持 已合并 4. https://github.com/apache/incubator-linkis/issues/2103 修复了未使用 kerberos，在执行JDBC引擎任务时 kinit 线程启动的错误 已合并 5. https://github.com/apache/incubator-linkis/issues/2110 移除了源码中的二进制文件.mvn/wrapper/maven-wrapper.jar，调整.mvn/*相关的LICENSE说明 6. https://github.com/apache/incubator-linkis/pull/2113 升级 py4j-0.10.7-src.zip 至 py4j-0.10.9.5-src.zip 7. https://github.com/apache/incubator-linkis/pull/2116 linkis-storage 模块将 cglib 替换为spring 内置的cglib 8. https://github.com/apache/incubator-linkis/pull/2131 移除对pandas的引入，解决python引擎因为缺失依赖导致启动失败的问题 9. https://github.com/apache/incubator-linkis/pull/2133 数据源kafka与hive的临时存储路径增加检查自动创建目录功能 10. https://github.com/apache/incubator-linkis/pull/2142 修复JDBC Engine 控制台配置修改后无法立即生效的问题（cache时间调整为配置项） 11. https://github.com/apache/incubator-linkis/pull/2160 任务提交的消费队列支持配置特定大容量用户 12. https://github.com/apache/incubator-linkis/pull/2161 新增对结果集导出到 excel文件时，自动格式化参数的支持 待合并 1. https://github.com/apache/incubator-linkis/pull/2173 添加presto引擎插件的支持 2. https://github.com/apache/incubator-linkis/pull/2164 entrance 支持任务重试次数的参数- RetryCountLabel Asf头部缺失 3. https://github.com/apache/incubator-linkis/pull/2163 增加任务与执行EC的记录，EC信息记录到任务的 Metrics字段中 4. https://github.com/apache/incubator-linkis/pull/2159 EC的log日志支持按大小和时间切割滚动 5. https://github.com/apache/incubator-linkis/pull/2150 common和entrance模块都存在自定义变量替换的逻辑，优化聚集到common模块中处理 6. https://github.com/apache/incubator-linkis/pull/2147 dependabot的gson从2.8.5升级至2.8.9 7. https://github.com/apache/incubator-linkis-website/pull/265 引擎实现细节文档补充 待合并 构建失败 issue 1. https://github.com/apache/incubator-linkis/issues/2144 不同系统编译出现shell脚本授权+x权限失败问题 陶克路跟进 2. https://github.com/apache/incubator-linkis/issues/2141 将 JDBC 引擎中的 dbcp 更改为 dbcp2
