+1 Interesting! Linkis builds a computation middleware layer to reduce the difficulty for users to use multiple compute engines, and it seems that they have built a community around Links I'd like to see Links enter the Apache incubator for better development
Best Regards --------------- Apache DolphinScheduler PMC Chair David lidong...@apache.org Linkedin: https://www.linkedin.com/in/dailidong Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> --------------- On Fri, Jul 16, 2021 at 10:07 AM Xun Liu <liu...@apache.org> wrote: > +1 > > Linkis builds a layer of computing middleware between the upper application > and the underlying engine. > Solve the docking problems between various big data systems. > Linkis helps users to use Apache's Hadoop, hive, spark, zeppelin and other > systems more conveniently. > I think that allowing Linkis to enter Apache for better development can > also help other Apache Better use of the project. > I think Linkis is a very valuable project. > Good luck! > > Best regards, > Xun Liu > > On Thu, Jul 15, 2021 at 3:32 PM ShaoFeng Shi <shaofeng...@apache.org> > wrote: > > > I'd like to second this proposal. Linkis builds an active open source > > community and has been adopted by many companies. Bringing it to > > Apache software foundation can accelerate its development, and add value > to > > Apache. Good luck! > > > > Best regards, > > > > Shaofeng Shi 史少锋 > > Apache Kylin PMC > > Email: shaofeng...@apache.org > > > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > > > 俊平堵 <junping...@apache.org> 于2021年7月13日周二 上午9:32写道： > > > > > +1. > > > Linkis is an interesting project that builds a unified layer to > decouple > > > the upper layer computation/application and under layer data engines, > > such > > > as Spark, Presto, Flink, etc. Also it looks already build a community > > > around the project. > > > Good luck! > > > > > > > > > Shuai Di <shuaidi1...@163.com>于2021年7月12日 周一上午11:13写道： > > > > > > > Greetings! > > > > > > > > > > > > We would like to start an open discussion on bringing Linkis ( > > > > https://github.com/WeBankFinTech/Linkis), a computation middleware > > > > project, to the Apache Incubator. > > > > The proposal can be found below and is also listed in the Incubator > > wiki: > > > > https://cwiki.apache.org/confluence/display/INCUBATOR/LinkisProposal > , > > > > thanks @Junping Du for creating the page! > > > > We appreciate anyone who would give guidance or be willing to support > > us > > > a > > > > an additional mentor. > > > > ====== > > > > Linkis Proposal > > > > > > > > > > > > =Abstract= > > > > Linkis builds a computation middleware layer to decouple the upper > > > > applications and the underlying data engines, provides standardized > > > > interfaces (REST, JDBC, WebSocket etc.) to easily connect to various > > > > underlying engines (Spark, Presto, Flink, etc.), while enables cross > > > engine > > > > context sharing, unified job& engine governance and orchestration. > > > > Linkis codebase: https://github.com/WeBankFinTech/Linkis > > > > > > > > > > > > =Proposal= > > > > Linkis is designed to solve computation governance problems in > complex > > > > distributed environments (typically in a big data platform), where > you > > > have > > > > to deal with different types, versions, or clusters of underlying > data > > > > engines and hundreds of diversified engine clients at the upper > > > application > > > > layer as well. > > > > Linkis acts as a proxy between the upper applications layer and > > > underlying > > > > engines layer. By abstracting and implementing the 3 common phases > of a > > > > job/request for submit, prepare and execute, Linkis is able to > > facilitate > > > > the connectivity, governance and orchestration capabilities of > > different > > > > kind of engines like OLAP, OLTP (developing), Streaming, and handle > all > > > > these "computation governance" affairs in a standardized reusable > way. > > > > We are actively operating the Linkis community and we are looking > > forward > > > > to increase community activity continuously.We propose to contribute > > the > > > > Linkis codebase to the Apache Software Foundation. We believe that > > > bringing > > > > Linkis into Apache Software Foundation and following the > COMMUNITY-LED > > > > DEVELOPMENT "APACHE WAY" could continuously improve project quality > and > > > > community vitality. > > > > > > > > > > > > =Background= > > > > In today's complex and distributed environment, the communication, > > > > coordination and governance of application services have developed > > mature > > > > solutions from SOA to micro-services, and many practices from ESB to > > > > Service Mesh to decouple different services. > > > > However, things go different while an application service needs to > > > > communicate with the underlying engines. Engines are isolated from > each > > > > other, and the client-server tight coupling pattern goes everywhere. > > Each > > > > and every upper application has to directly connect to and access > > various > > > > underlying engines in a tightly coupled way, and solves the > > "computation > > > > governance" problems on its own, including maintaining different > client > > > > environments, submiting the job, monitoring job status, fetching the > > > > output, handling large number of concurrent client instances, > watching > > > the > > > > bad jobs, adapt to engine version changes, etc. > > > > It lacks a common layer of "computation middleware" between the > > numerous > > > > upper-layer applications and the countless underlying engines to > handle > > > all > > > > these "computation governance" affairs in a standardized reusable > way, > > > > that's why we started the Linkis project. > > > > Firstly, Linkis could reduce the complexity of connectivity. Instead > of > > > > maintaining a variety of engine client environments, users now only > > need > > > to > > > > install the Linkis client, or even just HTTP client while using the > > REST > > > > interface. Routing query to desired clusters could be done by simply > > > > providing a tag. > > > > Secondly, Links provides governance capabilities such as > multi-tenancy, > > > > concurrency control, resource management, query validation, privilege > > > > enhancement and auditing. > > > > Meanwhile, Linkis enables orchestration strategies such as routing, > > > > load-balance, active-active and hybrid computation across engines > (some > > > > still under development). > > > > > > > > > > > > =Rationale= > > > > Linkis is built on distributed microservice architecture with great > > > > scalability and extendibility. The enhancements of high concurrency > and > > > > fault tolerance make it more stable and reliable. It has already > > > supported > > > > many production environments with large number of daily jobs over a > > long > > > > term. > > > > Linkis's microservices are divided into 3 groups: Computation > > Governance > > > > Services, Public Enhancement Services, and Microservice Governance > > > Services. > > > > Computation Governance Services(CGS) group is responsible for the > core > > > > process of job/request submission, preparation and execution, > lifecycle > > > > management, resource management, validation and orchestration. > > > > Public enhancement Services(PES) group provides basic public > functions > > > > including job context sharing, material management and data source > > > > management, to serve other Linkis services and upper application > > systems. > > > > Microservice Governance Services(MGS) group includes customized > Spring > > > > Cloud Gateway, Eureka and Open Feign, to provide basic functions like > > > > routing, service registration and discovery, and RPC framework. > > > > By providing capabilities of multi-tenant, high concurrency, job > > > > dispatching/management policies, unified resource control and > > > > orchestration, Linkis makes the submission, preparation and execution > > of > > > > computation jobs more flexible, reliable and controllable, and > > > successfully > > > > return the results. It could greatly reduce the overall development, > > > > operation and maintenance costs, and the architecture complexity. > > > > Based on Linkis the computation middleware, new upper layer > > applications > > > > could be quickly developed by reusing the Linkis computation > governance > > > > functions, as what’s done in the open source big data platform suite > > > > “WeDataSphere” (https://github.com/WeBankFinTech/WeDataSphere). > > > > Linkis currently mainly supports OLAP and Streaming engines, and we > are > > > > planning to support OLTP engines better. Containerization is also one > > of > > > > the important development directions of Linkis. > > > > > > > > > > > > =Initial Goal= > > > > - Migrate the existing codebase, website, and documentation to > > > > Apache-hosted infrastructure. > > > > - Work with the infrastructure team to implement and approve our code > > > > review, build, and testing workflows in the context of the ASF. > > > > - Incremental development and release under Apache guidelines. > > > > - Grow and diversify the Linkis community in the Apache Way. > > > > > > > > > > > > =Current Status= > > > > ==Meritocracy== > > > > Linkis project was started at WeBank and has been an open-source > > project > > > > on GitHub since July 2019. Linkis has been quickly adopted by many > > > > organizations, more than 500 organizations have tested Linkis based > on > > > our > > > > sandbox application records, dozens of them have introduced Linkis > into > > > > production based on the users’ spontaneous feedbacks, distributed in > > > > various industries including banking, telecommunications, insurance, > > > > manufacturing, education, internet, etc. > > > > Linkis already has contributors and users from different companies. > > We’ve > > > > set up the Committer team and we’re constantly seeking for potential > > new > > > > committer. New Contributors are always highly welcomed and guided by > > > > existed committers. Users could get timely support from community IM > > > groups > > > > and GitHub. > > > > > > > > > > > > ==Community== > > > > Linkis now has 15 committers from 6 companies including WeBank, China > > > > Telecom, Kanzhun Ltd., iQIYI Inc., HONOR Mobile Phone, and Samoyed > > > Digital. > > > > We have a developer IM group for more than 100 people from different > > > > organizations, and 9 user IM groups for more than 4,500 people. > > > > > > > > > > > > ==Core Developers== > > > > The core developers of Linkis are working in the big data team of > > > > different companies, mainly in WeBank since the project was initiated > > > there. > > > > - Shuai Di (WeBank) > > > > - Qiang Yin (WeBank) > > > > - Heping Wang (WeBank) > > > > - Yongkun Yang (WeBank) > > > > - Zhiyue Yang (WeBank) > > > > - You Liu (WeBank) > > > > - XiaoGang Wang (China Telecom) > > > > - Hui Zhu (Kanzhun) > > > > - Zheng Wang (iQiyi) > > > > - Rong Zhang (Honor) > > > > > > > > > > > > ==Releases== > > > > Linkis has released multiple versions as listed here: > > > > https://github.com/WeBankFinTech/Linkis/releases > > > > We will follow the ASF guidelines more closely, and adopt the ASF > > source > > > > release process upon joining the incubator. > > > > > > > > > > > > ==Code Reviews== > > > > Linkis’s code reviews are currently public on Github: > > > > https://github.com/WeBankFinTech/Linkis/pulls . > > > > > > > > > > > > ==Alignment== > > > > As Linkis was built to address connectivity and other computation > > > > governance issues with various underlying engines, it depends on > > multiple > > > > ASF projects such as Spark, Flink, Hive and Hadoop. Linkis’s Engine > > > > Connector Manager service will start different Engine Connectors to > > > connect > > > > to different underlying engines, providing computation governance > > > abilities > > > > which benefits the usage and maintenance of these engines. Linkis > will > > > > continue to expand the types of engines it supports in ASF projects, > > such > > > > as HBase, Kylin, and more. > > > > > > > > > > > > =Known Risks= > > > > ==Orphaned Products== > > > > The risk of Linkis becoming an orphan product is very low, because > it’s > > > > already been the core infrastructure component in the production > > > > environments of dozens of companies' big data platforms, including > > large > > > > companies like WeBank, China Telecom, Ping An Insurance Company, > > > Hikvision, > > > > etc. Hundreds of thousands of computation jobs are performed through > > > Linkis > > > > in these companies everyday. Developers from these companies are > > > > increasingly joining the Linkis community as contributors. > > > > Linkis has 12 major releases so far, and received 355 PRs from > > > > contributors, which indicates the activity and vitality of the Linkis > > > > community. Linkis is also the core component of the open source big > > data > > > > platform suite “WeDataSphere”, even more users and developers are > > already > > > > active in this larger community. > > > > We are looking forward to further expand and diversify the community > by > > > > joining Apache. We are also further improving the adherence to the > > > > Community-Led development pattern, and the standardization and > > > transparency > > > > of community governance. > > > > > > > > > > > > ==Inexperience with Open Source== > > > > Linkis’s core developers have been running Linkis as a > > community-oriented > > > > open source project for a period of time, some of them already have > > > > experience working with other open source communities. The current > > Linkis > > > > user group scale of more than 4500 people is also a proof of our > > > commitment > > > > and passion for operating the open source community. > > > > Meanwhile, we’ve begun to refine our community governance efforts > under > > > > the guidance of Apache mentors, and we’ll learn more about how to > > operate > > > > the open source community effectively and properly by following the > > > Apache > > > > way in our incubator journey. > > > > > > > > > > > > ==Homogenous Developers== > > > > Most of the current core developers work at WeBank where the Linkis > > > > project started. We also had developers from China Telecom, Kanzhun, > > > iQiyi > > > > and Honor Mobile Phone elected to the committer group, and already > have > > > led > > > > the release of several versions of Linkis. Samoyed Digital has the > > latest > > > > nominated committer because of their solid contributions to Linkis > data > > > > source management module. > > > > Though Linkis community may not be diverse enough yet, we are > > constantly > > > > looking for new contributors and potential committers to enhance the > > > > diversity of the community and the vitality of the project. > > > > > > > > > > > > ==An Excessive Fascination with the Apache Brand== > > > > We acknowledge that the Apache brand would add a lot of value and > > > > reputation to Linkis, and will benefit the cooperation and promotion > at > > > the > > > > global scale. However, our primary purpose is to build a more diverse > > and > > > > viable community and to gain stability for long-term development as > > > > submitting Linkis to Apache. We will also strictly follow the ASF's > > rules > > > > and policies under the guidance of the Incubator PMC. > > > > > > > > > > > > =Documentation= > > > > Documentation about Linkis can be found at > > > > https://github.com/WeBankFinTech/Linkis-Doc . Following links > provide > > > > more information: > > > > - Codebase at Github: https://github.com/WeBankFinTech/Linkis > > > > - Issue Tracking: https://github.com/WeBankFinTech/Linkis/issues > > > > - Releases: https://github.com/WeBankFinTech/Linkis/releases > > > > =Initial Source= > > > > https://github.com/WeBankFinTech/Linkis > > > > > > > > > > > > =External Dependencies= > > > > Back-end: > > > > | Dependencies |License|Comment| > > > > |caffeine|Apache 2.0| > > > > | cglib | Apache 2.0 | > > > > | commons-beanutils | Apache 2.0 | > > > > | commons-codec | Apache 2.0 | > > > > | commons-collections | Apache 2.0 | > > > > | commons-dbcp | Apache 2.0 | > > > > | commons-exec | Apache 2.0 | > > > > | commons-io | Apache 2.0 | > > > > | commons-lang3 | Apache 2.0 | > > > > | commons-math3 | Apache 2.0 | > > > > | commons-net | Apache 2.0 | > > > > | commons-text | Apache 2.0 | > > > > | dozer-core | Apache 2.0 | > > > > | druid | Apache 2.0 | > > > > | fastjson | Apache 2.0 | > > > > | gson | Apache 2.0 | > > > > | guava | Apache 2.0 | > > > > | hadoop-auth | Apache 2.0 | > > > > | hadoop-client | Apache 2.0 | > > > > | hadoop-common | Apache 2.0 | > > > > | hadoop-hdfs | Apache 2.0 | > > > > | hadoop-yarn-client | Apache 2.0 | > > > > > > > > | hive-common | Apache 2.0 | > > > > | hive-exec | Apache 2.0 | > > > > | hive-jdbc | Apache 2.0 | > > > > | httpclient | Apache 2.0 | > > > > | httpmime | Apache 2.0 | > > > > | jackson-annotations | Apache 2.0 | > > > > | jackson-databind | Apache 2.0 | > > > > | jackson-module-scala | Apache 2.0 | > > > > | javacsv | LGPL | > > > > | jaxrs-ri | CDDL, GPL 1.1 | will remove | > > > > | jersey-container-servlet | CDDL, GPL 1.1 | will remove | > > > > | jersey-container-servlet-core | CDDL, GPL 1.1 | will remove | > > > > | jersey-entity-filtering | CDDL, GPL 1.1 | will remove | > > > > | jersey-json | CDDL, GPL 1.1 | will remove | > > > > | jersey-media-json-jackson | CDDL, GPL 1.1 | will remove | > > > > | jersey-media-multipart | CDDL, GPL 1.1 | will remove | > > > > | jersey-server | CDDL, GPL 1.1 | will remove | > > > > | jersey-servlet | CDDL, GPL 1.1 | will remove | > > > > | jersey-spring3 | CDDL, GPL 1.1 | will remove | > > > > | jetty-server | Apache 2.0, EPL 1.0 | > > > > | jetty-webapp | Apache 2.0, EPL 1.0 | > > > > | json4s-jackson | Apache 2.0 | > > > > | jsp-api | CDDL, GPL 2.0 | will remove | > > > > | junit | EPL 1.0 | > > > > | libthrift | Apache 2.0 | > > > > | log4j-1.2-api | Apache 2.0 | > > > > | log4j-api | Apache 2.0 | > > > > | log4j-core | Apache 2.0 | > > > > | log4j-slf4j-impl | Apache 2.0 | > > > > | mockito-all | MIT | > > > > | mybatis-plus-boot-starter | Apache 2.0 | > > > > | mysql-connector-java | GPL 2.0 | will remove | > > > > | netty-all | Apache 2.0 | > > > > | pagehelper | MIT | > > > > | poi-ooxml | Apache 2.0 | > > > > | protostuff-api | Apache 2.0 | > > > > | protostuff-core | Apache 2.0 | > > > > | protostuff-runtime | Apache 2.0 | > > > > | py4j | BSD 2-clause | > > > > | reactor-netty | Apache 2.0 | > > > > | reflections | BSD 2-clause | > > > > | scalacheck | BSD 3-clause | > > > > | scalacheck-shapeless | Apache 2.0 | > > > > | scala-compiler | Apache 2.0 | > > > > | scala-library | Apache 2.0 | > > > > | scalamock-scalatest-support | MIT | > > > > | scalap | Apache 2.0 | > > > > | scala-reflect | Apache 2.0 | > > > > | scalatest | Apache 2.0 | > > > > | slf4j-api | MIT | > > > > | spark-core | Apache 2.0 | > > > > | spark-hive | Apache 2.0 | > > > > | spark-repl | Apache 2.0 | > > > > | spark-sql | Apache 2.0 | > > > > | spark-testing-base | Apache 2.0 | > > > > | spoiwo | MIT | > > > > | spring-boot | Apache 2.0 | > > > > | spring-boot-actuator-autoconfigure | Apache 2.0 | > > > > | spring-boot-starter | Apache 2.0 | > > > > | spring-boot-starter-actuator | Apache 2.0 | > > > > | spring-boot-starter-aop | Apache 2.0 | > > > > | spring-boot-starter-cache | Apache 2.0 | > > > > | spring-boot-starter-jetty | Apache 2.0 | > > > > | spring-boot-starter-log4j2 | Apache 2.0 | > > > > | spring-boot-starter-quartz | Apache 2.0 | > > > > | spring-boot-starter-reactor-netty | Apache 2.0 | > > > > | spring-boot-starter-web | Apache 2.0 | > > > > | spring-cloud-commons | Apache 2.0 | > > > > | spring-cloud-config-client | Apache 2.0 | > > > > | spring-cloud-context | Apache 2.0 | > > > > | spring-cloud-gateway-core | Apache 2.0 | > > > > | spring-cloud-starter | Apache 2.0 | > > > > | spring-cloud-starter-config | Apache 2.0 | > > > > | spring-cloud-starter-gateway | Apache 2.0 | > > > > | spring-cloud-starter-netflix-eureka-client | Apache 2.0 | > > > > | spring-cloud-starter-netflix-eureka-server | Apache 2.0 | > > > > | spring-cloud-starter-openfeign | Apache 2.0 | > > > > | spring-core | Apache 2.0 | > > > > | spring-jdbc | Apache 2.0 | > > > > | spring-security-crypto | Apache 2.0 | > > > > | spring-test | Apache 2.0 | > > > > | spring-tx | Apache 2.0 | > > > > | spring-web | Apache 2.0 | > > > > | websocket-client | Apache 2.0, EPL 1.0 | > > > > | websocket-server | Apache 2.0, EPL 1.0 | > > > > | xlsx-streamer | Apache 2.0 | > > > > > > > > | xstream | BSD 3-clause | > > > > > > > > > > > > Front-end: > > > > |Dependencies|License|Comment| > > > > | axios | MIT | > > > > | highlight.js | BSD-3-Clause | > > > > | iview | MIT | > > > > > > > > | lodash | MIT | > > > > | moment | MIT | > > > > | monaco-editor | MIT | > > > > | sql-formatter | MIT | > > > > | svgo | MIT | > > > > | vue | MIT | > > > > | vue-i18n | MIT | > > > > | vue-router | MIT | > > > > | vuedraggable | MIT | > > > > | vuescroll | MIT | > > > > > > > > =Required Resources= > > > > ==Mailing List== > > > > Currently Linkis has no mailing list. The usual mailing lists are > > > expected > > > > to be set up when entering incubation: > > > > - private@linkis.incubator.apache.orgfor PPMC discussions; > > > > - d...@linkis.incubator.apache.org for development discussions; > > > > - notificat...@linkis.incubator.apache.org for user notifications, > and > > > > notifications from GitHub. > > > > > > > > ==Git Repositories== > > > > Upon entering incubation, we request to move the existing repository > > from > > > > https://github.com/WeBankFinTech/Linkis to Apache infrastructure > like > > > > https://github.com/apache/Incubator-Linkis. > > > > > > > > ==Issue Tracking== > > > > The Linkis community would like to continue using GitHub Issues if > > > > possible. > > > > > > > > ==Other Resources== > > > > Apache Jenkins > > > > > > > > =Source and Intellectual Property Submission Plan= > > > > Most of the current code is Apache 2.0 licensed and the copyright is > > > > assigned to WeBank. If the project enters incubator, WeBank will > > transfer > > > > the source code & trademark ownership to ASF via a Software Grant > > > Agreement. > > > > > > > > > > > > > > > > =Initial Committers= > > > > - Shuai Di (shuaidi1...@163.com) > > > > - Qiang Yin (690574...@qq.com) > > > > - Heping Wang (374126...@qq.com) > > > > - Yongkun Yang (wimkun...@gmail.com) > > > > - Zhiyue Yang (904666...@qq.com) > > > > - You Liu (405240...@qq.com) > > > > - Deyi Hua (david_hua1...@hotmail.com) > > > > - Le Bai (120190...@qq.com) > > > > - Xiaogang Wang (913546...@qq.com) > > > > - Hui Zhu (46580...@qq.com) > > > > - Zhen Wang (643348...@qq.com) > > > > - Rong Zhang (693404...@qq.com) > > > > - Xiaohua Yi (405078...@qq.com) > > > > - Ke Zhou (zhouke...@vip.qq.com) > > > > - Jian Xie (xj...@163.com) > > > > > > > > =Affiliations= > > > > Shuai Di, Qiang Yin, Heping Wang, Yongkun Yang, Zhiyue Yang, You Liu, > > > Deyi > > > > Hua, Le Bai, Ke Zhou and Jian Xie of the initial committers are > > employees > > > > of WeBank. > > > > Xiaogang Wang of the initial committers is an employee of China > > Telecom. > > > > Hui Zhu of the initial committers is an employee of Kanzhun. > > > > Zhen Wang of the initial committers is an employee of iQiyi. > > > > Rong Zhang of the initial committers is an employee of HONOR Mobile > > > Phone. > > > > Xiaohua Yi of the initial committers is an employee of Samoyed > Digital. > > > > > > > > =Sponsors= > > > > ==Champion== > > > > Junping_Du (ASF Member, IPMC Member), junping...@apache.org > > > > > > > > ==Nominated Mentors== > > > > Shao Feng Shi (ASF Member, IPMC Member), shaofeng...@apache.org > > > > Duo Zhang (ASF Member, IPMC Member), zhang...@apache.org > > > > Jerry Shao (ASF Member, IPMC Member), js...@apache.org > > > > Lidong Dai (IPMC Member), lidong...@apache.org > > > > > > > > =Sponsoring Entity= > > > > We request the Apache Incubator to sponsor this project. > > > > ====== > > > > Best Regards, > > > > Shuai Di > > > > > >
