吴涛 <wutao.as.nevercha...@gmail.com> 于2020年6月8日周一 下午5:22写道：
> Hi, Willem, > > This is the historical issue we are going to resolve recently. > > We've reduced the incompatible modifications on facebook/rocksdb [1] where > pegasus-rocksdb was forked, and I believe we will soon no longer need to > maintain > this repo. It will be completely used as an external dependency. > > As for rdsn, since the original repo (fully MIT-licensed) > microsoft/rDSN [2] has been > unmaintained for a long time, we plan to merge the two repo together. We've > endeavored a lot > to improve and refactor rdsn. I'm not sure if we should ask microsoft for > any CLA for the > donation of our code. > I am afraid you need to donate this, you have to. Unmaintained can't change the fact, its IP belongs to the original team. Sheng Wu 吴晟 Twitter, wusheng1108 > > [1] https://github.com/facebook/rocksdb > [2] https://github.com/XiaoMi/rdsn > > Willem Jiang <willem.ji...@gmail.com> 于2020年6月8日周一 下午4:12写道： > > > Hi, > > > > I just went through the proposal and found there are two source code > > repos[1][2] which are forked. > > > > Are you planning to donate these two repos into Apache as a part of > > Pegasus project? > > It makes the donation of the Pegasus complicated as we need to address > > this two code repo belonging issue first. > > Normally we just contribute the patch to the upstream as it consume > > lots of resources if we maintain the forked repo. > > > > [1] https://github.com/XiaoMi/rdsn > > [2] https://github.com/XiaoMi/pegasus-rocksdb > > > > > > Willem Jiang > > > > Twitter: willemjiang > > Weibo: 姜宁willem > > > > On Tue, Jun 2, 2020 at 3:49 PM 吴涛 <wutao.as.nevercha...@gmail.com> > wrote: > > > > > > Dear Apache Incubator Community, > > > > > > I'd like to open up a discussion about incubating Pegasus at Apache. > Our > > proposal can be found at https://pegasus-kv.github.io/community/proposal > > and is also included below. > > > > > > We are looking for possible Champion if anyone would like to volunteer. > > Thanks a lot! > > > > > > Best regards > > > Tao Wu > > > > > > Pegasus Proposal > > > > > > == Abstract == > > > > > > Pegasus is a distributed key-value storage system that is designed to > be > > horizontally scalable, strongly consistent and high-performance. > > > > > > - Pegasus codebase: https://github.com/XiaoMi/pegasus > > > - Website: https://pegasus-kv.github.io > > > > > > == Proposal == > > > > > > Pegasus is a key-value database that delivers low-latency data access > > together with horizontal scalability, using hash-based partitioning. > > Pegasus uses PacificA protocol for strong consistency and RocksDB as the > > underlying storage engine. > > > > > > We propose to contribute the Pegasus codebase and associated artifacts > > (e.g., documentation, website content, etc.) to the Apache Software > > Foundation, and aim to build an open community around Pegasus’s continued > > development in the ‘Apache Way’. > > > > > > == Background == > > > > > > Apache HBase was recognized as mostly the only large-scale KV store > > solution in XiaoMi Corp until Pegasus came out in 2015. The original > > purpose of Pegasus was to solve the problems caused by HBase’s two-level > > architecture and implementation, including high latency because of Java > GC > > and RPC overhead of the underlying distributed filesystem, and long > > failover time because of single point of RegionServer and recovery > overhead > > of splitting and replaying the HLog files. > > > > > > Pegasus aims to fill the gap between Redis and HBase. As the former is > > in-memory, low latency, but does not provide a strong-consistency > > guarantee. And unlike the latter, Pegasus server is entirely written in > C++ > > and its read-write path relies merely on the local filesystem. > > > > > > Apart from performance requirements, we also need a storage system to > > ensure multiple-level data safety and support fast data migration among > > data centers, automatic load balancing, and online partition splitting. > > > > > > After investigating lots of existing storage systems in the open source > > world, we could hardly find a suitable solution to satisfy all the > > requirements. So the journey of Pegasus begins. > > > > > > === Rationale === > > > > > > Pegasus is a mature and active project which has been widely adopted in > > XiaoMi. After the initial release of open source project in 2017, we have > > seen a great amount of interest across a diverse set of users and > companies. > > > > > > Our experiences at committers and PMC members on other Apache projects > > have convinced us that having a long-term home at Apache foundation would > > be a great fit for the project, to ensure that processes and procedures > are > > in place to keep project and community ‘healthy’ and free of any > > commercial, political or legal faults. > > > > > > === Initial Goal === > > > > > > Move the existing codebase, website, documentation, and mailing lists > to > > Apache-hosted infrastructure. > > > Work with the infrastructure team to implement and approve our code > > review, build, and testing workflows in the context of the ASF. > > > Incremental development and releases along with Apache guidelines. > > > > > > == Current Status == > > > > > > Pegasus has been an open-source project on GitHub > > https://github.com/XiaoMi/pegasus since October 2017. > > > > > > === Meritocracy === > > > > > > The intent of this proposal is to start building a diverse developer > and > > user community around Pegasus following the ASF meritocracy model. We > plan > > to invite more people as committers if they contribute to this project. > > > > > > === Releases === > > > > > > Pegasus has undergone multiple public releases, listed here: > > https://github.com/XiaoMi/pegasus/releases. > > > > > > These old releases were not performed in the typical ASF fashion. We > > will adopt the ASF source release process upon joining the incubator. > > > > > > === Code Reviews === > > > > > > Pegasus’s code reviews are currently public on Github > > https://github.com/XiaoMi/pegasus/pulls. > > > > > > === Community === > > > > > > Pegasus seeks to develop developer and user communities during > > incubation. > > > > > > === Core Developers === > > > > > > Currently most of the core developers of Pegasus are working in the > > KV-Storage Team of Xiaomi. Yingchun Lai is one of the Apache Kudu PMC > > members. Zuoyan Qin is an experienced open-source developer who created > > sofa-pbrpc in his last job in Baidu. Wei Huang is also an active > > contributor of Apache Doris (Incubating). > > > > > > - Zuoyan Qin (https://github.com/qinzuoyan) > > > - Yuchen He (https://github.com/hycdong) > > > - Tao Wu (https://github.com/neverchanje) > > > - Yingchun Lai (https://github.com/acelyc111) > > > - Wei Huang (https://github.com/vagetablechicken) > > > - Shuo Jia (https://github.com/Shuo-Jia) > > > - Liwei Zhao (https://github.com/levy5307) > > > > > > === Alignment === > > > > > > Pegasus is aligned with several other ASF projects. > > > > > > We are working on a new feature to load data from the HDFS filesystem. > > Pegasus can also generate and store checkpoints to HDFS, for both backup > > and analysis purpose. We currently support offline analysis on > checkpoints > > powered by Apache Spark. > > > > > > == Known Risks == > > > > > > === Orphaned Products === > > > > > > The core developers of XiaoMi’s Pegasus team work full time on this > > project. There is very little risk of Pegasus getting orphaned since at > > least one large company (XiaoMi) is extensively using it in production, > > with currently a scale of 70+ clusters, 800+ tables, and more than 70TB > > data. Furthermore, since Pegasus was open sourced at the beginning of > > October 2017, it has received more than 1200 stars and been forked more > > than 200 times, and also received some issues and pull requests from > > developers and users outside XiaoMi. We plan to extend and diversify this > > community further through Apache. > > > > > > === Inexperience with Open Source === > > > > > > The core developers are all active users and followers of open source. > > They are already committers and contributors to the Pegasus Github > project. > > All have been involved with the source code that has been released under > an > > open source license, and several of them also have experience developing > > code in an open source environment. > > > > > > Several of the developers in XiaoMi’s storage team are committers > and/or > > PMC members on other ASF projects (Kudu, HBase, Doris, etc.). They will > > guide others to practice the Apache Way together along with other > incubator > > mentors. > > > > > > === Homogenous Developers === > > > > > > The project has received some contributions from developers outside of > > XiaoMi, and is starting to attract a user community as well. We hope to > > continue to encourage contributions from these developers and community > > members, and grow them into committers as they have time to continue > their > > contributions. > > > > > > === Reliance on Salaried Developers === > > > > > > XiaoMi invested in Pegasus as a general key-value storage used in > > company widely. The core developers have been dedicated to this project > for > > nearly five years. > > > > > > Besides, we look forward to attracting more people outside XiaoMi to > > contribute to this project, either payed engineers working on storage > area, > > or individual volunteers, as long as they have enthusiasm for the Pegasus > > project. > > > > > > === An Excessive Fascination with the Apache Brand === > > > > > > Pegasus is proposing to enter incubation at Apache in order to help > > efforts to diversify the committer-base, not so much to capitalize on the > > Apache brand. The Pegasus project is in production use already inside > > XiaoMi, but is not expected to be a XiaoMi product for external > customers. > > As such, the Pegasus project is not seeking to use the Apache brand as a > > marketing tool. > > > > > > == Documentation == > > > > > > Information about Pegasus can be found at > > https://github.com/XiaoMi/pegasus. The following links provide more > > information about Pegasus in open source: > > > > > > - Pegasus Website: https://pegasus-kv.github.io > > > - Codebase at Github: https://github.com/XiaoMi/pegasus > > > - Issue Tracking: https://github.com/XiaoMi/pegasus/issues > > > - Releases: https://pegasus-kv.github.io/releases > > > - Community Guide: https://pegasus-kv.github.io/community > > > > > > == Initial Source == > > > > > > Besides the core codebase, Pegasus also hosts its side projects on > > Github under XiaoMi Group. Specifically, the initial source includes: > > > > > > Client libraries with different languages: > > > > > > - Java-Client: https://github.com/XiaoMi/pegasus-java-client > > > - Scala-Client: https://github.com/XiaoMi/pegasus-scala-client > > > - NodeJS-Client: https://github.com/XiaoMi/pegasus-nodejs-client > > > - Go-Client: https://github.com/XiaoMi/pegasus-go-client > > > - Python-Client: https://github.com/XiaoMi/pegasus-python-client > > > > > > Components of Pegasus: > > > > > > - rDSN: https://github.com/XiaoMi/rdsn > > > - RocksDB: https://github.com/XiaoMi/pegasus-rocksdb > > > > > > rDSN was initially a distributed framework developed by Zhenyu Guo from > > Microsoft, and we have heavily refactored and improved it to make it more > > fit for Pegasus. rDSN is MIT & Apache-2.0 dual-licensed. The code > licensed > > Apache-2.0 belongs to XiaoMi and the copyright of MIT-licensed code is > > assigned to Microsoft. It’s in our plan to merge Pegasus and rDSN as one > > project. > > > > > > RocksDB is a Facebook-developed storage engine. Pegasus added some > > enhancements and modifications that may be incompatible with the original > > implementation. RocksDB is licensed under Apache 2.0 License. > > > > > > == External Dependencies == > > > > > > Pegasus has the following external dependencies. > > > > > > - RocksDB (Apache) > > > - Apache Thrift (Apache Software License v2.0) > > > - Boost (Boost Software License) > > > - Apache Zookeeper (Apache) > > > - Google s2geometry (BSD) > > > - Google gflags (BSD) > > > - fmtlib (BSD) > > > - POCO (Boost Software License) > > > - rapidjson (Tencent) > > > - libevent (BSD) > > > - Google gperftools (BSD) > > > - cameron314/concurrentqueue (BSD) > > > - cameron314/readerwriterqueue (BSD) > > > - XiaoMi/galaxy-fds-sdk-cpp (No License) > > > - jupp0r/prometheus-cpp (MIT) > > > - curl (The curl license) > > > - nlohmann/json (MIT) > > > - abseil-cpp (Apache 2.0) > > > - antirez/linenoise (BSD-2) > > > - antirez/sds (BSD-2) > > > > > > Build and test dependencies: > > > > > > - Apache Maven (Apache Software License v2.0) > > > - cmake (BSD) > > > - Google gtest (Apache Software License v2.0) > > > > > > == Required Resources == > > > > > > === Mailing List === > > > > > > There are currently no mailing lists. The usual mailing lists are > > expected to be set up when entering incubation: > > > > > > - priv...@pegasus.incubator.apache.org > > > - d...@pegasus.incubator.apache.org > > > - comm...@pegasus.incubator.apache.org > > > > > > === Git Repositories === > > > > > > Upon entering incubation, we want to move the existing repository from > > https://github.com/XiaoMi/pegasus to Apache infrastructure like > > https://github.com/apache/incubator-pegasus. > > > > > > === Issue Tracking === > > > > > > Pegasus currently uses Github to track issues. Would like to continue > to > > do so while we discuss migration possibilities with the ASF Infra > committee. > > > > > > === Other Resources ==== > > > > > > The existing code already has unit tests so we will make use of > existing > > Apache continuous testing infrastructure. The resulting load should not > be > > very large. > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > Most of the current code is Apache 2.0 licensed and the copyright is > > assigned to XiaoMi. If the project enters incubator, XiaoMi will transfer > > the source code & trademark ownership to ASF via a Software Grant > Agreement. > > > > > > But due to historical issues, Pegasus was based on an MIT licensed code > > that was initially written by microsoft/rDSN, which has long been > actively > > developed by Pegasus because the original project is unmaintained > (modified > > code is licensed under Apache License 2.0). We aren’t sure if we should > > request Microsoft for any Contributor License Agreement (CLA) during the > IP > > clearance process. > > > > > > == Initial Committers == > > > > > > - Zuoyan Qin (https://github.com/qinzuoyan, qinzuoyan@xiaomi dot com) > > > - Weijie Sun (https://github.com/shengofsun, luckyweijie@gmail dot > com) > > > - Yuchen He (https://github.com/hycdong, heyuchen@xiaomi dot com) > > > - Tao Wu (https://github.com/neverchanje, wutao1@xiaomi dot com) > > > - Yingchun Lai (https://github.com/acelyc111, laiyingchun@xiaomi dot > > com) > > > - Wei Huang (https://github.com/vagetablechicken, huangwei5@xiaomi dot > > com) > > > - Shuo Jia (https://github.com/Shuo-Jia, jiashuo1@xiaomi dot com) > > > - Liwei Zhao (https://github.com/levy5307, zhaoliwei@xiaomi dot com) > > > - Liuyang Cai (https://github.com/LoveHeat) > > > > > > == Affiliations == > > > > > > Seven of the initial committers are employees of Xiaomi. > > > > > > == Sponsors == > > > > > > === Champion === > > > > > > TODO > > > > > > === Nominated Mentors === > > > > > > TODO > > > > > > === Sponsoring Entity === > > > > > > We are requesting the Incubator to sponsor this project. > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > >
