Hi Well I don't see issues as no one discuss the proposal. So I will start the official vote tomorrow. Cheers Olivier
On 6 February 2017 at 14:08, Olivier Lamy <ol...@apache.org> wrote: > Hello everyone, > I would like to submit to you a proposal to bring Gooblin to the Apache > Software Foundation. > The text of the proposal is included below and available as a draft here > in the Wiki: https://wiki.apache.org/incubator/GobblinProposal > > We will appreciate any feedback and input. > > Olivier on behalf of the Gobblin community > > > = Apache Gobblin Proposal = > == Abstract == > Gobblin is a distributed data integration framework that simplifies common > aspects of big data integration such as data ingestion, replication, > organization and lifecycle management for both streaming and batch data > ecosystems. > > == Proposal == > > Gobblin is a universal data integration framework. The framework has been > used to build a variety of big data applications such as ingestion, > replication, and data retention. The fundamental constructs provided by the > Gobblin framework are: > > 1. An expandable set of connectors that allow data to be integrated from > a variety of sources and sinks. The range of connectors already available > in Gobblin are quite diverse and are an ever expanding set. To highlight > just a few examples, connectors exist for databases (e.g., MySQL, Oracle > Teradata, Couchbase etc.), web based technologies (REST APIs, FTP/SFTP > servers, Filers), scalable storage (HDFS, S3, Ambry etc,), streaming data > (Kafka, EventHubs etc.), and a variety of proprietary data sources and > sinks (e.g.Salesforce, Google Analytics, Google Webmaster etc.). Similarly, > Gobblin has a rich library of converters that allow for conversion of data > from one format to another as data moves across system boundaries (e.g. > AVRO in HDFS to JSON in another system). > > > 2. Gobblin has a well defined and customizable state management layer > that allows writing stateful applications. These are particularly useful > when solving problems like bulk incremental ingest and keeping several > clusters replicated in sync. The ability to record work that has been > completed and what remains in a scalable manner is critical to writing such > diverse applications successfully. > > > 3. Gobblin is agnostic to the underlying execution engine. It can be > tailored to run ontop of a variety of execution frameworks ranging from > multiple processes on a single node, to open source execution engines like > MapReduce, Spark or Samza, natively on top of raw containers like Yarn or > Mesos, and the public cloud like Amazon AWS or Microsoft Azure. We are > extending Gobblin to run on top of a self managed cluster when security is > vital. This allows different applications that require different degrees > of scalability, latency or security to be customized to for their specific > needs. For example, highly latency sensitive applications can be executed > in a streaming environment while batch based execution might benefit > applications where the priority might be geared towards optimal container > utilization. > > 4.Gobblin comes out of the box with several diagnosability features like > Gobblin metrics and error handling. Collectively, these features allow > Gobblin to operate at the scale of petabytes of data. To give just one > example, the ability to quarantine a few bad records from an isolated Kafka > topic without stopping the entire flow from continued execution is vital > when the number of Kafka topics range in the thousands and the collective > data handled is in the petabytes. > > Gobblin thus provides crisply defined software constructs that can be used > to build a vast array of data integration applications customizable for > varied user needs. It has become a preferred technology for data > integration use-cases by many organizations worldwide (see a partial list > here). > > == Background == > > Over the last decade, data integration has evolved use case by use case in > most companies. For example, at LinkedIn, when Kafka became a significant > part of the data ecosystem, a system called Camus was built to ingest this > data for analytics processing on Hadoop. Similarly, we had custom pipelines > to ingest data from Salesforce, Oracle and myriad other sources. This > pattern became the norm rather than the exception and one point, LinkedIn > was running at least fifteen different types of ingestion pipelines. This > fragmentation has several unfortunate implications. Operational costs scale > with the number of pipelines even if the myriad pipelines share a vasty > array of common features. Bug fixes and performance optimizations cannot be > shared across the pipelines. A common set of practices around debugging and > deployment does not emerge. Each pipeline operator will continue to invest > in his little silo of the data integration world completely oblivious to > the challenges of his fellow operator sitting five tables down. > > These experiences were the genesis behind the design and implementation of > Gobblin. Gobblin thus started out as a universal data ingestion framework > focussed on extracting, transforming, and synchronizing large volumes of > data between different data sources and sinks. Not surprisingly, given its > origins, the initial design of Gobblin placed great emphasis on > abstractions that can be leveraged repeatedly. These abstractions have > stood the test of time at LinkedIn and we have been able to leverage the > constructs well beyond ingest. Gobblin's architecture has allowed us at > LinkedIn to use it for a variety of applications ranging from from optimal > format conversion to adhering to compliance policies set by European > standards. Finally, as noted earlier, Gobblin can be deployed in a variety > of execution environments: it can be deployed as a library embedded in > another application or can be used to execute jobs on a public cloud. A > fluid architectural and execution design story has allowed Gobblin to > become a truly successful data integration platform. > > Gobblin has continued to evolve with a variety of utility packages like > Gobblin metrics and Gobblin config management. Collectively, these allow > organizations utilizing Gobblin to use a system that has been battle tested > at LinkedIn scale. This is something that its consumers have to come to > appreciate greatly. > > == Rationale == > > Gobblin's entry to the Apache foundation is beneficial to both the Gobblin > and the Apache communities. Gobblin has greatly benefited from its open > source roots. Its community and adoption has grown greatly as a result. > More importantly, the feedback from the community whether through > interactions at meetups or through the mailing list have allowed for a rich > exchange of ideas. In order to grow up the Gobblin community and improve > the project, we would like to propose Gobblin to the Apache incubator. The > Gobblin community will greatly benefit from the established development and > consensus processes that have worked well for other projects. The Apache > process has served many other open source projects well and we believe that > the Gobblin community will greatly benefit from these practices as well. > > == Initial Goals == > > Migrate the existing codebase to Apache > Study and Integrate with the Apache development process > Ensure all dependencies are compliant with Apache License version 2.0 > Incremental development and releases per Apache guidelines > Improve the relationship between Gobblin and other Apache projects > > == Current Status == > > Gobblin has undergone five major releases (0.5, 0.6, 0.7, 0.8, 0.9) and > many minor ones. The latest version, Gobblin 0.9 has just been released in > December, 2016. Gobblin is being used in production by over 20 > organizations. Gobblin codebase is currently hosted at github.com, which > will seed the Apache git repository. > > === Meritocracy === > > We plan to invest in supporting a meritocracy. We will discuss the > requirements in an open forum. Several companies have already expressed > interest in this project, and we intend to invite additional developers to > participate. We will encourage and monitor community participation so that > privileges can be extended to those that contribute. > > === Community === > > The need for a extensible and flexible data integration platform in the > open source is tremendous. Gobblin is currently being used by at least 20 > organizations worldwide (some examples are listed here). By bringing > Gobblin into Apache, we believe that the community will grow even bigger. > > === Core Developers === > > Gobblin was started by engineers at LinkedIn, and now has developers from > Google, Facebook, LinkedIn, Cloudera, Nerdwallet, Swisscom, and many other > companies. > > === Alignment === > > Gobblin aligns exceedingly well with the Apache ecosystem. Gobblin is > built leveraging several existing Apache projects (Apache Helix, Yarn, > Zookeeper etc.). As Gobblin matures, we expect to leverage several other > Apache projects further. This leverage invariably results in contributions > back to these projects (e.g., a contribution to Helix was made during the > Gobblin Yarn development). Finally, being an integration platform, it > serves as a bridge between several Apache projects like Apache Hadoop and > Apache Kafka. This integration is highly desired and their interaction > through Gobblin will lead to a virtuous cycle of greater adoption and newer > features in these projects. Thus, we believe that it will be a nice > addition to the current set of big data projects under the auspices of the > Apache foundation. > > == Known Risks == > > === Orphaned Products === > > The risk of the Gobblin project being abandoned is minimal. As noted > earlier, there are many organizations that have already invested in Gobblin > significantly and are thus incentivized to continue development. Many of > these organizations operate critical data ingest, compliance and retention > pipelines built with Gobblin and are thus heavily invested in the continued > success of Gobblin. > > === Inexperience with Open Source === > > Gobblin has existed as a healthy open source project for several years. > During that time, we have curated an open-source community successfully. > Any risks that we foresee are ones associated with scaling our open source > communication and operation process rather than with inherent inexperience > in operating an open source project. > > === Homogenous Developers === > > Gobblin’s committers are employed by companies of varying sizes and > industry. Committers come from well heeled internet companies like Google, > LinkedIn and Facebook. We also have developers from traditional enterprise > companies like SwissCom. Well funded startups like Nerdwallet are active in > the community of developers. We plan to double our efforts in cultivating > a diverse set of committers for Gobblin. > > === Reliance on Salaried Developers === > > It is expected that Gobblin development will occur on both salaried time > and on volunteer time, after hours. The majority of initial committers are > paid by their employer to contribute to this project. However, they are all > passionate about the project, and we are confident that the project will > continue even if no salaried developers contribute to the project. We are > committed to recruiting additional committers including non-salaried > developers. > > === Relationships with Other Apache Products === > > As noted earlier, Gobblin leverages several open source projects and > contributes back to them. There is also overlap with aspects of other > Apache projects that we will discuss briefly here. Apache Nifi, like > Gobblin aspires to reduce the operational overhead arising from data > heterogeneity. Apache Nifi is structured as a visual flow based approach > and provides built-in constructs for buffering data, prioritizing data, and > understanding data lineage as data flows across systems. Apache Nifi has > its own dataflow based execution engine with buffering, scheduling and > streaming capabilities. Apache Falcon is a Hadoop centric data governance > engine for defining, scheduling, and monitoring data management policies > through flow definition typically for data that has been ingested into > Hadoop already. Apache Falcon generally delegates data management jobs to > tools that already exist in the Hadoop ecosystem (e.g. Distcp, Sqoop, Hive > etc). Apache Sqoop is primarily geared for bulk ingest especially from > databases which is one part of Gobblin’s feature set. Apache Flume focuses > primarily on streaming data movement. Finally, general purpose data > processing engines like Apache Flink, Apache Samza, and Apache Spark focus > on generic computation. > > Gobblin design choices intersect with specific features in all of these > systems, however in aggregate, it is a different point in the design space. > It is designed to handle both streaming and batch data. It supports > execution through a standalone cluster mode as well as through existing > frameworks such as MR, Yarn, Hive, Samza etc allowing users to choose the > deployment model that is optimal for the specific data integration > challenge. It provides native optimized implementations for critical > integrations such as Kafka, Hadoop - Hadoop copies etc. Gobblin also > supports both Hadoop and non-Hadoop data, being able to ingest data into > Kafka as well as other key-value stores like Couchbase. Gobblin is also not > just a generic computation framework, it has specific constructs for data > integration patterns such as data quality metrics and policies. Gobblin’s > configuration management system allows it to be fully multi-tenant and take > advantage of grouped policies when required. For batch workloads, Gobblin > has a planning phase that provides for better resource utilization. > > In summary, there is healthy diversity in the number of systems > approaching the interesting and pressing problem of big data integration. > We believe that Gobblin will provide another compelling choice in that > design space. > > === An Excessive Fascination with the Apache Brand === > > Gobblin is already a healthy and well known open source project. This > proposal is not for the purpose of generating publicity. Rather, the > primary benefits to joining Apache are already outlined in the Rationale > section. > > == Documentation == > > The reader will find these websites highly relevant: > * Website: http://linkedin.github.io/gobblin/ > * Documentation: https://gobblin.readthedocs.io/en/latest/ > * Codebase: https://github.com/linkedin/gobblin/ > * User group: https://groups.google.com/forum/#!forum/gobblin-users > > == Source and Intellectual Property Submission Plan == > > The Gobblin codebase is currently hosted on Github. This is the exact > codebase that we would migrate to the Apache foundation.The Gobblin source > code is already licensed under Apache License Version 2.0. Going forward, > we will continue to have all the contributions licensed directly to the > Apache foundation through our signed Individual Contributor License > Agreements for all the committers on the project. > > == External Dependencies == > > To the best of our knowledge, all of Gobblin dependencies are distributed > under Apache compatible licenses. Upon acceptance to the incubator, we > would begin a thorough analysis of all transitive dependencies to verify > this fact and introduce license checking into the build and release process > (for instance integrating Apache Rat). > > == Cryptography == > > We do not expect Gobblin to be a controlled export item due to the use of > encryption. > > == Required Resources == > > === Mailing lists === > > * gobblin-user > * gobblin-dev > * gobblin-commits > * gobblin-private for private PMC discussions (with moderated > subscriptions) > > === Subversion Directory === > > Git is the preferred source control system: git://git.apache.org/gobblin > > === Issue Tracking === > > JIRA Gobblin (GOBBLIN) > > === Other Resources === > > The existing code already has unit and integration tests, so we would > like a Jenkins instance to run them whenever a new patch is submitted. This > can be added after project creation. > > == Initial Committers == > > * Abhishek Tiwari <abhishektiwari dot btech at gmail dot com> > * Shirshanka Das <shirshanka at apache dot org> > * Chavdar Botev <cbotev at gmail dot com> > * Sahil Takiar <takiar.sahil at gmail dot com> > * Yinan Li <liyinan926 at gmail dot com> > * Ziyang Liu <> > * Lorand Bendig <lbendig at gmail dot com> > * Issac Buenrostro <ibuenros at linkedin dot com> > * Hung Tran <hutran at linkedin dot com> > * Olivier Lamy <olamy at apache dot org> > * Jean-Baptiste Onofré <jbono...@apache.org> > > == Affiliations == > > * Abhishek Tiwari - LinkedIn > * Shirshanka Das - LinkedIn > * Chavdar Botev - Stealth Startup > * Sahil Takiar - Cloudera > * Yinan Li - Google > * Ziyang Liu - Facebook > * Lorand Bendig - Swisscom > * Issac Buenrostro - LinkedIn > * Hung Tran - LinkedIn > * Olivier Lamy - Webtide > * Jean-Baptiste Onofre - Talend > > == Sponsors == > > === Champion === > > Olivier Lamy < olamy at apache dot org> > > === Nominated Mentors === > > * Olivier Lamy <olamy at apache dot org> > * Jean-Baptiste Onofre <jbonofre at apache dot org> > * ? > * ? > > == Sponsoring Entity == > The Apache Incubator > -- Olivier Lamy http://twitter.com/olamy | http://linkedin.com/in/olamy
