If you need/want another mentor, I volunteer > On Feb 14, 2017, at 3:53 PM, Olivier Lamy <ol...@apache.org> wrote: > > Hi > Well I don't see issues as no one discuss the proposal. > So I will start the official vote tomorrow. > Cheers > Olivier > > On 6 February 2017 at 14:08, Olivier Lamy <ol...@apache.org> wrote: > >> Hello everyone, >> I would like to submit to you a proposal to bring Gooblin to the Apache >> Software Foundation. >> The text of the proposal is included below and available as a draft here >> in the Wiki: https://wiki.apache.org/incubator/GobblinProposal >> >> We will appreciate any feedback and input. >> >> Olivier on behalf of the Gobblin community >> >> >> = Apache Gobblin Proposal = >> == Abstract == >> Gobblin is a distributed data integration framework that simplifies common >> aspects of big data integration such as data ingestion, replication, >> organization and lifecycle management for both streaming and batch data >> ecosystems. >> >> == Proposal == >> >> Gobblin is a universal data integration framework. The framework has been >> used to build a variety of big data applications such as ingestion, >> replication, and data retention. The fundamental constructs provided by the >> Gobblin framework are: >> >> 1. An expandable set of connectors that allow data to be integrated from >> a variety of sources and sinks. The range of connectors already available >> in Gobblin are quite diverse and are an ever expanding set. To highlight >> just a few examples, connectors exist for databases (e.g., MySQL, Oracle >> Teradata, Couchbase etc.), web based technologies (REST APIs, FTP/SFTP >> servers, Filers), scalable storage (HDFS, S3, Ambry etc,), streaming data >> (Kafka, EventHubs etc.), and a variety of proprietary data sources and >> sinks (e.g.Salesforce, Google Analytics, Google Webmaster etc.). Similarly, >> Gobblin has a rich library of converters that allow for conversion of data >> from one format to another as data moves across system boundaries (e.g. >> AVRO in HDFS to JSON in another system). >> >> >> 2. Gobblin has a well defined and customizable state management layer >> that allows writing stateful applications. These are particularly useful >> when solving problems like bulk incremental ingest and keeping several >> clusters replicated in sync. The ability to record work that has been >> completed and what remains in a scalable manner is critical to writing such >> diverse applications successfully. >> >> >> 3. Gobblin is agnostic to the underlying execution engine. It can be >> tailored to run ontop of a variety of execution frameworks ranging from >> multiple processes on a single node, to open source execution engines like >> MapReduce, Spark or Samza, natively on top of raw containers like Yarn or >> Mesos, and the public cloud like Amazon AWS or Microsoft Azure. We are >> extending Gobblin to run on top of a self managed cluster when security is >> vital. This allows different applications that require different degrees >> of scalability, latency or security to be customized to for their specific >> needs. For example, highly latency sensitive applications can be executed >> in a streaming environment while batch based execution might benefit >> applications where the priority might be geared towards optimal container >> utilization. >> >> 4.Gobblin comes out of the box with several diagnosability features like >> Gobblin metrics and error handling. Collectively, these features allow >> Gobblin to operate at the scale of petabytes of data. To give just one >> example, the ability to quarantine a few bad records from an isolated Kafka >> topic without stopping the entire flow from continued execution is vital >> when the number of Kafka topics range in the thousands and the collective >> data handled is in the petabytes. >> >> Gobblin thus provides crisply defined software constructs that can be used >> to build a vast array of data integration applications customizable for >> varied user needs. It has become a preferred technology for data >> integration use-cases by many organizations worldwide (see a partial list >> here). >> >> == Background == >> >> Over the last decade, data integration has evolved use case by use case in >> most companies. For example, at LinkedIn, when Kafka became a significant >> part of the data ecosystem, a system called Camus was built to ingest this >> data for analytics processing on Hadoop. Similarly, we had custom pipelines >> to ingest data from Salesforce, Oracle and myriad other sources. This >> pattern became the norm rather than the exception and one point, LinkedIn >> was running at least fifteen different types of ingestion pipelines. This >> fragmentation has several unfortunate implications. Operational costs scale >> with the number of pipelines even if the myriad pipelines share a vasty >> array of common features. Bug fixes and performance optimizations cannot be >> shared across the pipelines. A common set of practices around debugging and >> deployment does not emerge. Each pipeline operator will continue to invest >> in his little silo of the data integration world completely oblivious to >> the challenges of his fellow operator sitting five tables down. >> >> These experiences were the genesis behind the design and implementation of >> Gobblin. Gobblin thus started out as a universal data ingestion framework >> focussed on extracting, transforming, and synchronizing large volumes of >> data between different data sources and sinks. Not surprisingly, given its >> origins, the initial design of Gobblin placed great emphasis on >> abstractions that can be leveraged repeatedly. These abstractions have >> stood the test of time at LinkedIn and we have been able to leverage the >> constructs well beyond ingest. Gobblin's architecture has allowed us at >> LinkedIn to use it for a variety of applications ranging from from optimal >> format conversion to adhering to compliance policies set by European >> standards. Finally, as noted earlier, Gobblin can be deployed in a variety >> of execution environments: it can be deployed as a library embedded in >> another application or can be used to execute jobs on a public cloud. A >> fluid architectural and execution design story has allowed Gobblin to >> become a truly successful data integration platform. >> >> Gobblin has continued to evolve with a variety of utility packages like >> Gobblin metrics and Gobblin config management. Collectively, these allow >> organizations utilizing Gobblin to use a system that has been battle tested >> at LinkedIn scale. This is something that its consumers have to come to >> appreciate greatly. >> >> == Rationale == >> >> Gobblin's entry to the Apache foundation is beneficial to both the Gobblin >> and the Apache communities. Gobblin has greatly benefited from its open >> source roots. Its community and adoption has grown greatly as a result. >> More importantly, the feedback from the community whether through >> interactions at meetups or through the mailing list have allowed for a rich >> exchange of ideas. In order to grow up the Gobblin community and improve >> the project, we would like to propose Gobblin to the Apache incubator. The >> Gobblin community will greatly benefit from the established development and >> consensus processes that have worked well for other projects. The Apache >> process has served many other open source projects well and we believe that >> the Gobblin community will greatly benefit from these practices as well. >> >> == Initial Goals == >> >> Migrate the existing codebase to Apache >> Study and Integrate with the Apache development process >> Ensure all dependencies are compliant with Apache License version 2.0 >> Incremental development and releases per Apache guidelines >> Improve the relationship between Gobblin and other Apache projects >> >> == Current Status == >> >> Gobblin has undergone five major releases (0.5, 0.6, 0.7, 0.8, 0.9) and >> many minor ones. The latest version, Gobblin 0.9 has just been released in >> December, 2016. Gobblin is being used in production by over 20 >> organizations. Gobblin codebase is currently hosted at github.com, which >> will seed the Apache git repository. >> >> === Meritocracy === >> >> We plan to invest in supporting a meritocracy. We will discuss the >> requirements in an open forum. Several companies have already expressed >> interest in this project, and we intend to invite additional developers to >> participate. We will encourage and monitor community participation so that >> privileges can be extended to those that contribute. >> >> === Community === >> >> The need for a extensible and flexible data integration platform in the >> open source is tremendous. Gobblin is currently being used by at least 20 >> organizations worldwide (some examples are listed here). By bringing >> Gobblin into Apache, we believe that the community will grow even bigger. >> >> === Core Developers === >> >> Gobblin was started by engineers at LinkedIn, and now has developers from >> Google, Facebook, LinkedIn, Cloudera, Nerdwallet, Swisscom, and many other >> companies. >> >> === Alignment === >> >> Gobblin aligns exceedingly well with the Apache ecosystem. Gobblin is >> built leveraging several existing Apache projects (Apache Helix, Yarn, >> Zookeeper etc.). As Gobblin matures, we expect to leverage several other >> Apache projects further. This leverage invariably results in contributions >> back to these projects (e.g., a contribution to Helix was made during the >> Gobblin Yarn development). Finally, being an integration platform, it >> serves as a bridge between several Apache projects like Apache Hadoop and >> Apache Kafka. This integration is highly desired and their interaction >> through Gobblin will lead to a virtuous cycle of greater adoption and newer >> features in these projects. Thus, we believe that it will be a nice >> addition to the current set of big data projects under the auspices of the >> Apache foundation. >> >> == Known Risks == >> >> === Orphaned Products === >> >> The risk of the Gobblin project being abandoned is minimal. As noted >> earlier, there are many organizations that have already invested in Gobblin >> significantly and are thus incentivized to continue development. Many of >> these organizations operate critical data ingest, compliance and retention >> pipelines built with Gobblin and are thus heavily invested in the continued >> success of Gobblin. >> >> === Inexperience with Open Source === >> >> Gobblin has existed as a healthy open source project for several years. >> During that time, we have curated an open-source community successfully. >> Any risks that we foresee are ones associated with scaling our open source >> communication and operation process rather than with inherent inexperience >> in operating an open source project. >> >> === Homogenous Developers === >> >> Gobblin’s committers are employed by companies of varying sizes and >> industry. Committers come from well heeled internet companies like Google, >> LinkedIn and Facebook. We also have developers from traditional enterprise >> companies like SwissCom. Well funded startups like Nerdwallet are active in >> the community of developers. We plan to double our efforts in cultivating >> a diverse set of committers for Gobblin. >> >> === Reliance on Salaried Developers === >> >> It is expected that Gobblin development will occur on both salaried time >> and on volunteer time, after hours. The majority of initial committers are >> paid by their employer to contribute to this project. However, they are all >> passionate about the project, and we are confident that the project will >> continue even if no salaried developers contribute to the project. We are >> committed to recruiting additional committers including non-salaried >> developers. >> >> === Relationships with Other Apache Products === >> >> As noted earlier, Gobblin leverages several open source projects and >> contributes back to them. There is also overlap with aspects of other >> Apache projects that we will discuss briefly here. Apache Nifi, like >> Gobblin aspires to reduce the operational overhead arising from data >> heterogeneity. Apache Nifi is structured as a visual flow based approach >> and provides built-in constructs for buffering data, prioritizing data, and >> understanding data lineage as data flows across systems. Apache Nifi has >> its own dataflow based execution engine with buffering, scheduling and >> streaming capabilities. Apache Falcon is a Hadoop centric data governance >> engine for defining, scheduling, and monitoring data management policies >> through flow definition typically for data that has been ingested into >> Hadoop already. Apache Falcon generally delegates data management jobs to >> tools that already exist in the Hadoop ecosystem (e.g. Distcp, Sqoop, Hive >> etc). Apache Sqoop is primarily geared for bulk ingest especially from >> databases which is one part of Gobblin’s feature set. Apache Flume focuses >> primarily on streaming data movement. Finally, general purpose data >> processing engines like Apache Flink, Apache Samza, and Apache Spark focus >> on generic computation. >> >> Gobblin design choices intersect with specific features in all of these >> systems, however in aggregate, it is a different point in the design space. >> It is designed to handle both streaming and batch data. It supports >> execution through a standalone cluster mode as well as through existing >> frameworks such as MR, Yarn, Hive, Samza etc allowing users to choose the >> deployment model that is optimal for the specific data integration >> challenge. It provides native optimized implementations for critical >> integrations such as Kafka, Hadoop - Hadoop copies etc. Gobblin also >> supports both Hadoop and non-Hadoop data, being able to ingest data into >> Kafka as well as other key-value stores like Couchbase. Gobblin is also not >> just a generic computation framework, it has specific constructs for data >> integration patterns such as data quality metrics and policies. Gobblin’s >> configuration management system allows it to be fully multi-tenant and take >> advantage of grouped policies when required. For batch workloads, Gobblin >> has a planning phase that provides for better resource utilization. >> >> In summary, there is healthy diversity in the number of systems >> approaching the interesting and pressing problem of big data integration. >> We believe that Gobblin will provide another compelling choice in that >> design space. >> >> === An Excessive Fascination with the Apache Brand === >> >> Gobblin is already a healthy and well known open source project. This >> proposal is not for the purpose of generating publicity. Rather, the >> primary benefits to joining Apache are already outlined in the Rationale >> section. >> >> == Documentation == >> >> The reader will find these websites highly relevant: >> * Website: http://linkedin.github.io/gobblin/ >> * Documentation: https://gobblin.readthedocs.io/en/latest/ >> * Codebase: https://github.com/linkedin/gobblin/ >> * User group: https://groups.google.com/forum/#!forum/gobblin-users >> >> == Source and Intellectual Property Submission Plan == >> >> The Gobblin codebase is currently hosted on Github. This is the exact >> codebase that we would migrate to the Apache foundation.The Gobblin source >> code is already licensed under Apache License Version 2.0. Going forward, >> we will continue to have all the contributions licensed directly to the >> Apache foundation through our signed Individual Contributor License >> Agreements for all the committers on the project. >> >> == External Dependencies == >> >> To the best of our knowledge, all of Gobblin dependencies are distributed >> under Apache compatible licenses. Upon acceptance to the incubator, we >> would begin a thorough analysis of all transitive dependencies to verify >> this fact and introduce license checking into the build and release process >> (for instance integrating Apache Rat). >> >> == Cryptography == >> >> We do not expect Gobblin to be a controlled export item due to the use of >> encryption. >> >> == Required Resources == >> >> === Mailing lists === >> >> * gobblin-user >> * gobblin-dev >> * gobblin-commits >> * gobblin-private for private PMC discussions (with moderated >> subscriptions) >> >> === Subversion Directory === >> >> Git is the preferred source control system: git://git.apache.org/gobblin >> >> === Issue Tracking === >> >> JIRA Gobblin (GOBBLIN) >> >> === Other Resources === >> >> The existing code already has unit and integration tests, so we would >> like a Jenkins instance to run them whenever a new patch is submitted. This >> can be added after project creation. >> >> == Initial Committers == >> >> * Abhishek Tiwari <abhishektiwari dot btech at gmail dot com> >> * Shirshanka Das <shirshanka at apache dot org> >> * Chavdar Botev <cbotev at gmail dot com> >> * Sahil Takiar <takiar.sahil at gmail dot com> >> * Yinan Li <liyinan926 at gmail dot com> >> * Ziyang Liu <> >> * Lorand Bendig <lbendig at gmail dot com> >> * Issac Buenrostro <ibuenros at linkedin dot com> >> * Hung Tran <hutran at linkedin dot com> >> * Olivier Lamy <olamy at apache dot org> >> * Jean-Baptiste Onofré <jbono...@apache.org> >> >> == Affiliations == >> >> * Abhishek Tiwari - LinkedIn >> * Shirshanka Das - LinkedIn >> * Chavdar Botev - Stealth Startup >> * Sahil Takiar - Cloudera >> * Yinan Li - Google >> * Ziyang Liu - Facebook >> * Lorand Bendig - Swisscom >> * Issac Buenrostro - LinkedIn >> * Hung Tran - LinkedIn >> * Olivier Lamy - Webtide >> * Jean-Baptiste Onofre - Talend >> >> == Sponsors == >> >> === Champion === >> >> Olivier Lamy < olamy at apache dot org> >> >> === Nominated Mentors === >> >> * Olivier Lamy <olamy at apache dot org> >> * Jean-Baptiste Onofre <jbonofre at apache dot org> >> * ? >> * ? >> >> == Sponsoring Entity == >> The Apache Incubator >> > > > > -- > Olivier Lamy > http://twitter.com/olamy | http://linkedin.com/in/olamy
--------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
