>From what I understand, that Accumulo may or may not suffer from "bifurcation" is debatable despite this blanket claim otherwise.
Regardless, Sean's point is valid, even if only a theoretical concern for NiFi at this point. If the podling is interested in growing outside community I'd be happy to sign on as a mentor to help that happen. On Thu, Nov 20, 2014 at 3:32 AM, Benson Margulies <bimargul...@gmail.com> wrote: > Sean, > > The precedent of Accumulo is that the govt people and agencies involved are > ready and able to have their staff collaborate openly in an Apache > community. There's no need to contemplate bifurcation; we have this > proposal because the management recognizes that this collaboration produces > better stuff that solves more problems than the 'inside the tent' > alternative. > > --benson > > > On Thu, Nov 20, 2014 at 1:50 AM, Sean Busbey <bus...@cloudera.com> wrote: > > > I'm really excited to see NiFi come to the incubator; it'd be a great > > addition to the ASF. > > > > A few points in the proposal: > > > > > >== Initial Goals == > > > > One of these should be to grow the community outside of the current > niche, > > IMHO. > > > > More on this below under "orphaned projects" > > > > > > * Determine and establish a mechanism, possibly including a > > > >sub-project construct, that allows for extensions to the core > > > >application to occur at a pace that differs from the core application > > > >itself. > > > > I don't think the proposal needs to include the "e.g. with sub-projects" > > part. Just noting > > that your goals in the incubator are to address the need to have > different > > release cycles > > for core and extensions is sufficient. > > > > > > > >=== Community === > > > >Over the past several years, NiFi has developed a strong community of > > > >both developers and operators within the U.S. government. We look > > > >forward to helping grow this to a broader base of industries. > > > > > > > > How much, if any, of this community do you expect to engage via the > > customary project > > lists once NiFi is established within the ASF? Will the project be able > to > > leverage this > > established group? > > > > > > > >=== Orphaned Products === > > > >Risk of orphaning is minimal. The project user and developer base is > > > >substantial, growing, and there is already extensive operational use > > > >of NiFi. > > > > Given that the established base is internal to the U.S. government, I'd > > encourage the > > podling to consider the risk of a bifurcated project should a substantial > > outside > > community fail to emerge or if those internal users should fail to engage > > with the > > outside community. > > > > You cover a related issue in your "Homogenous Developers" section. But I > > think > > building on the "Community" section of the current state to call this out > > as an > > independent issue is worthwhile. > > > > > > > >possible. This environment includes widely accessible source code > > > >repositories, published artifacts, ticket tracking, and extensive > > > >documentation. We also encourage contributions and frequent debate and > > > >hold regular, collaborative discussions through e-mail, chat rooms, > > > >and in-person meet-ups. > > > > Do you anticipate any difficulties moving these established communication > > mechanisms to ASF public lists? > > > > > >=== Documentation === > > > >At this time there is no NiFi documentation on the web. However, we > > > >have extensive documentation included within the application that > > > >details usage of the many functions. We will be rapidly expanding the > > > >available documentation to cover things like installation, developer > > > >guide, frequently asked questions, best practices, and more. This > > > >documentation will be posted to the NiFi wiki at apache.org. > > > > I love projects that start with documentation. :) > > > > I don't think the proposal needs to include that the documentation will > be > > posted > > to the NiFi wiki, since that's an implementation detail. Just say "this > > documentation > > will be made available via the NiFi project's use of incubator infra." > > > > (I'll save detail for the eventual dev@ list, but you should strongly > > consider not > > using the wiki to host this documentation.) > > > > -Sean > > > > On Wed, Nov 19, 2014 at 11:27 PM, Brock Noland <br...@cloudera.com> > wrote: > > > > > Hi Joe, > > > > > > I know you've done a tremendous amount of work to make this happen so I > > am > > > extremely happy this is *finally* making it's way to the incubator! > > > > > > I look forward to helping in anyway I can. > > > > > > Cheers! > > > Brock > > > > > > On Wed, Nov 19, 2014 at 8:11 PM, Mattmann, Chris A (3980) < > > > chris.a.mattm...@jpl.nasa.gov> wrote: > > > > > > > This is *fan freakin¹ tastic* Sounds like an awesome project and > > > > glad to hear a relationship to Tika! Awesome to see more government > > > > projects coming into the ASF! > > > > > > > > you already have a great set of mentors and I don¹t really have more > > > > time on my plate, but really happy and will try and monitor and help > > > > on the lists. > > > > > > > > Cheers! > > > > > > > > Chris > > > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Chris Mattmann, Ph.D. > > > > Chief Architect > > > > Instrument Software and Science Data Systems Section (398) > > > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > > > Office: 168-519, Mailstop: 168-527 > > > > Email: chris.a.mattm...@nasa.gov > > > > WWW: http://sunset.usc.edu/~mattmann/ > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > Adjunct Associate Professor, Computer Science Department > > > > University of Southern California, Los Angeles, CA 90089 USA > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Joe Witt <joe.w...@gmail.com> > > > > Reply-To: "general@incubator.apache.org" < > general@incubator.apache.org > > > > > > > Date: Thursday, November 20, 2014 at 3:02 AM > > > > To: "general@incubator.apache.org" <general@incubator.apache.org> > > > > Subject: [PROPOSAL] NiFi for Incubation > > > > > > > > >Hello, > > > > > > > > > > > > > > >I would like to propose NiFi as an Apache Incubator Project. > > > > > > > > > >In addition to the copy provided below the Wiki version of the > > > > >proposal can be found here: > > > > >http://wiki.apache.org/incubator/NiFiProposal > > > > > > > > > >Thanks > > > > > > > > > >Joe > > > > > > > > > > > > > > >= NiFi Proposal = > > > > > > > > > >== Abstract == > > > > >NiFi is a dataflow system based on the concepts of flow-based > > > programming. > > > > > > > > > >== Proposal == > > > > >NiFi supports powerful and scalable directed graphs of data routing, > > > > >transformation, and system mediation logic. Some of the high-level > > > > >capabilities and objectives of NiFi include: > > > > > * Web-based user interface for seamless experience between design, > > > > >control, feedback, and monitoring of data flows > > > > > * Highly configurable along several dimensions of quality of > service > > > > >such as loss tolerant versus guaranteed delivery, low latency versus > > > > >high throughput, and priority based queuing > > > > > * Fine-grained data provenance for all data received, forked, > > > > >joined, cloned, modified, sent, and ultimately dropped as data > reaches > > > > >its configured end-state > > > > > * Component-based extension model along well defined interfaces > > > > >enabling rapid development and effective testing > > > > > > > > > >== Background == > > > > >Reliable and effective dataflow between systems can be difficult > > > > >whether you're running scripts on a laptop or have a massive > > > > >distributed computing system operated by numerous teams and > > > > >organizations. As the volume and rate of data grows and as the > number > > > > >of systems, protocols, and formats increase and evolve so too does > the > > > > >complexity and need for greater insight and agility. These are the > > > > >dataflow challenges that NiFi was built to tackle. > > > > > > > > > >NiFi is designed in a manner consistent with the core concepts > > > > >described in flow-based programming as originally documented by J. > > > > >Paul Morrison in the 1970s. This model lends itself well to visual > > > > >diagramming, concurrency, componentization, testing, and reuse. In > > > > >addition to staying close to the fundamentals of flow-based > > > > >programming, NiFi provides integration system specific features such > > > > >as: guaranteed delivery; back pressure; ability to gracefully handle > > > > >backlogs and data surges; and an operator interface that enables > > > > >on-the-fly data flow generation, modification, and observation. > > > > > > > > > >== Rationale == > > > > >NiFi provides a reliable, scalable, manageable and accountable > > > > >platform for developers and technical staff to create and evolve > > > > >powerful data flows. Such a system is useful in many contexts > > > > >including large-scale enterprise integration, interaction with cloud > > > > >services and frameworks, business to business, intra-departmental, > and > > > > >inter-departmental flows. NiFi fits well within the Apache Software > > > > >Foundation (ASF) family as it depends on numerous ASF projects and > > > > >integrates with several others. We also anticipate developing > > > > >extensions for several other ASF projects such as Cassandra, Kafka, > > > > >and Storm in the near future. > > > > > > > > > >== Initial Goals == > > > > > * Ensure all dependencies are compliant with Apache License > version > > > > >2.0 and all that all code and documentation artifacts have the > correct > > > > >Apache licensing markings and notice. > > > > > * Establish a formal release process and schedule, allowing for > > > > >dependable release cycles in a manner consistent with the Apache > > > > >development process. > > > > > * Determine and establish a mechanism, possibly including a > > > > >sub-project construct, that allows for extensions to the core > > > > >application to occur at a pace that differs from the core > application > > > > >itself. > > > > > > > > > >== Current Status == > > > > >=== Meritocracy === > > > > >An integration platform is only as good as its ability to integrate > > > > >systems in a reliable, timely, and repeatable manner. The same can > be > > > > >said of its ability to attract talent and a variety of perspectives > as > > > > >integration systems by their nature are always evolving. We will > > > > >actively seek help and encourage promotion of influence in the > project > > > > >through meritocracy. > > > > > > > > > >=== Community === > > > > >Over the past several years, NiFi has developed a strong community > of > > > > >both developers and operators within the U.S. government. We look > > > > >forward to helping grow this to a broader base of industries. > > > > > > > > > >=== Core Developers === > > > > >The initial core developers are employed by the National Security > > > > >Agency and defense contractors. We will work to grow the community > > > > >among a more diverse set of developers and industries. > > > > > > > > > >=== Alignment === > > > > >From its inception, NiFi was developed with an open source > philosophy > > > > >in mind and with the hopes of eventually being truly open sourced. > > > > >The Apache way is consistent with the approach we have taken to > date. > > > > >The ASF clearly provides a mature and effective environment for > > > > >successful development as is evident across the spectrum of > well-known > > > > >projects. Further, NiFi depends on numerous ASF libraries and > > > > >projects including; ActiveMQ, Ant, Commons, Lucene, Hadoop, > > > > >HttpClient, Jakarta and Maven. We also anticipate extensions and > > > > >dependencies with several more ASF projects, including Accumulo, > Avro, > > > > >Casandra, HBase, JClouds, Storm, Kafka, Thrift, Tika, and others. > > > > >This existing alignment with Apache and the desired community makes > > > > >the Apache Incubator a good fit for NiFi. > > > > > > > > > >== Known Risks == > > > > >=== Orphaned Products === > > > > >Risk of orphaning is minimal. The project user and developer base > is > > > > >substantial, growing, and there is already extensive operational use > > > > >of NiFi. > > > > > > > > > >=== Inexperience with Open Source === > > > > >The initial committers to NiFi have limited experience with true > open > > > > >source software development. However, despite the project origins > > > > >being from closed source development we have modelled our behavior > and > > > > >community development on The Apache Way to the greatest extent > > > > >possible. This environment includes widely accessible source code > > > > >repositories, published artifacts, ticket tracking, and extensive > > > > >documentation. We also encourage contributions and frequent debate > and > > > > >hold regular, collaborative discussions through e-mail, chat rooms, > > > > >and in-person meet-ups. We are committed to the ideals of open > source > > > > >software and will eagerly seek out mentors and sponsors who can help > > > > >us quickly come up to speed. > > > > > > > > > >=== Homogenous Developers === > > > > >The initial committers of NiFi come from a limited set of entities > > > > >though we are committed to recruiting and developing additional > > > > >committers from a broad spectrum of industries and backgrounds. > > > > > > > > > >=== Reliance on Salaried Developers === > > > > >We expect NiFi development to continue on salaried time and through > > > > >volunteer time. The initial committers are paid by their employers > to > > > > >contribute to this project. We are committed to developing and > > > > >recruiting participation from developers both salaried and > > > > >non-salaried. > > > > > > > > > >=== Relationship with other Apache Projects === > > > > >As described in the alignment section, NiFi is already heavily > > > > >dependent on other ASF projects and we anticipate further dependence > > > > >and integration with new and emerging projects in the Apache family. > > > > > > > > > >=== An Excessive Fascination with the Apache Brand === > > > > >We respect the laudable Apache brand and that is certainly a factor > in > > > > >the decision to propose NiFi for the Apache Incubator. We intend to > > > > >provide a great deal of energy and capability to the ASF through > this > > > > >project. We will be sensitive to and respectful of any overuse of > the > > > > >Apache brand and ensure our focus remains on how we benefit the > Apache > > > > >community. > > > > > > > > > >=== Documentation === > > > > >At this time there is no NiFi documentation on the web. However, we > > > > >have extensive documentation included within the application that > > > > >details usage of the many functions. We will be rapidly expanding > the > > > > >available documentation to cover things like installation, developer > > > > >guide, frequently asked questions, best practices, and more. This > > > > >documentation will be posted to the NiFi wiki at apache.org. > > > > > > > > > >== Initial Source == > > > > >NiFi has been in active development since late 2006 with > contributions > > > > >from dozens of developers and feedback from hundreds of users and > > > > >developers. The core codebase is written in Java and includes > > > > >detailed Javadocs and feature documentation. > > > > > > > > > >== Source and Intellectual Property Submission == > > > > >Previously referred to as Niagarafiles, the NiFi code and > > > > >documentation materials will be submitted by the National Security > > > > >Agency. NiFi has been developed by a mix of government employees > and > > > > >private companies under government contract. Material developed by > > > > >the government employees is in the public domain and no U.S. > copyright > > > > >exists in works of the federal government. For the contractor > > > > >developed material in the initial submission, the U.S. Government > has > > > > >sufficient authority to open source per DFARS 252.227-7014. NSA > has > > > > >submitted the Software Grant Agreement and Corporate Contributor > > > > >License Agreement to the Apache Software Foundation. > > > > > > > > > >== External Dependencies == > > > > >We have at least one dependency on an LGPL library which we will > > > > >promptly address. Otherwise, we believe all current dependencies > are > > > > >compatible with the ASF guidelines. Our dependency licenses come > from > > > > >the following license styles: Apache v 2.0, BSD, Public Domain, > > > > >Eclipse Public v1, MIT, CDDL v1. > > > > > > > > > >== Cryptography == > > > > >Consistent with http://www.apache.org/licenses/exports/ we believe > > > > >NiFi is classified as ECCN 5D002. NiFi doesn't implement any > > > > >cryptographic algorithms but is designed to use algorithms provided > by > > > > >Oracle Java Cryptographic Extensions, BouncyCastle, and JCraft, Inc. > > > > >These cryptographic algorithm providers are used to support SSL, > > > > >SSH/SFTP, and the encryption and decryption of sensitive properties. > > > > >In the event that it becomes necessary we will engage with > appropriate > > > > >Apache members to ensure we file any necessary paperwork or > clarified > > > > >any cryptographic export license concerns. > > > > > > > > > >== Required Resources == > > > > >=== Mailing Lists === > > > > > * u...@nifi.incubator.apache.org > > > > > * d...@nifi.incubator.apache.org > > > > > * priv...@nifi.incubator.apache.org > > > > > * comm...@nifi.incubator.apache.org > > > > > > > > > >=== Source Control === > > > > >NiFi requests use of Git for source control > > > > >(git://git.apache.org/nifi.git). We request a writeable Git repo > for > > > > >NiFi with mirroring to be setup to Github through INFRA. We request > > > > >sponsor Benson Margulies (bimargulies) to assist with creating the > > > > >INFRA ticket for this. > > > > > > > > > >=== Issue Tracking === > > > > >JIRA NiFi (NIFI) > > > > > > > > > >=== Initial Committers === > > > > > * Brandon Devries <brandon.devries at gmail dot com> > > > > > * Matt Gilman <matt.c.gilman at gmail dot com> > > > > > * Tony Kurc <trkurc at gmail dot com> > > > > > * Mark Payne <markap14 at hotmail dot com> > > > > > * Adam Taft <adam at adamtaft dot com> > > > > > * Joseph Witt <joewitt at gmail dot com> > > > > > > > > > >=== Affiliations === > > > > > * Brandon Devries (Requitest, Inc.) > > > > > * Matt Gilman (Raytheon) > > > > > * Tony Kurc (National Security Agency) > > > > > * Mark Payne (Sotera Defense Solutions, Inc.) > > > > > * Adam Taft (Requitest, Inc.) > > > > > * Joseph Witt (National Security Agency) > > > > > > > > > >== Sponsors == > > > > >=== Champion === > > > > > * Benson Margulies (Basis Technology) <bimargulies at apache dot > > org> > > > > > > > > > >=== Nominated Mentors === > > > > > * Drew Farris (Booz Allen Hamilton) <drew at apache dot org> > > > > > * Brock Noland (Cloudera) <brock at apache dot org> > > > > > * Billie Rinaldi (Hortonworks) <billie at apache dot org> > > > > > > > > > >=== Sponsoring Entity === > > > > >We request the Apache Incubator to sponsor this project. > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > -- > > Sean > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
