Hey Joe, "Yeah for the three/commit James replied on in that link - i dont see how we can keep those. They import from something that I doubt very much is compatible with ALv2/ASF guidelines."
Sorry for confusing things but my statement was referring to the MiNiFi C++ Python processors that don't import as java would. I've asked the original author to chime in here so that he can comment on the MiNiFi C++ processors, especially on whether or not he feels the processors should stay and the other concerns raised. I'll continue to monitor this and create those follow up tickets when I am able to. Thanks, Marc On Mon, May 4, 2020 at 12:13 PM Joe Witt <[email protected]> wrote: > Yeah for the three/commit James replied on in that link - i dont see how we > can keep those. They import from something that I doubt very much is > compatible with ALv2/ASF guidelines. > > On Mon, May 4, 2020 at 12:01 PM Marc Parisi <[email protected]> wrote: > > > Afternoon Joe, > > > > "So in the case of minificpp we are certain there is no L&N concern > > regarding either our source OR convenience binary?" > > > > I believe that to be the case; however, per > > https://github.com/apache/nifi/pull/4242#issuecomment-622655276 I think > > that maybe this is moot and the author can convert those processors in > > MiNiFI to use that open source python lib. I'm not sure if he's on this > > chain so I will follow up on the MiNiFi tickets to ensure that is the > case > > when I am able to log in again ( have to fix some account issue ). > > > > "Lastly, why are the JIRAs still open if this is merged? Can you please > > close them and assign to the proper fix version if these are 'done'?" > > > > Sorry, when I am able to log in ( account issue ) and make the above > > comment I will close the tx. I would prefer the fix be a follow on so > this > > is all tracked. The ticket fell off the radar ( as did the account issue, > > to be frank ), so when that is resolved I will take these actions. You > know > > me Joe, I'm like a dog who sees a squirrel, but I've put a note on my > > laptop to serve as a reminder. > > > > Thanks, > > Marc > > > > On Mon, May 4, 2020 at 11:49 AM Joe Witt <[email protected]> wrote: > > > > > Marc > > > > > > I'm not opposed to it staying if there is zero license issue. I only > > > mentioned the maintenance as a 'additional thing to consider' - that is > > > more of a discussion item. L&N isn't discussion - it is hard rule. > > > > > > Ok so in the minificpp case you're saying "there is no source or binary > > > dependency (that we'd package)" that isn't entirely ALv2/ASF compatible > > and > > > so if people want to use this they can do so by adding in the 'h2o' > bits > > at > > > their own effort, right? In this line of thinking I still don't love > it > > > but it is arguably not a lot different that our components which talk > to > > > services you have to have a paid or otherwise agreement with like Cloud > > > services offering by AWS, Azure, GCP, etc.. > > > > > > So in the case of minificpp we are certain there is no L&N concern > > > regarding either our source OR convenience binary? > > > > > > Lastly, why are the JIRAs still open if this is merged? Can you please > > > close them and assign to the proper fix version if these are 'done'? > > > > > > Thanks > > > > > > On Mon, May 4, 2020 at 11:46 AM Marc Parisi <[email protected]> > wrote: > > > > > > > "We should find a way for vendors to offer extension points like this > > > > without having to take on the burden of maintenance. How can we > > possibly > > > do > > > > this well? We're learning this lesson the hard way in NiFi itself > and > > > this > > > > is why the registry is being formulated." > > > > > > > > I think there is a fine line when walking vendor tie-in. Having the > > > > registry allows for an arm's length agreement with the Apache > > community. > > > I > > > > know others will disagree but my opinion is that extension points > that > > > > don't introduce licensing concerns from libraries should be fair game > > but > > > > the fact that a paid product requires a license for that code to be > > > useful > > > > should not be a non-starter -- if that makes sense. I believe I > stated, > > > "I > > > > don't have a strong argument" as I don't really have a strong > opinion ( > > > and > > > > certainly can be wrong too ) -- so I'm interested in hearing others > in > > > > regards to this. > > > > > > > > > > > > > > > > On Mon, May 4, 2020 at 11:38 AM Marc Parisi <[email protected]> > > wrote: > > > > > > > > > Hi Joe, > > > > > > > > > > When merging I did not use the NiFi processors to test as I already > > > have > > > > > tooling around H20 driven ML in my home. While I don't admittedly > use > > > it > > > > > very often, I thought the python processors were quite cool and > could > > > > > certainly be useful for others. My rationale is that the > dependencies > > > are > > > > > not built into source or binary for MiNiFi C++. I would agree on > the > > > Java > > > > > side that this would be unavoidable. While it would be preferential > > to > > > > have > > > > > the NiFi analogue I do not think it would be required when > > considering > > > > the > > > > > MiNiFi C++ processors; however, James should be the arbiter of that > > > > > decision. > > > > > > > > > > While I disagree that the MiNiFi portion should be removed, I don't > > > have > > > > a > > > > > strong argument to keep or remove the MiNiFI C++ code so I'd be > happy > > > to > > > > > revert if the community feels the need to. > > > > > > > > > > Thanks, > > > > > Marc > > > > > > > > > > On Mon, May 4, 2020 at 10:50 AM Joe Witt <[email protected]> > wrote: > > > > > > > > > >> Team > > > > >> > > > > >> The below two commits raise serious concerns. > > > > >> > > > > >> I want to be clear here and point out that h2o is cool. Having > such > > > > >> integration is a neat concept and idea and one that certainly > > warrants > > > > >> determination on how best to do so. > > > > >> > > > > >> My issue is with these two commits as it relates to licensing and > > > > >> maintenance. > > > > >> > > > > >> License: > > > > >> We should support vendors like this wanting to bring their > > > capabilities > > > > >> into NiFi generally sure. But the licensing and mode of use is > > > critical. > > > > >> In talking about this with the contributor for NiFi as well it is > > > clear > > > > >> that at least some or important portions of this require the user > to > > > > have > > > > >> a > > > > >> 'driverless ai license' so they can include their jar or build > > > pipelines > > > > >> or > > > > >> whatnot. Thus it isn't usable without that first. So it might be > > the > > > > case > > > > >> that this stuff is source dependent only on ASF compatible > licenses > > in > > > > >> terms of source - but it certainly doesn't seem to be true in > terms > > of > > > > >> binary dependencies. Where in the PR or JIRA is there any > discussion > > > or > > > > >> review of the licensing? I see plenty from James to believe this > > > needs > > > > to > > > > >> be reverted. Any contrary info? > > > > >> https://github.com/apache/nifi/pull/4242#issuecomment-622654527 > > > > >> > > > > >> > > > > >> Maintenance: > > > > >> We should find a way for vendors to offer extension points like > this > > > > >> without having to take on the burden of maintenance. How can we > > > possibly > > > > >> do > > > > >> this well? We're learning this lesson the hard way in NiFi itself > > and > > > > >> this > > > > >> is why the registry is being formulated. > > > > >> > > > > >> JIRA/hygiene: > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1199 > > > > >> and > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1201 > > > > >> Both are open. Yet we've merged commits that claim to be against > > > each. > > > > >> > > > > >> I believe both of these commits need to be reverted as I do not > > > believe > > > > >> the > > > > >> licensing considerations have been addressed. I'd like to see > > > > discussion > > > > >> on the above maintenance concern as well but that is less > pressing. > > > > >> > > > > >> > > > > >> Thanks > > > > >> Joe > > > > >> > > > > >> > > > > >> commit 7206c62240647520cf35649868d5d87903a256c2 > > > > >> Author: James Medel <[email protected]> > > > > >> Date: Wed Apr 29 12:38:04 2020 -0700 > > > > >> > > > > >> MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766) > > > > >> > > > > >> MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi (#766) > > > > >> > > > > >> commit 6e5f96518764df7791519c0ee625a94a207ddc69 > > > > >> Author: James Medel <[email protected]> > > > > >> Date: Wed Apr 29 12:37:00 2020 -0700 > > > > >> > > > > >> MINIFI-1199: Integrate H2O Driverless AI PSP in MiNiFi (#763) > > > > >> > > > > >> * MINIFI-1199: Integrate H2O Driverless AI in MiNiFi > > > > >> > > > > >> MiNiFi C++ and H2O Driverless AI Integration via Custom Python > > > > >> Processors: > > > > >> Integrates MiNiFi C++ with H2O's Driverless AI by Using > > Driverless > > > > >> AI's > > > > >> Python Scoring Pipeline and MiNiFi's Custom Python Processors. > Uses > > > the > > > > >> Python Processors to execute the Python Scoring Pipeline scorer to > > do > > > > >> batch > > > > >> scoring and real-time scoring for one or more predicted labels on > > test > > > > >> data > > > > >> in the incoming flow file content. I would like to contribute my > > > > >> processors > > > > >> to MiNiFi C++ as a new feature. > > > > >> > > > > >> * Update H2oPspScoreBatches processor > > > > >> > > > > >> This update includes passing "index=False" to > > > > >> "batch_scores_df.to_string(index=False)". By updating this line of > > > code, > > > > >> we > > > > >> tell pandas to not include the DataFrame index when converting the > > > > >> DataFrame to a string. The reason for this update is that we don't > > > want > > > > >> this extra column pandas tries to add to our predictions frame, we > > > only > > > > >> want the predictions. Thus, later when the predictions get saved > to > > a > > > > csv > > > > >> file, it will only include the predictions. > > > > >> > > > > >> * Moved ConvertDsToCsv to h2o base dir > > > > >> > > > > >> Since this ConvertDsToCsv python processor is used by the > > > > >> Python Scoring Pipeline and MOJO Scoring Pipeline processors, > > > > >> I moved ConvertDsToCsv to h2o base dir for easier access to > it. > > > > >> > > > > > > > > > > > > > > >
