Hey Joe, I was trying to advocate for H2O-3 as a way to resolve this. I think I was responding too tersely and jumping around between tasks so I've created a follow on https://issues.apache.org/jira/browse/MINIFICPP-1214 to clarify. We can follow from there and anyone monitoring this thread should do the same so we're good from all concerned parties.
Thanks, Marc On Mon, May 4, 2020 at 12:38 PM Joe Witt <[email protected]> wrote: > Yep got ya. I'm talking about lines like > > > https://github.com/apache/nifi-minifi-cpp/commit/6e5f96518764df7791519c0ee625a94a207ddc69#diff-fa43d8bfc508fc5db1c2a320e3d03cb5R33 > > Does that seem fine to you? > > On Mon, May 4, 2020 at 12:29 PM Marc Parisi <[email protected]> wrote: > > > Hey Joe, > > > > "Yeah for the three/commit James replied on in that link - i dont see how > > we > > can keep those. They import from something that I doubt very much is > > compatible with ALv2/ASF guidelines." > > > > Sorry for confusing things but my statement was referring to the MiNiFi > C++ > > Python processors that don't import as java would. I've asked the > original > > author to chime in here so that he can comment on the MiNiFi C++ > > processors, especially on whether or not he feels the processors should > > stay and the other concerns raised. I'll continue to monitor this and > > create those follow up tickets when I am able to. > > > > Thanks, > > Marc > > > > On Mon, May 4, 2020 at 12:13 PM Joe Witt <[email protected]> wrote: > > > > > Yeah for the three/commit James replied on in that link - i dont see > how > > we > > > can keep those. They import from something that I doubt very much is > > > compatible with ALv2/ASF guidelines. > > > > > > On Mon, May 4, 2020 at 12:01 PM Marc Parisi <[email protected]> wrote: > > > > > > > Afternoon Joe, > > > > > > > > "So in the case of minificpp we are certain there is no L&N concern > > > > regarding either our source OR convenience binary?" > > > > > > > > I believe that to be the case; however, per > > > > https://github.com/apache/nifi/pull/4242#issuecomment-622655276 I > > think > > > > that maybe this is moot and the author can convert those processors > in > > > > MiNiFI to use that open source python lib. I'm not sure if he's on > this > > > > chain so I will follow up on the MiNiFi tickets to ensure that is the > > > case > > > > when I am able to log in again ( have to fix some account issue ). > > > > > > > > "Lastly, why are the JIRAs still open if this is merged? Can you > > please > > > > close them and assign to the proper fix version if these are 'done'?" > > > > > > > > Sorry, when I am able to log in ( account issue ) and make the above > > > > comment I will close the tx. I would prefer the fix be a follow on so > > > this > > > > is all tracked. The ticket fell off the radar ( as did the account > > issue, > > > > to be frank ), so when that is resolved I will take these actions. > You > > > know > > > > me Joe, I'm like a dog who sees a squirrel, but I've put a note on my > > > > laptop to serve as a reminder. > > > > > > > > Thanks, > > > > Marc > > > > > > > > On Mon, May 4, 2020 at 11:49 AM Joe Witt <[email protected]> wrote: > > > > > > > > > Marc > > > > > > > > > > I'm not opposed to it staying if there is zero license issue. I > only > > > > > mentioned the maintenance as a 'additional thing to consider' - > that > > is > > > > > more of a discussion item. L&N isn't discussion - it is hard rule. > > > > > > > > > > Ok so in the minificpp case you're saying "there is no source or > > binary > > > > > dependency (that we'd package)" that isn't entirely ALv2/ASF > > compatible > > > > and > > > > > so if people want to use this they can do so by adding in the 'h2o' > > > bits > > > > at > > > > > their own effort, right? In this line of thinking I still don't > love > > > it > > > > > but it is arguably not a lot different that our components which > talk > > > to > > > > > services you have to have a paid or otherwise agreement with like > > Cloud > > > > > services offering by AWS, Azure, GCP, etc.. > > > > > > > > > > So in the case of minificpp we are certain there is no L&N concern > > > > > regarding either our source OR convenience binary? > > > > > > > > > > Lastly, why are the JIRAs still open if this is merged? Can you > > please > > > > > close them and assign to the proper fix version if these are > 'done'? > > > > > > > > > > Thanks > > > > > > > > > > On Mon, May 4, 2020 at 11:46 AM Marc Parisi <[email protected]> > > > wrote: > > > > > > > > > > > "We should find a way for vendors to offer extension points like > > this > > > > > > without having to take on the burden of maintenance. How can we > > > > possibly > > > > > do > > > > > > this well? We're learning this lesson the hard way in NiFi > itself > > > and > > > > > this > > > > > > is why the registry is being formulated." > > > > > > > > > > > > I think there is a fine line when walking vendor tie-in. Having > the > > > > > > registry allows for an arm's length agreement with the Apache > > > > community. > > > > > I > > > > > > know others will disagree but my opinion is that extension points > > > that > > > > > > don't introduce licensing concerns from libraries should be fair > > game > > > > but > > > > > > the fact that a paid product requires a license for that code to > be > > > > > useful > > > > > > should not be a non-starter -- if that makes sense. I believe I > > > stated, > > > > > "I > > > > > > don't have a strong argument" as I don't really have a strong > > > opinion ( > > > > > and > > > > > > certainly can be wrong too ) -- so I'm interested in hearing > others > > > in > > > > > > regards to this. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 4, 2020 at 11:38 AM Marc Parisi <[email protected] > > > > > > wrote: > > > > > > > > > > > > > Hi Joe, > > > > > > > > > > > > > > When merging I did not use the NiFi processors to test as I > > already > > > > > have > > > > > > > tooling around H20 driven ML in my home. While I don't > admittedly > > > use > > > > > it > > > > > > > very often, I thought the python processors were quite cool and > > > could > > > > > > > certainly be useful for others. My rationale is that the > > > dependencies > > > > > are > > > > > > > not built into source or binary for MiNiFi C++. I would agree > on > > > the > > > > > Java > > > > > > > side that this would be unavoidable. While it would be > > preferential > > > > to > > > > > > have > > > > > > > the NiFi analogue I do not think it would be required when > > > > considering > > > > > > the > > > > > > > MiNiFi C++ processors; however, James should be the arbiter of > > that > > > > > > > decision. > > > > > > > > > > > > > > While I disagree that the MiNiFi portion should be removed, I > > don't > > > > > have > > > > > > a > > > > > > > strong argument to keep or remove the MiNiFI C++ code so I'd be > > > happy > > > > > to > > > > > > > revert if the community feels the need to. > > > > > > > > > > > > > > Thanks, > > > > > > > Marc > > > > > > > > > > > > > > On Mon, May 4, 2020 at 10:50 AM Joe Witt <[email protected]> > > > wrote: > > > > > > > > > > > > > >> Team > > > > > > >> > > > > > > >> The below two commits raise serious concerns. > > > > > > >> > > > > > > >> I want to be clear here and point out that h2o is cool. > Having > > > such > > > > > > >> integration is a neat concept and idea and one that certainly > > > > warrants > > > > > > >> determination on how best to do so. > > > > > > >> > > > > > > >> My issue is with these two commits as it relates to licensing > > and > > > > > > >> maintenance. > > > > > > >> > > > > > > >> License: > > > > > > >> We should support vendors like this wanting to bring their > > > > > capabilities > > > > > > >> into NiFi generally sure. But the licensing and mode of use is > > > > > critical. > > > > > > >> In talking about this with the contributor for NiFi as well it > > is > > > > > clear > > > > > > >> that at least some or important portions of this require the > > user > > > to > > > > > > have > > > > > > >> a > > > > > > >> 'driverless ai license' so they can include their jar or build > > > > > pipelines > > > > > > >> or > > > > > > >> whatnot. Thus it isn't usable without that first. So it might > > be > > > > the > > > > > > case > > > > > > >> that this stuff is source dependent only on ASF compatible > > > licenses > > > > in > > > > > > >> terms of source - but it certainly doesn't seem to be true in > > > terms > > > > of > > > > > > >> binary dependencies. Where in the PR or JIRA is there any > > > discussion > > > > > or > > > > > > >> review of the licensing? I see plenty from James to believe > > this > > > > > needs > > > > > > to > > > > > > >> be reverted. Any contrary info? > > > > > > >> > https://github.com/apache/nifi/pull/4242#issuecomment-622654527 > > > > > > >> > > > > > > >> > > > > > > >> Maintenance: > > > > > > >> We should find a way for vendors to offer extension points > like > > > this > > > > > > >> without having to take on the burden of maintenance. How can > we > > > > > possibly > > > > > > >> do > > > > > > >> this well? We're learning this lesson the hard way in NiFi > > itself > > > > and > > > > > > >> this > > > > > > >> is why the registry is being formulated. > > > > > > >> > > > > > > >> JIRA/hygiene: > > > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1199 > > > > > > >> and > > > > > > >> https://issues.apache.org/jira/browse/MINIFICPP-1201 > > > > > > >> Both are open. Yet we've merged commits that claim to be > > against > > > > > each. > > > > > > >> > > > > > > >> I believe both of these commits need to be reverted as I do > not > > > > > believe > > > > > > >> the > > > > > > >> licensing considerations have been addressed. I'd like to see > > > > > > discussion > > > > > > >> on the above maintenance concern as well but that is less > > > pressing. > > > > > > >> > > > > > > >> > > > > > > >> Thanks > > > > > > >> Joe > > > > > > >> > > > > > > >> > > > > > > >> commit 7206c62240647520cf35649868d5d87903a256c2 > > > > > > >> Author: James Medel <[email protected]> > > > > > > >> Date: Wed Apr 29 12:38:04 2020 -0700 > > > > > > >> > > > > > > >> MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi > (#766) > > > > > > >> > > > > > > >> MINIFI-1201: Integrate H2O Driverless AI MSP in MiNFi > (#766) > > > > > > >> > > > > > > >> commit 6e5f96518764df7791519c0ee625a94a207ddc69 > > > > > > >> Author: James Medel <[email protected]> > > > > > > >> Date: Wed Apr 29 12:37:00 2020 -0700 > > > > > > >> > > > > > > >> MINIFI-1199: Integrate H2O Driverless AI PSP in MiNiFi > > (#763) > > > > > > >> > > > > > > >> * MINIFI-1199: Integrate H2O Driverless AI in MiNiFi > > > > > > >> > > > > > > >> MiNiFi C++ and H2O Driverless AI Integration via Custom > > Python > > > > > > >> Processors: > > > > > > >> Integrates MiNiFi C++ with H2O's Driverless AI by Using > > > > Driverless > > > > > > >> AI's > > > > > > >> Python Scoring Pipeline and MiNiFi's Custom Python Processors. > > > Uses > > > > > the > > > > > > >> Python Processors to execute the Python Scoring Pipeline > scorer > > to > > > > do > > > > > > >> batch > > > > > > >> scoring and real-time scoring for one or more predicted labels > > on > > > > test > > > > > > >> data > > > > > > >> in the incoming flow file content. I would like to contribute > my > > > > > > >> processors > > > > > > >> to MiNiFi C++ as a new feature. > > > > > > >> > > > > > > >> * Update H2oPspScoreBatches processor > > > > > > >> > > > > > > >> This update includes passing "index=False" to > > > > > > >> "batch_scores_df.to_string(index=False)". By updating this > line > > of > > > > > code, > > > > > > >> we > > > > > > >> tell pandas to not include the DataFrame index when converting > > the > > > > > > >> DataFrame to a string. The reason for this update is that we > > don't > > > > > want > > > > > > >> this extra column pandas tries to add to our predictions > frame, > > we > > > > > only > > > > > > >> want the predictions. Thus, later when the predictions get > saved > > > to > > > > a > > > > > > csv > > > > > > >> file, it will only include the predictions. > > > > > > >> > > > > > > >> * Moved ConvertDsToCsv to h2o base dir > > > > > > >> > > > > > > >> Since this ConvertDsToCsv python processor is used by the > > > > > > >> Python Scoring Pipeline and MOJO Scoring Pipeline > > processors, > > > > > > >> I moved ConvertDsToCsv to h2o base dir for easier access > to > > > it. > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
