Thanks for the reply Joe. If the "marketplace" concept can be scoped to something like "not in core NiFi but can be downloaded here", then it could fit in the definition of achievable in 2.0. The concept of the marketplace being this magical dynamic processor lookup service always felt a little too vaporware to me.
One thing that would help the NiFi build size in general is to really focus on what needs to be in "core" and what can just be external nars that the dataflow/systems manager can add to the 'extensions' directory. We've struggled with having really large builds, so maybe this is an opportunity to address that in 2.0. Radical brainstorm warning: Imagine a completely "thin" NiFi distribution. e.g. with no processors or other components at all. Then a configuration manager could source additional nar functionality from a different distribution source. We could still build processor bundles, but they would just be completely separate from the framework. Separate git repository, separate versioning, separate download, etc. This would be the baby steps towards the marketplace concept. Let the ecosystem of processors and other components thrive outside of the main framework distribution. In this way, things like PostHTTP could live on easily without burdening the framework. And groups of similar processor functionality can exist in separately managed nars. Thanks for entertaining the thought at least. /Adam On Fri, Jul 30, 2021 at 4:50 PM Joe Witt <[email protected]> wrote: > Adam > > In cases of ‘what happened’ it just hasnt happened. The ideas are still > good. > > I do agree with the ‘why kill anything’ mentality. We can simply park > these somewhere. That said there is an ongoing maint cost we do have to > rationalize. Builds are longer, deps to maintain, etc.. > > Also we need to prob pumps the breaks a bit on thoughs of removing or > changing many things. If we do we may well get to 2.0 but nobody could > adopt it. We have honestly a pretty massive deployment base and we cannot > make it a pain for users. We got away with things at 0.x to 1.0 that we > cannot get away with on 2.0 > > Thanks > > On Fri, Jul 30, 2021 at 3:41 PM Adam Taft <[email protected]> wrote: > > > I'm not seeing the side thread that was going to discuss deprecation of > > PostHTTP. Has that thread started and I just don't see it? > > > > One (significant?) concern with PostHTTP is the smooth integration of > > NiFi-to-NiFi communication that is very transparently enabled with the > > ListenHTTP and PostHTTP processors. There's some special logic in there > for > > handling flowfiles that InvokeHTTP doesn't really (nor should really) > have. > > > > I know of several (large) NiFi installations that rely on the PostHTTP / > > ListenHTTP combination. It has enabled NiFi to NiFi communication for > folks > > reluctant or unable to enable site-to-site type configuration. > > > > Honestly, I don't know why we'd want to "deprecate" any processor, as > > opposed to just moving it to a new location. If these processors can be > > ported and maintained to whatever the 2.0 API looks like, there's > possibly > > little harm keeping them around. > > > > And by the way, what happened to the "marketplace" concept? Is this being > > considered for 2.0 as well? Because relocating the deprecated processors > > to an external nar might be the best solution. Losing PostHTTP entirely I > > think would be a mistake, but I'd conceptually support relocating it. > > > > Thanks, > > > > /Adam > > > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <[email protected]> wrote: > > > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > > > I felt like we had a label folks were using at one point but quickly > > > looking revealed nothing exciting. After this confluence page > > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <[email protected]> > > > wrote: > > > > > > > > I find myself wishing I had a list of all the jiras / issues that > have > > > > been put off for a 2.0 release because they required some change or > > > another > > > > :( > > > > > > > > From: Joe Witt <[email protected]> <[email protected]> > > > > Reply: [email protected] <[email protected]> < > [email protected]> > > > > Date: July 27, 2021 at 12:30:35 > > > > To: [email protected] <[email protected]> <[email protected]> > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > A few thoughts: > > > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > > various ways to help motivate users to move off things to more > > > > supportable things. That is not a prerequisite for anything happening > > > > however. Just a good feature/nice thing to do for users when someone > > > > is able to tackle it. > > > > > > > > 2. The decision to deprecate something and to further remove it need > > > > not mean there is a superior solution available. If that thing itself > > > > isn't getting the love/attention it needs to be > > > > maintained/supported/healthy going forward that alone is enough to > > > > remove it. That might well be the case with PostHTTP [1] and for > > > > comparison you can see how much effort has gone into InvokeHTTP [2]. > > > > > > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the > > > > further away from reality such a release will become. We'll have to > > > > get very specific about 'musts' vs 'wants'. > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > > > [2] > > > > > > > > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > > > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > > > <[email protected]> wrote: > > > > > > > > > > Thanks Mark, providing a template or comparison statistics with > Java > > > > > versions and component configuration details would be very helpful. > > If > > > it > > > > > is possible to run tests using a public API or deployable service, > > that > > > > > would also help confirm potential differences. > > > > > > > > > > Showing a deprecation notice in the UI could be helpful, perhaps > as a > > > > > configurable option. NIFI-8650 describes a general Flow Analysis > > > > > capability, and it seems like that might be one possible way to > > surface > > > > > deprecation warnings. For something more specific to component > > > > deprecation, > > > > > it seems important to find a balance between making it obvious and > > > making > > > > > it something that ends up getting ignored. > > > > > > > > > > Regards, > > > > > David Handermann > > > > > > > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <[email protected]> > > > wrote: > > > > > > > > > > > I'll start a new thread for PostHTTP when I get a template and/or > > > > detailed > > > > > > stats. > > > > > > > > > > > > I know the deprecation is noted in the documentation. That's a > > > > necessary > > > > > > and minimum level of notification. I was suggesting it be more > > > obvious > > > > in > > > > > > the UI. I think it would be beneficial to somehow be aware of the > > > > > > deprecation status simply by looking at the flow (perhaps even on > > the > > > > > > summary pages too), and without having to open the documentation > > for > > > > every > > > > > > processor to confirm whether or not a component is marked as > > > > deprecated. > > > > > > > > > > > > Thanks, > > > > > > Mark > > > > > > > > > > > > > > > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Mark, > > > > > > > > > > > > > > Thanks for the feedback. It may be better to start a separate > > > thread > > > > on > > > > > > > PostHTTP, but can you provide an example flow demonstrating the > > > > > > performance > > > > > > > differences between PostHTTP and InvokeHTTP? > > > > > > > > > > > > > > PostHTTP uses the Apache HttpComponents library, whereas > > InvokeHTTP > > > > uses > > > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates > for > > > > OkHttp, > > > > > > > so it would be important to test with the most recent version. > It > > > is > > > > also > > > > > > > worth noting that test classes for PostHTTP are now integration > > > tests > > > > as > > > > > > > opposed to unit tests, which means they are not executed as > part > > of > > > > the > > > > > > > automated builds. > > > > > > > > > > > > > > The NiFi documentation includes the contents of the > > > DeprecationNotice > > > > for > > > > > > > PostHTTP: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > > > > > > > > > > > Regards, > > > > > > > David Handermann > > > > > > > > > > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean < > [email protected] > > > > > > > wrote: > > > > > > > > > > > > > > > I'm strongly in favor of reducing tech debt, and moving > > > > deliberately > > > > > > to a > > > > > > > > 2.0 release. I have a concern with just one processor that is > > > > currently > > > > > > > > marked as deprecated: PostHTTP. (I have not evaluated > > > specifically > > > > any > > > > > > > > other deprecated components; I cannot say if there are or are > > not > > > > > > similar > > > > > > > > issues elsewhere.) I understand the rationale for deprecating > > > this > > > > > > > > processor in that it eliminates a processor whose > functionality > > > is > > > > > > > > available elsewhere, specifically in the more flexible > > > InvokeHTTP. > > > > > > > However, > > > > > > > > in my experience and testing, PostHTTP performs transfers > with > > > far > > > > > > > greater > > > > > > > > throughput than InvokeHTTP. I would not be in favor of > removing > > > > > > PostHTTP > > > > > > > > unless/until InvokeHTTP is refactored to increase its > > throughput > > > > > > > > capability. > > > > > > > > > > > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for > > > > similar > > > > > > > > reasons? Or, is there a performance-related configuration for > > > > > > InvokeHTTP > > > > > > > I > > > > > > > > may have missed? > > > > > > > > > > > > > > > > Also, in order to help facilitate a smooth transition to 2.0 > > > from a > > > > > > user > > > > > > > > perspective, would it be advisable to add some sort of visual > > > > indicator > > > > > > > in > > > > > > > > the UI for components that are currently annotated as > > > @Deprecated? > > > > This > > > > > > > > might help users proactively modify their flow prior to a > > release > > > > that > > > > > > > > would otherwise break it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende < > [email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > With the merging of MiNiFi and registry into the NiFi repo, > > > we've > > > > > > > > > actually gone more towards a mono-repo where everything is > > > > released > > > > > > > > > together. Eventually it would still be nice to have a > smaller > > > > base > > > > > > > > > distribution containing just the framework and standard > NARs, > > > but > > > > it > > > > > > > > > is hard to tackle that until we provide a way for users to > > > easily > > > > > > > > > obtain the NARs in some other way. > > > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Given the major version number shift and the spliting up > of > > > > > > > processors > > > > > > > > > into > > > > > > > > > > multiple NAR's. I'd like to suggest that we start > > > individually > > > > > > > > versioning > > > > > > > > > > individual NARs/ bundles. > > > > > > > > > > > > > > > > > > > > I can see this bringing a large number of benifits > > including > > > > making > > > > > > > > Nifi > > > > > > > > > > more deployable with things RPM, but also potentially > > > allowing > > > > > > those > > > > > > > > that > > > > > > > > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > > > > > > > > > > > > > > > Edward > > > > > > > > > > > > > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, < > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > The issue with updating the aws sdk is if it breaks any > > one > > > > of > > > > > > the > > > > > > > > > > > processors. > > > > > > > > > > > the Web Gateway API invoke processor for example is not > > > using > > > > a > > > > > > > high > > > > > > > > > level > > > > > > > > > > > purpose build client and may break. > > > > > > > > > > > > > > > > > > > > > > If we change the aws version, we need to coordinate in > > > such a > > > > way > > > > > > > > that > > > > > > > > > they > > > > > > > > > > > all > > > > > > > > > > > can come along reasonably. > > > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: David Handermann <[email protected]> > > > > > > > > > > > <[email protected]> > > > > > > > > > > > Reply: [email protected] <[email protected]> < > > > > > > > > [email protected]> > > > > > > > > > > > Date: July 26, 2021 at 09:33:42 > > > > > > > > > > > To: [email protected] <[email protected]> < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > > > > > > > > > > > > > > > Chris, > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply and recommendations. It seems like > > > some > > > > of > > > > > > the > > > > > > > > > work to > > > > > > > > > > > reorganize the module structure could be done outside > of > > a > > > > major > > > > > > > > > release, > > > > > > > > > > > but it would be great to target any breaking changes > for > > > 2.0. > > > > > > > > Perhaps a > > > > > > > > > > > separate feature proposal on module restructuring, with > > the > > > > goal > > > > > > of > > > > > > > > > > > supporting optimized builds, would be a helpful way to > > move > > > > that > > > > > > > part > > > > > > > > > of > > > > > > > > > > > the discussion forward. > > > > > > > > > > > > > > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like > > that > > > > might > > > > > > > be > > > > > > > > > > > possible now. I haven't taken a close look at the > > > referencing > > > > > > > > > components, > > > > > > > > > > > so I'm not sure about the level of effort involved. > Minor > > > > NiFi > > > > > > > > version > > > > > > > > > > > updates have incorporated major new versions of > > > dependencies. > > > > For > > > > > > > > > example, > > > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 > to > > 5. > > > > On > > > > > > the > > > > > > > > one > > > > > > > > > > > hand, including the AWS SDK update as part of a major > > > release > > > > > > seems > > > > > > > > > > > helpful, but unless there are changes that break > existing > > > > > > component > > > > > > > > > > > properties, upgrading the AWS SDK could be worked > > > > independently. > > > > > > > > > Others may > > > > > > > > > > > have more insight into particular usage of that > library. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Might be worth considering refactoring the build as > > part > > > of > > > > > > this > > > > > > > > work > > > > > > > > > > > too, > > > > > > > > > > > > e.g. only building the bits of the repo affected by a > > > > commit, > > > > > > > etc. > > > > > > > > - > > > > > > > > > > > > discussed briefly in previous threads but don't think > > any > > > > > > changes > > > > > > > > > made > > > > > > > > > > > yet. > > > > > > > > > > > > If NARs/components are likely to be split up and > > > refactored > > > > > > then > > > > > > > > such > > > > > > > > > > > work > > > > > > > > > > > > around the build probably makes sense to consider. > > > > > > > > > > > > > > > > > > > > > > > > I've a couple of PRs open that include updates to > > > > Elasticsearch > > > > > > > > > versions > > > > > > > > > > > > already, although I stopped at 7.10.2 (after which > > > Elastic > > > > > > > changed > > > > > > > > > > > licence) > > > > > > > > > > > > in case there were licence concerns. But more work > can > > be > > > > done > > > > > > to > > > > > > > > > tidy up > > > > > > > > > > > > the processors, absolutely. > > > > > > > > > > > > > > > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a > > > > refactor > > > > > > of > > > > > > > > > those > > > > > > > > > > > > processors as well. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > Chris Sampson > > > > > > > > > > > > > > > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for pointing out the standard NAR bundles, > > Mark. > > > > There > > > > > > > > are a > > > > > > > > > > > > number > > > > > > > > > > > > > of components in the standard NAR bundles with > > > particular > > > > > > > > > dependencies > > > > > > > > > > > > that > > > > > > > > > > > > > would make more sense in separate NARs. > Reorganizing > > > the > > > > > > > standard > > > > > > > > > NAR > > > > > > > > > > > to > > > > > > > > > > > > > components with limited dependencies and wide > > > > applicability > > > > > > > would > > > > > > > > > > > > > definitely help with future maintenance. > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > There’s also some code that exists in order to > > > maintain > > > > > > > > backward > > > > > > > > > > > > > > compatibility in the repositories. I would very > > much > > > > like > > > > > > the > > > > > > > > > > > > > repositories > > > > > > > > > > > > > > to contain no unnecessary code. And swap file > > format > > > > > > supports > > > > > > > > > really > > > > > > > > > > > > old > > > > > > > > > > > > > > formats. And the old impls of the repositories > > > > themselves, > > > > > > > like > > > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv > Repo, > > > etc. > > > > > > Lots > > > > > > > of > > > > > > > > > stuff > > > > > > > > > > > > > there > > > > > > > > > > > > > > that can be removed. And some methods in > > > ProcessSession > > > > > > that > > > > > > > > are > > > > > > > > > > > never > > > > > > > > > > > > > used > > > > > > > > > > > > > > by any processor in the codebase but exists in > the > > > > public > > > > > > API > > > > > > > > so > > > > > > > > > > > can’t > > > > > > > > > > > > be > > > > > > > > > > > > > > removed till 2.0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think his is also a great time to clean up the > > > > “standard > > > > > > > > nar.” > > > > > > > > > At > > > > > > > > > > > > this > > > > > > > > > > > > > > point, it’s something like 70 MB. And many of the > > > > > > components > > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > > not > > > > > > > > > > > > > > really “standard” - things like connecting to > FTP & > > > > SFTP > > > > > > > > > servers, XML > > > > > > > > > > > > > > processing, Jolt transform, etc. could > potentially > > be > > > > moved > > > > > > > > into > > > > > > > > > > > other > > > > > > > > > > > > > > nars. The > > > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war > > > > > > is > > > > > > > > > 6.9 MB > > > > > > > > > > > is > > > > > > > > > > > > > not > > > > > > > > > > > > > > necessary for stateless or minifi java. Lots of > > > things > > > > > > > probably > > > > > > > > > to > > > > > > > > > > > > > > reconsider within the standard nar. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I definitely think this is a reasonable approach, > > to > > > > allow > > > > > > > for > > > > > > > > a > > > > > > > > > 2.0 > > > > > > > > > > > > that > > > > > > > > > > > > > > is not a huge feature release but allows the > > project > > > to > > > > be > > > > > > > > > simpler > > > > > > > > > > > and > > > > > > > > > > > > > more > > > > > > > > > > > > > > nimble. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > -Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > > > > > > > > [email protected] > > > > > > > > > > > > > <mailto: > > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Russell, > > > > > > > > > > > > > > > > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low > > level > > > > REST > > > > > > > > > client is > > > > > > > > > > > > > > still fine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > > > > > > > > > > > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs > > at > > > > > > present. > > > > > > > > One > > > > > > > > > > > uses > > > > > > > > > > > > > > OkHttp and the other uses that low level Elastic > > REST > > > > > > client. > > > > > > > > > > > > > > Therefore, I think we're fine on licensing for > the > > > > moment. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Bringing up Elastic also reminds me that the > > Elastic > > > > > > > framework > > > > > > > > > has > > > > > > > > > > > just > > > > > > > > > > > > > > recently transitioned out of Open Source, so to > > > > acknowledge > > > > > > > > that, > > > > > > > > > > > maybe > > > > > > > > > > > > > > some effort toward OpenSearch--I say this not > > > > understanding > > > > > > > > > exactly > > > > > > > > > > > how > > > > > > > > > > > > > > this sort of thing is considered in a > large-scale, > > > > > > > world-class > > > > > > > > > > > software > > > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor, > > > just > > > > a > > > > > > > > grateful > > > > > > > > > > > > > > consumer.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > > > > > > > > > > > Along with the itemized list for ancient > components > > > we > > > > > > should > > > > > > > > > look at > > > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for > > external > > > > > > systems > > > > > > > > > such as > > > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be > > breaking > > > > > > changes > > > > > > > > but > > > > > > > > > 2.0 > > > > > > > > > > > > > > is probably the right time to get things up to > date > > > to > > > > make > > > > > > > > them > > > > > > > > > more > > > > > > > > > > > > > > useful to more people. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > > > > > > > > [email protected] > > > > > > > > > > > > > <mailto: > > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > I'm a +1 for removing pretty much all of this > > stuff. > > > > There > > > > > > > are > > > > > > > > > > > security > > > > > > > > > > > > > > implications to keeping old dependencies around, > so > > > the > > > > > > more > > > > > > > > old > > > > > > > > > code > > > > > > > > > > > > we > > > > > > > > > > > > > > can remove the better. I agree that eventually we > > > need > > > > to > > > > > > > move > > > > > > > > to > > > > > > > > > > > > > > supporting only Java 11+, and as our next release > > > will > > > > > > > probably > > > > > > > > > be > > > > > > > > > > > > about > > > > > > > > > > > > > 4 > > > > > > > > > > > > > > - 6 months from now that doesn't seem too soon. > We > > > > could > > > > > > > > > potentially > > > > > > > > > > > > > break > > > > > > > > > > > > > > this in two and remove the deprecated processors > > and > > > > leave > > > > > > > 1.x > > > > > > > > on > > > > > > > > > > > Java > > > > > > > > > > > > 8, > > > > > > > > > > > > > > and finally start on 2.x which would support only > > > Java > > > > 11. > > > > > > > I'm > > > > > > > > > unsure > > > > > > > > > > > > of > > > > > > > > > > > > > > what implications changing the date and time > > handling > > > > would > > > > > > > > have > > > > > > > > > - > > > > > > > > > > > for > > > > > > > > > > > > > > running systems that use long term historical > logs, > > > > > > > unexpected > > > > > > > > > > > impacts > > > > > > > > > > > > to > > > > > > > > > > > > > > time logging could be a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As Joe says I think feature work will have to be > > > > dedicated > > > > > > to > > > > > > > > > 2.x and > > > > > > > > > > > > we > > > > > > > > > > > > > > could support 1.x for security fixes for some > > period > > > of > > > > > > time. > > > > > > > > 2.x > > > > > > > > > > > seems > > > > > > > > > > > > > > like a gargantuan task but it's probably time to > > get > > > > > > started. > > > > > > > > Not > > > > > > > > > > > sure > > > > > > > > > > > > > how > > > > > > > > > > > > > > we handle all open PRs and the transition between > > 1.x > > > > and > > > > > > > 2.x. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < > > > > > > > [email protected] > > > > > > > > > > > <mailto: > > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > > > > > > > > > You're right we have to be careful and you're > right > > > > there > > > > > > are > > > > > > > > > still > > > > > > > > > > > > > > significant Java 8 users out there. But we also > > have > > > to > > > > be > > > > > > > > > careful > > > > > > > > > > > > > > about security and sustainability of the > codebase. > > If > > > > we > > > > > > had > > > > > > > > > talked > > > > > > > > > > > > > > about this last year when that article came out > I'd > > > > have > > > > > > > agreed > > > > > > > > > it is > > > > > > > > > > > > > > too early. Interestingly that link seems to get > > > updated > > > > > > and I > > > > > > > > > tried > > > > > > > > > > > > > > [1] and found more recent data (not sure how > > recent). > > > > > > Anyway > > > > > > > it > > > > > > > > > > > > > > suggests Java 8 is still the top dog but we see > > good > > > > growth > > > > > > > on > > > > > > > > > 11. In > > > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too. > > > > Customers > > > > > > > didn't > > > > > > > > > seem > > > > > > > > > > > > > > to care about Java 11 until later half last year > > and > > > > now > > > > > > > > > suddenly it > > > > > > > > > > > > > > is all over the place. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd > see > > > > rapid > > > > > > > > > decrease in > > > > > > > > > > > > > > work on the 1.x line just being blunt. We did > this > > > many > > > > > > years > > > > > > > > ago > > > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a > while > > > > (maybe > > > > > > a > > > > > > > > > year or > > > > > > > > > > > > > > so) but it was purely bug fix/security related > > bits. > > > We > > > > > > would > > > > > > > > > need to > > > > > > > > > > > > > > do something similar. But feature work would > almost > > > > > > certainly > > > > > > > > go > > > > > > > > > to > > > > > > > > > > > > > > the 2.x line. Maybe there are other workable > models > > > but > > > > my > > > > > > > > > instinct > > > > > > > > > > > > > > suggests this is likely to follow a similar path. > > > > > > > > > > > > > > > > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to > > > dump > > > > Java > > > > > > > 8. > > > > > > > > We > > > > > > > > > > > > > > need to make the call in both the interests of > the > > > user > > > > > > base > > > > > > > > and > > > > > > > > > the > > > > > > > > > > > > > > contributor base of the community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < > > > > > > [email protected] > > > > > > > > > <mailto: > > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But > > also > > > > now > > > > > > you > > > > > > > > can > > > > > > > > > > > > > > download the flow definition in JSON (upload i > > think > > > is > > > > > > there > > > > > > > > now > > > > > > > > > > > > > > too). Templates offered a series of challenges > such > > > as > > > > we > > > > > > > store > > > > > > > > > them > > > > > > > > > > > > > > in the flow definition which has made flows > massive > > > in > > > > an > > > > > > > > > unintended > > > > > > > > > > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > > > > > > > > > > > > > > > > > > > We have a couple cases where we headed down a > > > > particular > > > > > > > > concept > > > > > > > > > and > > > > > > > > > > > > > > came up with better approaches later. We need to > > > > reconcile > > > > > > > > these > > > > > > > > > with > > > > > > > > > > > > > > the benefit of hindsight, and while being careful > > to > > > be > > > > not > > > > > > > > > overly > > > > > > > > > > > > > > disruptive to existing users, to reduce the > > > > > > > > codebase/maintenance > > > > > > > > > > > > > > burden and allow continued evolution of the > > project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > > > > > > > > > [email protected] > > > > > > > > > > > > > > <mailto:[email protected]>> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Joe, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what > > > > replaces > > > > > > > > > templates? > > > > > > > > > > > > > > The Registry? Templates rocked and we have used > > them > > > > since > > > > > > > > 0.5.x. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > > > > > > > > > David, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think this is a highly reasonable approach and > > > such a > > > > > > focus > > > > > > > > > will > > > > > > > > > > > > > > greatly help make a 2.0 release far more > > approachable > > > > to > > > > > > > knock > > > > > > > > > out. > > > > > > > > > > > > > > Not only that but tech debt reduction would help > > make > > > > work > > > > > > > > > towards > > > > > > > > > > > > > > major features we'd think about in a 'major > > release' > > > > sense > > > > > > > more > > > > > > > > > > > > > > approachable. > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should remove all deprecated things (as well > as > > > > verify > > > > > > we > > > > > > > > > have the > > > > > > > > > > > > > > right list). We should remove/consider removal of > > > > > > deprecated > > > > > > > > > > > > > > concepts > > > > > > > > > > > > > > like templates. We should consider whether we can > > > > resolve > > > > > > the > > > > > > > > > > > > > > various > > > > > > > > > > > > > > ways we've handled what are now parameters down > to > > > one > > > > > > clean > > > > > > > > > > > > > > approach. > > > > > > > > > > > > > > We should remove options in the nifi.properties > > which > > > > turn > > > > > > > out > > > > > > > > to > > > > > > > > > > > > > > never be used quite right (if there are). There > is > > > > quite a > > > > > > > bit > > > > > > > > we > > > > > > > > > > > > > > can > > > > > > > > > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Lots to consider here but I think this is the > right > > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Than ks > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < > > > > > > > [email protected] > > > > > > > > > > > <mailto: > > > > > > > > > > > > > > [email protected]>> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under > > > > "Removing > > > > > > > > > > > > > > Deprecated > > > > > > > > > > > > > > Components", but I think we should also look at > > > > anything > > > > > > that > > > > > > > > has > > > > > > > > > > > > > > been > > > > > > > > > > > > > > marked as deprecated throughout the code base as > a > > > > > > candidate > > > > > > > > for > > > > > > > > > > > > > > removal. There are quite a few classes, methods, > > > > > > properties, > > > > > > > > etc > > > > > > > > > > > > > > that > > > > > > > > > > > > > > have been waiting for a chance to be removed. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > > > > > > > > > <[email protected]<mailto: > > > > > > > > [email protected] > > > > > > > > > >> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Team, > > > > > > > > > > > > > > > > > > > > > > > > > > > > With all of the excellent work that many have > > > > contributed > > > > > > to > > > > > > > > NiFi > > > > > > > > > > > > > > over the > > > > > > > > > > > > > > years, the code base has also accumulated some > > amount > > > > of > > > > > > > > > technical > > > > > > > > > > > > > > debt. A > > > > > > > > > > > > > > handful of components have been marked as > > deprecated, > > > > and > > > > > > > some > > > > > > > > > > > > > > components > > > > > > > > > > > > > > remain in the code base to support integration > with > > > old > > > > > > > > versions > > > > > > > > > > > > > > of various > > > > > > > > > > > > > > products. Following the principles of semantic > > > > versioning, > > > > > > > > > > > > > > introducing a > > > > > > > > > > > > > > major release would provide the opportunity to > > remove > > > > these > > > > > > > > > > > > > > deprecated and > > > > > > > > > > > > > > unsupported components. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Rather than focusing the next major release on > new > > > > > > features, > > > > > > > > what > > > > > > > > > > > > > > do you > > > > > > > > > > > > > > think about focusing on technical debt removal? > > This > > > > > > approach > > > > > > > > > > > > > > would not > > > > > > > > > > > > > > make for the most interesting release, but it > > > provides > > > > the > > > > > > > > > > > > > > opportunity to > > > > > > > > > > > > > > clean up elements that involve breaking changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Focusing on technical debt, at least three > primary > > > > goals > > > > > > come > > > > > > > > to > > > > > > > > > > > > > > mind for > > > > > > > > > > > > > > the next major release: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained > > components > > > > > > > > > > > > > > 2. Require Java 11 as the minimum supported > version > > > > > > > > > > > > > > 3. Transition internal date and time handling to > > JSR > > > > 310 > > > > > > > > > java.time > > > > > > > > > > > > > > components > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Removing Deprecated Components* > > > > > > > > > > > > > > > > > > > > > > > > > > > > Removing support for older and deprecated > > components > > > > > > > provides a > > > > > > > > > > > > > > great > > > > > > > > > > > > > > opportunity to improve the overall security > posture > > > > when it > > > > > > > > comes > > > > > > > > > > > > > > to > > > > > > > > > > > > > > maintaining dependencies. The OWASP dependency > > plugin > > > > > > report > > > > > > > > > > > > > > currently > > > > > > > > > > > > > > generates 50 MB of HTML for questionable > > > dependencies, > > > > many > > > > > > > of > > > > > > > > > > > > > > which are > > > > > > > > > > > > > > related to old versions of various libraries. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As a starting point, here are a handful of > > components > > > > and > > > > > > > > > > > > > > extension modules > > > > > > > > > > > > > > that could be targeted for removal in a major > > > version: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - PostHTTP and GetHTTP > > > > > > > > > > > > > > - ListenLumberjack and the entire > > > > nifi-lumberjack-bundle > > > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle > > > > > > > > > > > > > > - Elasticsearch 5 components > > > > > > > > > > > > > > - Hive 1 and 2 components > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Requiring Java 11* > > > > > > > > > > > > > > > > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has > > > > supported > > > > > > > > > general > > > > > > > > > > > > > > compatibility with Java 11 for several years. > NiFi > > > > 1.14.0 > > > > > > > > > > > > > > incorporated > > > > > > > > > > > > > > internal improvements specifically related to TLS > > > 1.3, > > > > > > which > > > > > > > > > > > > > > allowed > > > > > > > > > > > > > > closing out the long-running Java 11 > compatibility > > > epic > > > > > > > > > NIFI-5174. > > > > > > > > > > > > > > Making > > > > > > > > > > > > > > Java 11 the minimum required version provides the > > > > > > opportunity > > > > > > > > to > > > > > > > > > > > > > > address > > > > > > > > > > > > > > any lingering edge cases and put NiFi in a better > > > > position > > > > > > to > > > > > > > > > > > > > > support > > > > > > > > > > > > > > current Java versions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > *JSR 310 for Date and Time Handling* > > > > > > > > > > > > > > > > > > > > > > > > > > > > Without making the scope too broad, transitioning > > > > internal > > > > > > > date > > > > > > > > > > > > > > and time > > > > > > > > > > > > > > handling to use DateTimeFormatter instead of > > > > > > SimpleDateFormat > > > > > > > > > > > > > > would provide > > > > > > > > > > > > > > a number of advantages. The Java Time components > > > > provide > > > > > > much > > > > > > > > > > > > > > better > > > > > > > > > > > > > > clarity when it comes to handling localized date > > and > > > > time > > > > > > > > > > > > > > representations, > > > > > > > > > > > > > > and also avoid the inherent confusion of > > > java.sql.Date > > > > > > > > extending > > > > > > > > > > > > > > java.util.Date. Many internal components, > > > specifically > > > > > > > > > > > > > > Record-oriented > > > > > > > > > > > > > > processors and services, rely on date parsing, > > > leading > > > > to > > > > > > > > > > > > > > confusion and > > > > > > > > > > > > > > various workarounds. The pattern formats of > > > > > > SimpleDateFormat > > > > > > > > and > > > > > > > > > > > > > > DateTimeFormatter are very similar, but there > are a > > > few > > > > > > > subtle > > > > > > > > > > > > > > differences. > > > > > > > > > > > > > > Making this transition would provide a much > better > > > > > > foundation > > > > > > > > > > > > > > going forward. > > > > > > > > > > > > > > *Conclusion* > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for giving this proposal some > consideration. > > > > Many of > > > > > > > you > > > > > > > > > > > > > > have been > > > > > > > > > > > > > > developing NiFi for years and I look forward to > > your > > > > > > > feedback. > > > > > > > > I > > > > > > > > > > > > > > would be > > > > > > > > > > > > > > glad to put together a more formalized > > recommendation > > > > on > > > > > > > > > > > > > > Confluence and > > > > > > > > > > > > > > write up Jira epics if this general approach > sounds > > > > > > agreeable > > > > > > > > to > > > > > > > > > > > > > > the > > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
