I created a JIRA ticket to investigate and improve the performance of InvokeHTTP. It includes a flow definition for benchmarking the performance of both PostHTTP and InvokeHTTP.
https://issues.apache.org/jira/browse/NIFI-8968 Thanks, Mark On Fri, Jul 30, 2021 at 6:41 PM Adam Taft <[email protected]> wrote: > I'm not seeing the side thread that was going to discuss deprecation of > PostHTTP. Has that thread started and I just don't see it? > > One (significant?) concern with PostHTTP is the smooth integration of > NiFi-to-NiFi communication that is very transparently enabled with the > ListenHTTP and PostHTTP processors. There's some special logic in there for > handling flowfiles that InvokeHTTP doesn't really (nor should really) have. > > I know of several (large) NiFi installations that rely on the PostHTTP / > ListenHTTP combination. It has enabled NiFi to NiFi communication for folks > reluctant or unable to enable site-to-site type configuration. > > Honestly, I don't know why we'd want to "deprecate" any processor, as > opposed to just moving it to a new location. If these processors can be > ported and maintained to whatever the 2.0 API looks like, there's possibly > little harm keeping them around. > > And by the way, what happened to the "marketplace" concept? Is this being > considered for 2.0 as well? Because relocating the deprecated processors > to an external nar might be the best solution. Losing PostHTTP entirely I > think would be a mistake, but I'd conceptually support relocating it. > > Thanks, > > /Adam > > On Tue, Jul 27, 2021 at 2:11 PM Joe Witt <[email protected]> wrote: > > > Looks like we just need to knock out 5 JIRAs :) [1] > > > > I felt like we had a label folks were using at one point but quickly > > looking revealed nothing exciting. After this confluence page > > stabilizes a bit we can probably knock out some JIRAs and such. > > > > [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599 > > > > On Tue, Jul 27, 2021 at 1:06 PM Otto Fowler <[email protected]> > > wrote: > > > > > > I find myself wishing I had a list of all the jiras / issues that have > > > been put off for a 2.0 release because they required some change or > > another > > > :( > > > > > > From: Joe Witt <[email protected]> <[email protected]> > > > Reply: [email protected] <[email protected]> <[email protected]> > > > Date: July 27, 2021 at 12:30:35 > > > To: [email protected] <[email protected]> <[email protected]> > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > A few thoughts: > > > > > > 1. I would love to see deprecation notices show up in the UI in > > > various ways to help motivate users to move off things to more > > > supportable things. That is not a prerequisite for anything happening > > > however. Just a good feature/nice thing to do for users when someone > > > is able to tackle it. > > > > > > 2. The decision to deprecate something and to further remove it need > > > not mean there is a superior solution available. If that thing itself > > > isn't getting the love/attention it needs to be > > > maintained/supported/healthy going forward that alone is enough to > > > remove it. That might well be the case with PostHTTP [1] and for > > > comparison you can see how much effort has gone into InvokeHTTP [2]. > > > > > > 3. When discussing a 2.0 release each thing we add as a 'must do' the > > > further away from reality such a release will become. We'll have to > > > get very specific about 'musts' vs 'wants'. > > > > > > [1] > > > > > > https://github.com/apache/nifi/commits/11e9ff377333784974fa55f41483c4281d80da50/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java > > > [2] > > > > > > https://github.com/apache/nifi/commits/cc554a6b110dfa45767bcb13d834ea4265d6dfe6/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/InvokeHTTP.java > > > > > > On Tue, Jul 27, 2021 at 8:47 AM David Handermann > > > <[email protected]> wrote: > > > > > > > > Thanks Mark, providing a template or comparison statistics with Java > > > > versions and component configuration details would be very helpful. > If > > it > > > > is possible to run tests using a public API or deployable service, > that > > > > would also help confirm potential differences. > > > > > > > > Showing a deprecation notice in the UI could be helpful, perhaps as a > > > > configurable option. NIFI-8650 describes a general Flow Analysis > > > > capability, and it seems like that might be one possible way to > surface > > > > deprecation warnings. For something more specific to component > > > deprecation, > > > > it seems important to find a balance between making it obvious and > > making > > > > it something that ends up getting ignored. > > > > > > > > Regards, > > > > David Handermann > > > > > > > > On Tue, Jul 27, 2021 at 10:28 AM Mark Bean <[email protected]> > > wrote: > > > > > > > > > I'll start a new thread for PostHTTP when I get a template and/or > > > detailed > > > > > stats. > > > > > > > > > > I know the deprecation is noted in the documentation. That's a > > > necessary > > > > > and minimum level of notification. I was suggesting it be more > > obvious > > > in > > > > > the UI. I think it would be beneficial to somehow be aware of the > > > > > deprecation status simply by looking at the flow (perhaps even on > the > > > > > summary pages too), and without having to open the documentation > for > > > every > > > > > processor to confirm whether or not a component is marked as > > > deprecated. > > > > > > > > > > Thanks, > > > > > Mark > > > > > > > > > > > > > > > On Tue, Jul 27, 2021 at 11:16 AM David Handermann < > > > > > [email protected]> wrote: > > > > > > > > > > > Mark, > > > > > > > > > > > > Thanks for the feedback. It may be better to start a separate > > thread > > > on > > > > > > PostHTTP, but can you provide an example flow demonstrating the > > > > > performance > > > > > > differences between PostHTTP and InvokeHTTP? > > > > > > > > > > > > PostHTTP uses the Apache HttpComponents library, whereas > InvokeHTTP > > > uses > > > > > > OkHttp. NiFi 1.13.2 and 1.14.0 included major version updates for > > > OkHttp, > > > > > > so it would be important to test with the most recent version. It > > is > > > also > > > > > > worth noting that test classes for PostHTTP are now integration > > tests > > > as > > > > > > opposed to unit tests, which means they are not executed as part > of > > > the > > > > > > automated builds. > > > > > > > > > > > > The NiFi documentation includes the contents of the > > DeprecationNotice > > > for > > > > > > PostHTTP: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.PostHTTP/index.html > > > > > > > > > > > > Regards, > > > > > > David Handermann > > > > > > > > > > > > On Tue, Jul 27, 2021 at 9:56 AM Mark Bean <[email protected] > > > > > wrote: > > > > > > > > > > > > > I'm strongly in favor of reducing tech debt, and moving > > > deliberately > > > > > to a > > > > > > > 2.0 release. I have a concern with just one processor that is > > > currently > > > > > > > marked as deprecated: PostHTTP. (I have not evaluated > > specifically > > > any > > > > > > > other deprecated components; I cannot say if there are or are > not > > > > > similar > > > > > > > issues elsewhere.) I understand the rationale for deprecating > > this > > > > > > > processor in that it eliminates a processor whose functionality > > is > > > > > > > available elsewhere, specifically in the more flexible > > InvokeHTTP. > > > > > > However, > > > > > > > in my experience and testing, PostHTTP performs transfers with > > far > > > > > > greater > > > > > > > throughput than InvokeHTTP. I would not be in favor of removing > > > > > PostHTTP > > > > > > > unless/until InvokeHTTP is refactored to increase its > throughput > > > > > > > capability. > > > > > > > > > > > > > > Has anyone else continued to use PostHTTP over InvokeHTTP for > > > similar > > > > > > > reasons? Or, is there a performance-related configuration for > > > > > InvokeHTTP > > > > > > I > > > > > > > may have missed? > > > > > > > > > > > > > > Also, in order to help facilitate a smooth transition to 2.0 > > from a > > > > > user > > > > > > > perspective, would it be advisable to add some sort of visual > > > indicator > > > > > > in > > > > > > > the UI for components that are currently annotated as > > @Deprecated? > > > This > > > > > > > might help users proactively modify their flow prior to a > release > > > that > > > > > > > would otherwise break it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende <[email protected]> > > > wrote: > > > > > > > > > > > > > > > With the merging of MiNiFi and registry into the NiFi repo, > > we've > > > > > > > > actually gone more towards a mono-repo where everything is > > > released > > > > > > > > together. Eventually it would still be nice to have a smaller > > > base > > > > > > > > distribution containing just the framework and standard NARs, > > but > > > it > > > > > > > > is hard to tackle that until we provide a way for users to > > easily > > > > > > > > obtain the NARs in some other way. > > > > > > > > > > > > > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes < > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Given the major version number shift and the spliting up of > > > > > > processors > > > > > > > > into > > > > > > > > > multiple NAR's. I'd like to suggest that we start > > individually > > > > > > > versioning > > > > > > > > > individual NARs/ bundles. > > > > > > > > > > > > > > > > > > I can see this bringing a large number of benifits > including > > > making > > > > > > > Nifi > > > > > > > > > more deployable with things RPM, but also potentially > > allowing > > > > > those > > > > > > > that > > > > > > > > > have to deploy managed Nifi instances easier to upgrade. > > > > > > > > > > > > > > > > > > Edward > > > > > > > > > > > > > > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, < > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > The issue with updating the aws sdk is if it breaks any > one > > > of > > > > > the > > > > > > > > > > processors. > > > > > > > > > > the Web Gateway API invoke processor for example is not > > using > > > a > > > > > > high > > > > > > > > level > > > > > > > > > > purpose build client and may break. > > > > > > > > > > > > > > > > > > > > If we change the aws version, we need to coordinate in > > such a > > > way > > > > > > > that > > > > > > > > they > > > > > > > > > > all > > > > > > > > > > can come along reasonably. > > > > > > > > > > IE: what happens if 1 or 2 break but the rest or OK? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: David Handermann <[email protected]> > > > > > > > > > > <[email protected]> > > > > > > > > > > Reply: [email protected] <[email protected]> < > > > > > > > [email protected]> > > > > > > > > > > Date: July 26, 2021 at 09:33:42 > > > > > > > > > > To: [email protected] <[email protected]> < > > > > > [email protected] > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals > > > > > > > > > > > > > > > > > > > > Chris, > > > > > > > > > > > > > > > > > > > > Thanks for the reply and recommendations. It seems like > > some > > > of > > > > > the > > > > > > > > work to > > > > > > > > > > reorganize the module structure could be done outside of > a > > > major > > > > > > > > release, > > > > > > > > > > but it would be great to target any breaking changes for > > 2.0. > > > > > > > Perhaps a > > > > > > > > > > separate feature proposal on module restructuring, with > the > > > goal > > > > > of > > > > > > > > > > supporting optimized builds, would be a helpful way to > move > > > that > > > > > > part > > > > > > > > of > > > > > > > > > > the discussion forward. > > > > > > > > > > > > > > > > > > > > Regarding updating AWS SDK to version 2, it seems like > that > > > might > > > > > > be > > > > > > > > > > possible now. I haven't taken a close look at the > > referencing > > > > > > > > components, > > > > > > > > > > so I'm not sure about the level of effort involved. Minor > > > NiFi > > > > > > > version > > > > > > > > > > updates have incorporated major new versions of > > dependencies. > > > For > > > > > > > > example, > > > > > > > > > > NiFi 1.14 included an upgrade from Spring Framework 4 to > 5. > > > On > > > > > the > > > > > > > one > > > > > > > > > > hand, including the AWS SDK update as part of a major > > release > > > > > seems > > > > > > > > > > helpful, but unless there are changes that break existing > > > > > component > > > > > > > > > > properties, upgrading the AWS SDK could be worked > > > independently. > > > > > > > > Others may > > > > > > > > > > have more insight into particular usage of that library. > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson > > > > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > Might be worth considering refactoring the build as > part > > of > > > > > this > > > > > > > work > > > > > > > > > > too, > > > > > > > > > > > e.g. only building the bits of the repo affected by a > > > commit, > > > > > > etc. > > > > > > > - > > > > > > > > > > > discussed briefly in previous threads but don't think > any > > > > > changes > > > > > > > > made > > > > > > > > > > yet. > > > > > > > > > > > If NARs/components are likely to be split up and > > refactored > > > > > then > > > > > > > such > > > > > > > > > > work > > > > > > > > > > > around the build probably makes sense to consider. > > > > > > > > > > > > > > > > > > > > > > I've a couple of PRs open that include updates to > > > Elasticsearch > > > > > > > > versions > > > > > > > > > > > already, although I stopped at 7.10.2 (after which > > Elastic > > > > > > changed > > > > > > > > > > licence) > > > > > > > > > > > in case there were licence concerns. But more work can > be > > > done > > > > > to > > > > > > > > tidy up > > > > > > > > > > > the processors, absolutely. > > > > > > > > > > > > > > > > > > > > > > AWS libraries to v2 would seem a sensible move and a > > > refactor > > > > > of > > > > > > > > those > > > > > > > > > > > processors as well. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > Chris Sampson > > > > > > > > > > > > > > > > > > > > > > On Sat, 24 Jul 2021, 17:47 David Handermann, < > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for pointing out the standard NAR bundles, > Mark. > > > There > > > > > > > are a > > > > > > > > > > > number > > > > > > > > > > > > of components in the standard NAR bundles with > > particular > > > > > > > > dependencies > > > > > > > > > > > that > > > > > > > > > > > > would make more sense in separate NARs. Reorganizing > > the > > > > > > standard > > > > > > > > NAR > > > > > > > > > > to > > > > > > > > > > > > components with limited dependencies and wide > > > applicability > > > > > > would > > > > > > > > > > > > definitely help with future maintenance. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > There’s also some code that exists in order to > > maintain > > > > > > > backward > > > > > > > > > > > > > compatibility in the repositories. I would very > much > > > like > > > > > the > > > > > > > > > > > > repositories > > > > > > > > > > > > > to contain no unnecessary code. And swap file > format > > > > > supports > > > > > > > > really > > > > > > > > > > > old > > > > > > > > > > > > > formats. And the old impls of the repositories > > > themselves, > > > > > > like > > > > > > > > > > > > > PersistentProvRepo instead of WriteAheadProv Repo, > > etc. > > > > > Lots > > > > > > of > > > > > > > > stuff > > > > > > > > > > > > there > > > > > > > > > > > > > that can be removed. And some methods in > > ProcessSession > > > > > that > > > > > > > are > > > > > > > > > > never > > > > > > > > > > > > used > > > > > > > > > > > > > by any processor in the codebase but exists in the > > > public > > > > > API > > > > > > > so > > > > > > > > > > can’t > > > > > > > > > > > be > > > > > > > > > > > > > removed till 2.0. > > > > > > > > > > > > > > > > > > > > > > > > > > I think his is also a great time to clean up the > > > “standard > > > > > > > nar.” > > > > > > > > At > > > > > > > > > > > this > > > > > > > > > > > > > point, it’s something like 70 MB. And many of the > > > > > components > > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > > not > > > > > > > > > > > > > really “standard” - things like connecting to FTP & > > > SFTP > > > > > > > > servers, XML > > > > > > > > > > > > > processing, Jolt transform, etc. could potentially > be > > > moved > > > > > > > into > > > > > > > > > > other > > > > > > > > > > > > > nars. The > > > nifi-standard-content-viewer-1.15.0-SNAPSHOT.war > > > > > is > > > > > > > > 6.9 MB > > > > > > > > > > is > > > > > > > > > > > > not > > > > > > > > > > > > > necessary for stateless or minifi java. Lots of > > things > > > > > > probably > > > > > > > > to > > > > > > > > > > > > > reconsider within the standard nar. > > > > > > > > > > > > > > > > > > > > > > > > > > I definitely think this is a reasonable approach, > to > > > allow > > > > > > for > > > > > > > a > > > > > > > > 2.0 > > > > > > > > > > > that > > > > > > > > > > > > > is not a huge feature release but allows the > project > > to > > > be > > > > > > > > simpler > > > > > > > > > > and > > > > > > > > > > > > more > > > > > > > > > > > > > nimble. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > -Mark > > > > > > > > > > > > > > > > > > > > > > > > > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen < > > > > > > > > [email protected] > > > > > > > > > > > > <mailto: > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Russell, > > > > > > > > > > > > > > > > > > > > > > > > > > AFAICT from looking at Elastic's repos, the low > level > > > REST > > > > > > > > client is > > > > > > > > > > > > > still fine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java > > > > > > > > > > > > > > > > > > > > > > > > > > Our Elasticsearch support is spread over two NARs > at > > > > > present. > > > > > > > One > > > > > > > > > > uses > > > > > > > > > > > > > OkHttp and the other uses that low level Elastic > REST > > > > > client. > > > > > > > > > > > > > Therefore, I think we're fine on licensing for the > > > moment. > > > > > > > > > > > > > > > > > > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman < > > > > > > > > > > [email protected] > > > > > > > > > > > > > <mailto:[email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Bringing up Elastic also reminds me that the > Elastic > > > > > > framework > > > > > > > > has > > > > > > > > > > just > > > > > > > > > > > > > recently transitioned out of Open Source, so to > > > acknowledge > > > > > > > that, > > > > > > > > > > maybe > > > > > > > > > > > > > some effort toward OpenSearch--I say this not > > > understanding > > > > > > > > exactly > > > > > > > > > > how > > > > > > > > > > > > > this sort of thing is considered in a large-scale, > > > > > > world-class > > > > > > > > > > software > > > > > > > > > > > > > project like Apache NiFi. (I'm not a contributor, > > just > > > a > > > > > > > grateful > > > > > > > > > > > > > consumer.) > > > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 10:28 AM, Matt Burgess wrote: > > > > > > > > > > > > > Along with the itemized list for ancient components > > we > > > > > should > > > > > > > > look at > > > > > > > > > > > > > updating versions of drivers, SDKs, etc. for > external > > > > > systems > > > > > > > > such as > > > > > > > > > > > > > Elasticsearch, Cassandra, etc. There may be > breaking > > > > > changes > > > > > > > but > > > > > > > > 2.0 > > > > > > > > > > > > > is probably the right time to get things up to date > > to > > > make > > > > > > > them > > > > > > > > more > > > > > > > > > > > > > useful to more people. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough < > > > > > > > > [email protected] > > > > > > > > > > > > <mailto: > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > I'm a +1 for removing pretty much all of this > stuff. > > > There > > > > > > are > > > > > > > > > > security > > > > > > > > > > > > > implications to keeping old dependencies around, so > > the > > > > > more > > > > > > > old > > > > > > > > code > > > > > > > > > > > we > > > > > > > > > > > > > can remove the better. I agree that eventually we > > need > > > to > > > > > > move > > > > > > > to > > > > > > > > > > > > > supporting only Java 11+, and as our next release > > will > > > > > > probably > > > > > > > > be > > > > > > > > > > > about > > > > > > > > > > > > 4 > > > > > > > > > > > > > - 6 months from now that doesn't seem too soon. We > > > could > > > > > > > > potentially > > > > > > > > > > > > break > > > > > > > > > > > > > this in two and remove the deprecated processors > and > > > leave > > > > > > 1.x > > > > > > > on > > > > > > > > > > Java > > > > > > > > > > > 8, > > > > > > > > > > > > > and finally start on 2.x which would support only > > Java > > > 11. > > > > > > I'm > > > > > > > > unsure > > > > > > > > > > > of > > > > > > > > > > > > > what implications changing the date and time > handling > > > would > > > > > > > have > > > > > > > > - > > > > > > > > > > for > > > > > > > > > > > > > running systems that use long term historical logs, > > > > > > unexpected > > > > > > > > > > impacts > > > > > > > > > > > to > > > > > > > > > > > > > time logging could be a problem. > > > > > > > > > > > > > > > > > > > > > > > > > > As Joe says I think feature work will have to be > > > dedicated > > > > > to > > > > > > > > 2.x and > > > > > > > > > > > we > > > > > > > > > > > > > could support 1.x for security fixes for some > period > > of > > > > > time. > > > > > > > 2.x > > > > > > > > > > seems > > > > > > > > > > > > > like a gargantuan task but it's probably time to > get > > > > > started. > > > > > > > Not > > > > > > > > > > sure > > > > > > > > > > > > how > > > > > > > > > > > > > we handle all open PRs and the transition between > 1.x > > > and > > > > > > 2.x. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:57 AM Joe Witt < > > > > > > [email protected] > > > > > > > > > > <mailto: > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Jon > > > > > > > > > > > > > > > > > > > > > > > > > > You're right we have to be careful and you're right > > > there > > > > > are > > > > > > > > still > > > > > > > > > > > > > significant Java 8 users out there. But we also > have > > to > > > be > > > > > > > > careful > > > > > > > > > > > > > about security and sustainability of the codebase. > If > > > we > > > > > had > > > > > > > > talked > > > > > > > > > > > > > about this last year when that article came out I'd > > > have > > > > > > agreed > > > > > > > > it is > > > > > > > > > > > > > too early. Interestingly that link seems to get > > updated > > > > > and I > > > > > > > > tried > > > > > > > > > > > > > [1] and found more recent data (not sure how > recent). > > > > > Anyway > > > > > > it > > > > > > > > > > > > > suggests Java 8 is still the top dog but we see > good > > > growth > > > > > > on > > > > > > > > 11. In > > > > > > > > > > > > > my $dayjob this aligns to what I'm seeing too. > > > Customers > > > > > > didn't > > > > > > > > seem > > > > > > > > > > > > > to care about Java 11 until later half last year > and > > > now > > > > > > > > suddenly it > > > > > > > > > > > > > is all over the place. > > > > > > > > > > > > > > > > > > > > > > > > > > I think once we put out a NiFi 2.0 release we'd see > > > rapid > > > > > > > > decrease in > > > > > > > > > > > > > work on the 1.x line just being blunt. We did this > > many > > > > > years > > > > > > > ago > > > > > > > > > > > > > with 0.x to 1.x and we stood behind 0.x for a while > > > (maybe > > > > > a > > > > > > > > year or > > > > > > > > > > > > > so) but it was purely bug fix/security related > bits. > > We > > > > > would > > > > > > > > need to > > > > > > > > > > > > > do something similar. But feature work would almost > > > > > certainly > > > > > > > go > > > > > > > > to > > > > > > > > > > > > > the 2.x line. Maybe there are other workable models > > but > > > my > > > > > > > > instinct > > > > > > > > > > > > > suggests this is likely to follow a similar path. > > > > > > > > > > > > > > > > > > > > > > > > > > ...anyway I agree it isn't that easy of a call to > > dump > > > Java > > > > > > 8. > > > > > > > We > > > > > > > > > > > > > need to make the call in both the interests of the > > user > > > > > base > > > > > > > and > > > > > > > > the > > > > > > > > > > > > > contributor base of the community. > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > https://www.jetbrains.com/lp/devecosystem-2021/java/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:46 AM Joe Witt < > > > > > [email protected] > > > > > > > > <mailto: > > > > > > > > > > > > > [email protected]>> wrote: > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > Yeah the flow registry is a key part of it. But > also > > > now > > > > > you > > > > > > > can > > > > > > > > > > > > > download the flow definition in JSON (upload i > think > > is > > > > > there > > > > > > > now > > > > > > > > > > > > > too). Templates offered a series of challenges such > > as > > > we > > > > > > store > > > > > > > > them > > > > > > > > > > > > > in the flow definition which has made flows massive > > in > > > an > > > > > > > > unintended > > > > > > > > > > > > > way which isn't fun for cluster behavior. > > > > > > > > > > > > > > > > > > > > > > > > > > We have a couple cases where we headed down a > > > particular > > > > > > > concept > > > > > > > > and > > > > > > > > > > > > > came up with better approaches later. We need to > > > reconcile > > > > > > > these > > > > > > > > with > > > > > > > > > > > > > the benefit of hindsight, and while being careful > to > > be > > > not > > > > > > > > overly > > > > > > > > > > > > > disruptive to existing users, to reduce the > > > > > > > codebase/maintenance > > > > > > > > > > > > > burden and allow continued evolution of the > project. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman < > > > > > > > > > > [email protected] > > > > > > > > > > > > > <mailto:[email protected]>> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Joe, > > > > > > > > > > > > > > > > > > > > > > > > > > I apologize for the off-topic intrusion, but what > > > replaces > > > > > > > > templates? > > > > > > > > > > > > > The Registry? Templates rocked and we have used > them > > > since > > > > > > > 0.5.x. > > > > > > > > > > > > > > > > > > > > > > > > > > Russ > > > > > > > > > > > > > > > > > > > > > > > > > > On 7/23/21 8:31 AM, Joe Witt wrote: > > > > > > > > > > > > > David, > > > > > > > > > > > > > > > > > > > > > > > > > > I think this is a highly reasonable approach and > > such a > > > > > focus > > > > > > > > will > > > > > > > > > > > > > greatly help make a 2.0 release far more > approachable > > > to > > > > > > knock > > > > > > > > out. > > > > > > > > > > > > > Not only that but tech debt reduction would help > make > > > work > > > > > > > > towards > > > > > > > > > > > > > major features we'd think about in a 'major > release' > > > sense > > > > > > more > > > > > > > > > > > > > approachable. > > > > > > > > > > > > > > > > > > > > > > > > > > We should remove all deprecated things (as well as > > > verify > > > > > we > > > > > > > > have the > > > > > > > > > > > > > right list). We should remove/consider removal of > > > > > deprecated > > > > > > > > > > > > > concepts > > > > > > > > > > > > > like templates. We should consider whether we can > > > resolve > > > > > the > > > > > > > > > > > > > various > > > > > > > > > > > > > ways we've handled what are now parameters down to > > one > > > > > clean > > > > > > > > > > > > > approach. > > > > > > > > > > > > > We should remove options in the nifi.properties > which > > > turn > > > > > > out > > > > > > > to > > > > > > > > > > > > > never be used quite right (if there are). There is > > > quite a > > > > > > bit > > > > > > > we > > > > > > > > > > > > > can > > > > > > > > > > > > > do purely in the name of tech debt reduction. > > > > > > > > > > > > > > > > > > > > > > > > > > Lots to consider here but I think this is the right > > > > > > discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > Than ks > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende < > > > > > > [email protected] > > > > > > > > > > <mailto: > > > > > > > > > > > > > [email protected]>> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I'm a +1 for this... Not sure if this falls under > > > "Removing > > > > > > > > > > > > > Deprecated > > > > > > > > > > > > > Components", but I think we should also look at > > > anything > > > > > that > > > > > > > has > > > > > > > > > > > > > been > > > > > > > > > > > > > marked as deprecated throughout the code base as a > > > > > candidate > > > > > > > for > > > > > > > > > > > > > removal. There are quite a few classes, methods, > > > > > properties, > > > > > > > etc > > > > > > > > > > > > > that > > > > > > > > > > > > > have been waiting for a chance to be removed. > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 23, 2021 at 10:13 AM David Handermann > > > > > > > > > > > > > <[email protected]<mailto: > > > > > > > [email protected] > > > > > > > > >> > > > > > > > > > > > wrote: > > > > > > > > > > > > > Team, > > > > > > > > > > > > > > > > > > > > > > > > > > With all of the excellent work that many have > > > contributed > > > > > to > > > > > > > NiFi > > > > > > > > > > > > > over the > > > > > > > > > > > > > years, the code base has also accumulated some > amount > > > of > > > > > > > > technical > > > > > > > > > > > > > debt. A > > > > > > > > > > > > > handful of components have been marked as > deprecated, > > > and > > > > > > some > > > > > > > > > > > > > components > > > > > > > > > > > > > remain in the code base to support integration with > > old > > > > > > > versions > > > > > > > > > > > > > of various > > > > > > > > > > > > > products. Following the principles of semantic > > > versioning, > > > > > > > > > > > > > introducing a > > > > > > > > > > > > > major release would provide the opportunity to > remove > > > these > > > > > > > > > > > > > deprecated and > > > > > > > > > > > > > unsupported components. > > > > > > > > > > > > > > > > > > > > > > > > > > Rather than focusing the next major release on new > > > > > features, > > > > > > > what > > > > > > > > > > > > > do you > > > > > > > > > > > > > think about focusing on technical debt removal? > This > > > > > approach > > > > > > > > > > > > > would not > > > > > > > > > > > > > make for the most interesting release, but it > > provides > > > the > > > > > > > > > > > > > opportunity to > > > > > > > > > > > > > clean up elements that involve breaking changes. > > > > > > > > > > > > > > > > > > > > > > > > > > Focusing on technical debt, at least three primary > > > goals > > > > > come > > > > > > > to > > > > > > > > > > > > > mind for > > > > > > > > > > > > > the next major release: > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Removal of deprecated and unmaintained > components > > > > > > > > > > > > > 2. Require Java 11 as the minimum supported version > > > > > > > > > > > > > 3. Transition internal date and time handling to > JSR > > > 310 > > > > > > > > java.time > > > > > > > > > > > > > components > > > > > > > > > > > > > > > > > > > > > > > > > > *Removing Deprecated Components* > > > > > > > > > > > > > > > > > > > > > > > > > > Removing support for older and deprecated > components > > > > > > provides a > > > > > > > > > > > > > great > > > > > > > > > > > > > opportunity to improve the overall security posture > > > when it > > > > > > > comes > > > > > > > > > > > > > to > > > > > > > > > > > > > maintaining dependencies. The OWASP dependency > plugin > > > > > report > > > > > > > > > > > > > currently > > > > > > > > > > > > > generates 50 MB of HTML for questionable > > dependencies, > > > many > > > > > > of > > > > > > > > > > > > > which are > > > > > > > > > > > > > related to old versions of various libraries. > > > > > > > > > > > > > > > > > > > > > > > > > > As a starting point, here are a handful of > components > > > and > > > > > > > > > > > > > extension modules > > > > > > > > > > > > > that could be targeted for removal in a major > > version: > > > > > > > > > > > > > > > > > > > > > > > > > > - PostHTTP and GetHTTP > > > > > > > > > > > > > - ListenLumberjack and the entire > > > nifi-lumberjack-bundle > > > > > > > > > > > > > - ListenBeats and the entire nifi-beats-bundle > > > > > > > > > > > > > - Elasticsearch 5 components > > > > > > > > > > > > > - Hive 1 and 2 components > > > > > > > > > > > > > > > > > > > > > > > > > > *Requiring Java 11* > > > > > > > > > > > > > > > > > > > > > > > > > > Java 8 is now over seven years old, and NiFi has > > > supported > > > > > > > > general > > > > > > > > > > > > > compatibility with Java 11 for several years. NiFi > > > 1.14.0 > > > > > > > > > > > > > incorporated > > > > > > > > > > > > > internal improvements specifically related to TLS > > 1.3, > > > > > which > > > > > > > > > > > > > allowed > > > > > > > > > > > > > closing out the long-running Java 11 compatibility > > epic > > > > > > > > NIFI-5174. > > > > > > > > > > > > > Making > > > > > > > > > > > > > Java 11 the minimum required version provides the > > > > > opportunity > > > > > > > to > > > > > > > > > > > > > address > > > > > > > > > > > > > any lingering edge cases and put NiFi in a better > > > position > > > > > to > > > > > > > > > > > > > support > > > > > > > > > > > > > current Java versions. > > > > > > > > > > > > > > > > > > > > > > > > > > *JSR 310 for Date and Time Handling* > > > > > > > > > > > > > > > > > > > > > > > > > > Without making the scope too broad, transitioning > > > internal > > > > > > date > > > > > > > > > > > > > and time > > > > > > > > > > > > > handling to use DateTimeFormatter instead of > > > > > SimpleDateFormat > > > > > > > > > > > > > would provide > > > > > > > > > > > > > a number of advantages. The Java Time components > > > provide > > > > > much > > > > > > > > > > > > > better > > > > > > > > > > > > > clarity when it comes to handling localized date > and > > > time > > > > > > > > > > > > > representations, > > > > > > > > > > > > > and also avoid the inherent confusion of > > java.sql.Date > > > > > > > extending > > > > > > > > > > > > > java.util.Date. Many internal components, > > specifically > > > > > > > > > > > > > Record-oriented > > > > > > > > > > > > > processors and services, rely on date parsing, > > leading > > > to > > > > > > > > > > > > > confusion and > > > > > > > > > > > > > various workarounds. The pattern formats of > > > > > SimpleDateFormat > > > > > > > and > > > > > > > > > > > > > DateTimeFormatter are very similar, but there are a > > few > > > > > > subtle > > > > > > > > > > > > > differences. > > > > > > > > > > > > > Making this transition would provide a much better > > > > > foundation > > > > > > > > > > > > > going forward. > > > > > > > > > > > > > *Conclusion* > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for giving this proposal some consideration. > > > Many of > > > > > > you > > > > > > > > > > > > > have been > > > > > > > > > > > > > developing NiFi for years and I look forward to > your > > > > > > feedback. > > > > > > > I > > > > > > > > > > > > > would be > > > > > > > > > > > > > glad to put together a more formalized > recommendation > > > on > > > > > > > > > > > > > Confluence and > > > > > > > > > > > > > write up Jira epics if this general approach sounds > > > > > agreeable > > > > > > > to > > > > > > > > > > > > > the > > > > > > > > > > > > > community. > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > David Handermann > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
