David, +1 to all of this. Especially for guaranteeing compatibility and maintainability, I think instrumenting a more formal approach for updating the API is a step in the right direction. The huge amount of purging, cleanup, and refactoring that has gone into 2.0 helps to highlight the importance of ensuring maintainability.
Thanks -Mark > On Aug 20, 2024, at 2:00 AM, Joe Witt <joe.w...@gmail.com> wrote: > > David > > Yeah well written and good points. > > I dont think this says we would relax our existing commitment we have shown > for things such as the http based api but rather is calling out specific > things which we will make changes to even more formalized. > > To that end +1 on the concept of proposals and +1 to breaking out the > nifi-api (fundamental designed extension points for flow development) in > particular. > > Thanks > Joe > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <a...@adamtaft.com> wrote: > >> These are all really appreciated concepts, David. Thank you for putting the >> thoughts and time in on this! >> >> Regarding this: >>> The NiFi REST API does not have quite the same level of concern, but may >> warrant inclusion. >> >> I hear what you're saying. However, the REST API (from my >> observation/experience) has gathered quite a number of useful tools and >> "hacks" for NiFi. Quite often, many different monitoring and alerting tools >> have been developed against the REST API by third parties and/or >> integrators of NiFi against their internal workflows. Having stable API >> versioning in the REST API possibly makes just as much sense as having the >> same for the nifi-api itself. This is a prime entry point for extensions >> and other features developed alongside NiFi, maybe even the weird stuff >> that you can't do with the nifi-api directly. >> >> Food for thought of course, but I would hope that we can treat the REST API >> as a proper first-class citizen in terms of documented versioning. It turns >> out it's quite a useful means for interacting with a running NiFi instance. >> >> /Adam >> >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann < >> exceptionfact...@apache.org> wrote: >> >>> Team, >>> >>> As we wrap up remaining items for NiFi 2, we should consider how to >>> continue improving the quality and maintainability of the NiFi >>> ecosystem. >>> >>> One primary focus of NiFi 2 has been the reduction of technical debt, >>> which involved the removal of numerous modules and thousands of lines >>> of code. In that process, it is worth highlighting that the core NiFi >>> API, and the NiFi Framework API, and the NiFi REST API have had >>> comparatively few breaking changes. This a testament to the quality of >>> the API design itself. The NiFi Version Schema and API Compatibility >>> [1] has provided a strong direction thus far. >>> >>> With that background, we should consider adopting a more formal >>> process around changes that impact the fundamental API contracts that >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of >>> this in the past, but did not include approval requirements. Kafka [3] >>> and Airflow [4] have more structured improvement proposal processes, >>> and that is what we should adopt going forward. >>> >>> Part of moving in this direction requires identifying the areas that >>> would require going through the Improvement Proposal process itself. >>> At minimum, this should include the nifi-api [5] module. The >>> nifi-framework-api [6] is also worth consideration for inclusion in >>> this category. The NiFi REST API does not have quite the same level of >>> concern, but may warrant inclusion. >>> >>> As part of this discussion, we should consider separating the nifi-api >>> module into its own repository, with its own versioning scheme. This >>> will provide a helpful distinction in terms of the scope of changes, >>> and allow the API to be released independently of the application, >>> providing strong version compatibility guarantees. >>> >>> Based on feedback for the general idea, I would be glad to draft a >>> NiFi Improvement Proposal page, outlining the recommended steps in >>> more detail so we can come to consensus on how this should work. >>> >>> Regards, >>> David Handermann >>> >>> [1] >>> >> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility >>> [2] >>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals >>> [3] >>> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >>> [4] >>> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals >>> [5] https://github.com/apache/nifi/tree/main/nifi-api >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api >>> >>
