[
https://issues.apache.org/jira/browse/NIP-11?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIP-11.
---------------------------------
Resolution: Fixed
> Introduce Connector extension point
> -----------------------------------
>
> Key: NIP-11
> URL: https://issues.apache.org/jira/browse/NIP-11
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> h1. Motivation
> The Apache NiFi ecosystem currently caters primarily to Data Engineers and
> Software Engineers, who possess the technical expertise required to navigate
> its learning curve. While NiFi offers powerful capabilities for creating and
> reusing versioned data flows, these flows often necessitate substantial
> modifications, particularly when adapting to diverse source/destination
> system authentication strategies. A consistent demand has emerged from less
> technical users for a more "self-service" solution, alongside requests for
> automated use cases where users can adjust a few properties without
> undertaking significant flow alterations. To address these challenges and
> broaden NiFi's accessibility, this proposal introduces a new "Connector"
> concept.
> h1. Scope
> The introduction of the Connector abstraction represents a significant
> undertaking that will necessitate extensive modifications across the NiFi
> architecture. This initiative will require substantial additions to the
> existing NiFi API and Framework. Furthermore, it will necessitate updates to
> the flow.json definition and the various Versioned object models to
> accommodate the new abstraction. Additionally, the user interface will
> require updates to provide a cohesive and intuitive experience for
> configuring and managing Connectors.
> h1. Description
> This proposal aims to introduce a new "Connector" abstraction and extension
> point within Apache NiFi. A Connector will encapsulate a complete NiFi flow,
> represented as a Process Group. This new entity will be responsible for the
> lifecycle management of its constituent components, including their starting
> and stopping. Crucially, a Connector will also be responsible for dynamically
> updating its encapsulated flow based on the configuration it exposes to the
> user. This dynamic update capability will encompass modifying Processor and
> Controller Service properties, and potentially even adding or removing
> components within the flow.
>
> Unlike a Processor, which typically presents a flat list of configurable
> properties, a Connector will feature a structured collection of
> PropertyGroups. This organizational approach will enable developers to
> logically separate different areas of configuration, such as distinct
> sections for "Source configuration," "Destination configuration,"
> "Transformations," and "Enrichments." This structured approach enhances
> clarity and ease of use for users.
>
> Use of Custom UIs, or even more tailored reusable UI components could be used
> in order to provide a clearer, more usable configuration for selecting source
> entities, as well. For example, a UI might offer the ability to choose which
> remote resources should be polled for data whereas we typically rely on
> domain specific languages such as glob patterns or regular expressions. While
> this is feasible at a Processor level, it can be made more reusable and
> consistent among components with Connectors, and that information can be used
> to more intelligently configure the flow for the user.
>
> Furthermore, the Connector abstraction will provide a robust foundation for
> more safely upgrading data flows from one version to another. This is
> achieved by enabling the execution of custom code to gracefully migrate
> configurations and flows. This capability extends beyond individual component
> updates, potentially allowing for complex operations such as migrating state
> between different components, which is not feasible at the individual
> component level.
>
> Finally, the Connector concept lays the groundwork for building future
> capabilities that are inherently meaningful at the flow level, as opposed to
> the single component (Processor or Controller Service) level. Examples of
> such capabilities include comprehensive validation of an entire flow,
> gathering holistic metrics that provide a more complete operational picture,
> or providing metrics specifically tailored for intelligent scaling decisions.
>
> h1. Expected API
> The NiFi API will be impacted significantly. While it’s difficult to entirely
> suss out the full API, an initial implementation can be found at
> [https://github.com/markap14/nifi-api/tree/connectors/src/main/java/org/apache/nifi/components/connector]
>
> Given the scope, the API will be noted as experimental and may change in a
> non-backward compatible manner between minor versions until the API has been
> more fully validated through developing concrete implementations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
