Hi everyone, I wanted to bring up a discussion around releasing the initial implementation of the Connectors feature that was introduced in NIP-11 [1]. The initial discussion happened on the mailing list back in August 2025 [2], followed by a vote in September 2025 to start the implementation work [3]. Since then, significant progress has been made over the past few months and we believe the feature is ready for an initial release.
As a reminder, the Connector concept introduces a new extension point in Apache NiFi that encapsulates a complete NiFi dataflow behind a higher-level, guided configuration experience. Rather than exposing the full complexity of a flow to the user, a Connector presents a structured set of Configuration Steps and Property Groups (for example: "Source Configuration", "Destination Configuration", "Transformations") that allow a user to configure an end-to-end data pipeline without needing to understand or manipulate the underlying flow directly. Under the hood, the Connector is responsible for dynamically building and reconfiguring its encapsulated flow based on the user's configuration choices. This includes managing the lifecycle (starting/stopping) of all components within the flow. The goal is to significantly simplify the user experience for common data movement use cases, making NiFi more accessible to a broader audience while also streamlining operations for experienced practitioners. Given the scope of the feature, the API is marked as experimental and may change in a non-backward compatible manner between minor versions until the API has been more fully validated through developing concrete Connector implementations.All of the development work happened in a feature branch named NIFI-15258 in both the nifi-api [4] and nifi [5] repositories. Here is a high-level summary of what was implemented: 1. NiFi API - The Connector API has been introduced under org.apache.nifi.components.connector. This includes the core Connector interface and an AbstractConnector base class, the ConfigurationStep and ConnectorPropertyGroup abstractions for structuring configuration, ConnectorPropertyDescriptor for defining properties with dependencies and allowable values, a FlowContext abstraction that provides both an active flow and a working flow for configuration verification, component facades (ProcessorFacade, ControllerServiceFacade, ConnectionFacade, ProcessGroupFacade, ParameterContextFacade) for Connectors to interact with their encapsulated flow, Secret references for handling sensitive values, Asset support, ConnectorMethod for exposing custom web endpoints, documentation writers for auto-generating Connector documentation, and updates to the Versioned flow model to accommodate Connectors. Comprehensive unit tests have been added for the API. 2. NiFi Maven NAR Plugin - Support has been added for creating Connector extension manifests in NAR files, so that the NiFi runtime can discover and load Connector extensions. This change is already on the main branch. 3. NiFi (Framework, REST API, UI, Tests) - The bulk of the implementation resides here: - Framework core: ConnectorNode and lifecycle management, connector facades for standalone process groups, a connector configuration repository, flow synchronization support for connectors, parameter context and secrets management integration, and cluster replication support. - REST API: Full set of endpoints for connector CRUD operations, lifecycle management (start/stop), configuration step retrieval and updates, property group management, status reporting, data drainage, connector actions, secrets, state management, verification, and documentation. - Frontend UI: Connectors are integrated into the flow designer, the documentation browser, and access policies. - Mock framework: A connector mock server and test runner to support unit and integration testing of Connector implementations. - Example connector: A Kafka-to-S3 connector implementation that serves as a reference for building Connectors. - CLI Toolkit: A ConnectorClient has been added for command-line interaction with Connectors. - System tests: 11 system test classes covering standalone and clustered scenarios including CRUD, lifecycle, assets, data drainage, parameter contexts, version resolution, and method marshalling. Release plan: We would like to proceed iteratively with three releases: - NiFi Maven NAR Plugin - This has no dependency on the other projects. The Connector manifest support is already on main so we can get started with a release candidate right away. I volunteer to take care of the release duties. - NiFi API - A pull request [6] has been filed to merge the NIFI-15258 feature branch into main. Once merged, we will proceed with a release candidate. - NiFi - Once the NiFi API release is published, we will proceed with the proper PRs (bumping the API version dependency and merging the NIFI-15258 feature branch), and we will look at releasing a new version of NiFi. Please have a look at the feature branches and share your feedback. Any comments, questions, or concerns are welcome. Thanks, Pierre [1] https://issues.apache.org/jira/browse/NIP-11 [2] https://lists.apache.org/thread/77c7o4yf8lt9owpjsjb8gl2slk3f7mcf [3] https://lists.apache.org/thread/skhn5kjkq6kfmrbqsshxs5348d968jvb [4] https://github.com/apache/nifi-api/tree/NIFI-15258 [5] https://github.com/apache/nifi/tree/NIFI-15258 [6] https://github.com/apache/nifi-api/pull/70
