mcgilman opened a new pull request, #11789: URL: https://github.com/apache/nifi/pull/11789
Apply Connector property migrations when loading configuration provider documents and persist the migrated form during flow synchronization. Preserve runtime-only defaults, retry failed saves and inheritance, and atomically replace complete Connector configurations. # Summary [NIFI-16442](https://issues.apache.org/jira/browse/NIFI-16442) Connector property migration previously ran while inheriting flow configuration, but configuration loaded from a `ConnectorConfigurationProvider` remained authoritative and could reintroduce obsolete property names. This caused validation to report that migrated properties were not defined by the current Connector. This change applies Connector property migration consistently when provider-managed configuration is loaded and persists the migrated representation during flow synchronization. ## Approach The implementation separates runtime configuration from persisted provider configuration: - Provider reads run `Connector.migrateProperties()` and install the migrated configuration in memory. - Missing required properties with defaults are added to the runtime configuration so newly introduced required properties do not invalidate existing Connectors. - Runtime-only defaults are excluded from the provider document. - Undeclared properties and steps removed by migration are excluded from the persisted representation. - Connectors that declare no configuration steps, including ghost Connectors for unavailable extensions, retain their stored configuration. `ConnectorNode.migrateConfiguration()` produces the defaults-free representation suitable for persistence. `replaceWorkingConfiguration(List<VersionedConfigurationStep>)` produces and installs the complete runtime representation, including required defaults. Provider-backed reads remain read-only. They migrate and atomically replace the in-memory working configuration without saving. This ensures validation and REST responses immediately use current property names even when the provider document has not yet been updated. During flow synchronization, the repository: 1. Loads the original active and working configurations. 2. Produces migrated persisted forms for comparison and provider persistence. 3. Compares the migrated forms against runtime configuration while ignoring only framework-applied required defaults. 4. Passes the original persisted forms to inheritance so step removals performed by migration remain observable. 5. Saves a new `ConnectorWorkingConfiguration` only after inheritance completes successfully. If provider persistence fails, synchronization continues with the migrated in-memory configuration and the unchanged provider document causes the save to be retried during the next flow synchronization. If inheritance fails, the provider document is not saved and the `UPDATE_FAILED` state forces inheritance to be retried before a later save. Complete active and working configuration replacements are atomic. Secret and Asset references are resolved before either live configuration map is modified. A resolution failure therefore leaves the prior configuration intact, and absent steps are removed in the same replacement operation. Property rename collisions preserve a value already stored under the new property name while removing the obsolete key. This behavior also preserves explicitly present null values and is consistent between the framework and connector mock implementations. ## Additional Behavior - Null persisted configuration lists are treated as empty. - Bundle reload applies migration in memory without writing to the provider. - Migration failures are isolated to the affected Connector and do not abort synchronization of other Connectors. - Callback failures notify all applicable working configuration steps and prevent provider persistence until inheritance succeeds. - Optional property defaults are not inserted because doing so could incorrectly satisfy property dependency conditions. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - Pull Request based on current revision of the `main` branch - Pull Request refers to a feature branch with one commit containing changes # Verification The following focused verification was completed: - `./mvnw -pl nifi-framework-bundle/nifi-framework/nifi-framework-core -Dtest=TestStandardConnectorNode,TestStandardConnectorRepository test` - 195 tests passed - `./mvnw -pl nifi-framework-bundle/nifi-framework/nifi-framework-components -Dtest=TestConnectorMigrateProperties test` - 9 tests passed - `./mvnw -pl nifi-connector-mock-bundle/nifi-connector-mock -Dtest=TestMockConnectorPropertyConfiguration test` - 9 tests passed - Checkstyle and PMD passed for: - `nifi-framework-core` - `nifi-framework-core-api` - `nifi-framework-components` - `nifi-connector-mock` - `git diff --check` completed successfully. ### Build - [ ] Build completed using `./mvnw clean install -P contrib-check` - [ ] JDK 21 - [ ] JDK 25 ### Licensing No dependencies were added or changed. - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation No user-facing documentation was changed. - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
