markap14 commented on code in PR #11872:
URL: https://github.com/apache/nifi/pull/11872#discussion_r4592090113


##########
nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/FlowEvolutionStatus.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.connector;
+
+import java.util.Collection;
+
+/**
+ * Reports local progress through the evolve-drain phase of a pending 
Connector flow evolution. Applying the upgraded flow
+ * after the evolve-drain phase is tracked through {@link ConnectorState}, as 
for a configuration update.
+ */
+public enum FlowEvolutionStatus {
+    /**
+     * No evolution is pending, so the node no longer needs the outgoing flow.
+     */
+    INACTIVE,
+
+    /**
+     * The node still has the outgoing flow and has not finished draining it.
+     */
+    EVOLVE_DRAIN_PENDING,
+
+    /**
+     * The node has drained the outgoing flow and has not yet applied the 
upgraded flow.
+     */
+    EVOLVE_DRAIN_COMPLETE,

Review Comment:
   I presume the intent is for the lifecycle to go from EVOLVE_DRAIN_PENDING -> 
EVOLVE_DRAIN_COMPLETE -> INACTIVE ? It's not really clear. From the comments 
here and looking at the enum it seems like EVOLVE_DRAIN_COMPLETE should be 
followed by EVOLUTION_COMPLETE or something but there is no such thing. Can we 
clarify the expected lifecycle in the javadocs?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -1041,6 +1097,19 @@ public Future<Void> stop(final FlowEngine scheduler) {
         return stopCompleteFuture;
     }
 
+    private boolean prepareFlowEvolutionStop(final CompletableFuture<Void> 
stopCompleteFuture) {

Review Comment:
   This method is confusing. It's a `prepare` method. But it's unclear what the 
return value is really telling us. And I would not expect a method named 
`prepareFlowEvolutionStop` to then call `abortFlowEvolution` - i would expect 
it only to do prep work



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/groups/ProcessGroup.java:
##########
@@ -976,6 +976,16 @@ default CompletableFuture<Void> stopComponents() {
      */
     void restoreFlowPreservingIdentifiers(VersionedExternalFlow 
proposedSnapshot);
 
+    /**
+     * Restores a previously persisted flow while controlling whether 
components resume their captured scheduled state.
+     * When {@code resumeComponentState} is false, the flow structure is 
restored so queued FlowFiles can reattach, but
+     * no component is started or enabled.
+     *
+     * @param proposedSnapshot the proposed flow whose Instance Identifiers 
should be used as runtime identifiers
+     * @param resumeComponentState whether components should resume the 
running or enabled state captured in the snapshot
+     */
+    void restoreFlowPreservingIdentifiers(VersionedExternalFlow 
proposedSnapshot, boolean resumeComponentState);

Review Comment:
   This is confusing to me. The method is named as if it will restore the flow 
without change IDs (which would be presumed). But the javadocs makes it sound 
like the intent is to restore the flow while optionally resuming the 
components' states. I guess this is intended just as an override to the 
`restoreFlowPreservingIdentifiers` method that takes just the 
VersionedExternalFlow argument, but it's confusing here, given the name and the 
JavaDocs. I also wonder if we need the override, rather than just adding an 
argument to the existing method.
   
   It's also worth noting that we tend to prefer enums on public methods and 
especially interfaces, rather than `boolean`s.
   
   My recommendation, I think, would be to just add a new enum argument to the 
existing method and eliminate this one. Something like:
   ```
   void restoreFlowPreservingIdentifiers(VerisonedExternalFlow 
proposedSnapshot, ResumeComponentState resumeStates)
   ```
   Where `ResumeComponentState` has two values, `RESUME_STATES`, 
`LEAVE_STOPPED` or `RESUME_STATES`, `LEAVE_UNCHANGED` or whatever. This helps 
to make it very clear when passing the argument exactly what your intent is.



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -766,6 +815,11 @@ private void releaseWorkingFlowContext(final 
WorkingFlowContextState workingCont
 
     @Override
     public void abortUpdate(final Throwable cause) {
+        if (isFlowEvolutionInProgress() || hasFlowEvolutionFailed()) {
+            failFlowEvolution(cause == null ? null : cause.getMessage(), 
cause);
+            return;
+        }
+
         stateTransition.setCurrentState(ConnectorState.UPDATE_FAILED);

Review Comment:
   [Claude Opus 5.5] Moving to UPDATE_FAILED here leaves any pending stop 
futures incomplete forever. `StandardConnectorStateTransition` completes stop 
futures only when the state becomes STOPPED. The latest push fixes the cancel 
race that led here, but there's still at least one path that reaches this point 
with a stop pending.
   
   A STOP that arrives while the upgraded flow is being applied 
(PREPARING_FOR_UPDATE, UPDATING, or UPDATED) gets here. The server [accepts 
STOP from any 
state](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L3812),
 even though the UI doesn't offer it in those states.
   1. `stop()` sets STOPPING.
   2. The coordinator's `prepareForUpdate` or `waitForState` then fails, and 
this method sets UPDATE_FAILED.
   3. If that happens before `stopComponent` runs, `stopComponent` [returns 
early](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1279-L1281)
 without completing its future.
   
   Separately, `Connector.stop` and `Connector.applyUpdate` can end up running 
at the same time on different threads.
   
   Anything that waits on a stop then blocks forever. [Orphan Connector 
removal](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/serialization/VersionedFlowSynchronizer.java#L1117)
 waits with no limit while a reconnecting node holds the flow service write 
lock. `restartConnector` waits with no limit too, and so would any future 
caller.
   
   Rather than fix each race on its own, I'd make 
`StandardConnectorStateTransition` guarantee that every stop future completes:
   - When the state moves to anything other than STOPPING or STOPPED, complete 
the pending stop futures *exceptionally*, with a message naming the new state.
   - Have `stopComponent` complete its own future the same way before it 
returns early.
   
   Completing them exceptionally matters, because callers like orphan removal 
go on to purge, which requires STOPPED. The normal configuration-update path 
has the same gap today, so this fixes both.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -1619,6 +1775,691 @@ public boolean isExtensionMissing() {
         return extensionMissing;
     }
 
+    @Override
+    public boolean isPendingFlowEvolution() {
+        synchronized (flowEvolutionLock) {
+            if (extensionMissing) {
+                return false;
+            }
+
+            final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot;
+            final Bundle snapshotBundle = snapshot == null ? null : 
snapshot.bundle();
+            return snapshotBundle != null && !sameBundle(snapshotBundle, 
bundleCoordinate);
+        }
+    }
+
+    @Override
+    public FlowEvolutionStatus getReportedFlowEvolutionStatus() {
+        return flowEvolutionProgress.get().reportedStatus();
+    }
+
+    private boolean isEvolveDrainComplete() {
+        return flowEvolutionProgress.get().isDrainComplete();
+    }
+
+    private boolean isApplyingEvolvedFlow() {
+        return getReportedFlowEvolutionStatus() == 
FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE;
+    }
+
+    boolean hasFlowEvolutionFailed() {
+        return flowEvolutionProgress.get().hasFailed();
+    }
+
+    private VersionedProcessGroup getAuthoritativeManagedProcessGroup() {
+        final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot;
+        return snapshot == null ? null : snapshot.flow();
+    }
+
+    @Override
+    public ConnectorFlowSnapshot getAuthoritativeFlowSnapshot() {
+        return authoritativeFlowSnapshot;
+    }
+
+    @Override
+    public boolean restorePendingFlowEvolution(final VersionedConnector 
persisted, final List<VersionedConfigurationStep> activeConfiguration,
+            final List<VersionedConfigurationStep> workingConfiguration) 
throws FlowUpdateException {
+        final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted);
+        final VersionedProcessGroup persistedManagedGroup = 
persisted.getManagedProcessGroup();
+        final Map<String, SecretReference> parameterSecretReferences = 
ParameterSecretReferences.convertManagedParameterSecretReferences(persisted);
+        // Components in the persisted group may name NAR versions that are no 
longer installed. A missing coordinate
+        // is rewritten only when exactly one compatible bundle is installed. 
A ghost has no initialization context,
+        // and its coordinates have to stay as persisted.
+        if (!extensionMissing) {
+            resolvePersistedBundles(persistedManagedGroup);
+        }
+
+        if (extensionMissing && persistedManagedGroup != null) {
+            // Adopt before inheritConfiguration. A flow save with no snapshot 
omits the managed group, and inherit
+            // rebuilds the live group before the persisted one is installed 
below.
+            adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+            return false;
+        }
+
+        if (!extensionMissing && !sameManagedBundle(managedFlowBundle) && 
persistedManagedGroup != null) {
+            // Skip inheritConfiguration. Its applyUpdate would replace the 
pre-upgrade group with the new bundle's
+            // flow. The restored group keeps the previous bundle, so the next 
start evolves it.
+            restorePersistedFlow(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, true);

Review Comment:
   [Claude Opus 5.5] A failure here can permanently lose the pre-upgrade flow 
and the Connector's configuration. Here's how:
   
   1. At startup, `restorePersistedFlow` resolves secrets before it touches the 
flow. If the secrets provider can't be reached, the parameter provider throws 
an `IllegalStateException` 
([StandardParameterProviderNode#L295](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/parameter/StandardParameterProviderNode.java#L295)),
 and nothing catches it before `syncConnector`.
   2. Because the restore comes first, 
`loadConfigurationsWithoutRebuildingFlow` and `adoptAuthoritativeSnapshot` 
never run. The Connector holds no snapshot and no configuration, so it isn't 
even treated as pending.
   3. `syncConnector` calls `markInvalid`, but the next validation clears that.
   4. On the next flow save, from any change anywhere, `mapConnector` writes no 
managed group, parameter context, or managed-flow bundle, because there's no 
snapshot 
([#L1151](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1151)).
 It also writes empty active and working configurations, taken from the live 
Connector 
([#L1132](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1132),
 
[#L1138](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/mapping/VersionedComponentFlowMapper.java#L1138)).
   5. After that, a restart with the vault back up finds nothing to restore. 
The queued FlowFiles stay in the repository as orphans with no connection to 
rejoin.
   
   The extension-missing path in `restorePersistedManagedFlow` already handles 
this case 
([#L1866-L1873](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1866-L1873)).
 I'd do the same here:
   
   1. Load the configurations before restoring the flow. Loading doesn't depend 
on the managed flow, and doing it first means the snapshot below records the 
real active configuration, since `adoptAuthoritativeSnapshot` copies it from 
the live Connector 
([#L1966](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1966)).
   2. Catch a failed restore and call `markManagedFlowRestoreFailed()`. 
`verifyCanStart` and the START action already block on that, with a message 
telling the user to restart.
   3. Record the persisted snapshot anyway, and return `true` so that 
`syncConnector` doesn't rebuild the flow from the new bundle.
   
   It's also worth considering whether an unreachable secrets provider should 
count as an unresolved secret. The code already handles that case by leaving 
components stopped.
   
   For a test, an existing restore test in `TestStandardConnectorNode` could 
add a secrets manager that throws. It should check that START is refused and 
that the mapped Connector still has its managed group and active configuration.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java:
##########
@@ -586,10 +618,155 @@ private void syncFromProviderForRead(final ConnectorNode 
connector) {
     }
 
     @Override
-    public Future<Void> startConnector(final ConnectorNode connector) {
+    public Future<Void> startConnector(final ConnectorNode connector, final 
ConnectorUpdateContext persistenceContext) {
+        Objects.requireNonNull(persistenceContext, "Flow persistence context 
is required");
+
+        final String connectorId = connector.getIdentifier();
+        final CompletableFuture<Void> runningEvolution = 
evolveThenStartFuturesByConnectorId.get(connectorId);
+        if (runningEvolution != null) {
+            return runningEvolution;
+        }
+        if (!connector.isPendingFlowEvolution()) {
+            return connector.start(lifecycleExecutor);
+        }
+
+        final CompletableFuture<Void> evolutionFuture = new 
CompletableFuture<>();
+        final CompletableFuture<Void> existingEvolution = 
evolveThenStartFuturesByConnectorId.putIfAbsent(connectorId, evolutionFuture);
+        if (existingEvolution != null) {
+            return existingEvolution;
+        }
+
+        try {
+            connector.prepareFlowEvolution();

Review Comment:
   [Claude Opus 5.5] START on a pending Connector fails silently when the 
Connector isn't STOPPED or UPDATE_FAILED:
   - `verifyCanStart` doesn't check the state, so the replicated START passes 
verification on every node.
   - `prepareFlowEvolution` then throws on a node that is STOPPING, PURGING, or 
running a user-requested drain. A user drain is allowed while an upgrade is 
pending, and STOPPING happens, for example, while a node is still retrying the 
stop of the pre-upgrade flow.
   - The catch below puts the error on the future without logging it, and 
`StandardConnectorDAO.startConnector` ignores the future, so the user gets a 
success response.
   - In a cluster, that node keeps reporting `EVOLVE_DRAIN_PENDING`, so every 
node where START worked waits on it in `waitForFlowEvolutionStatus` with no 
limit.
   
   I'd reject this START during verification when an upgrade is pending and the 
state isn't STOPPED or UPDATE_FAILED, and log the failure here as well.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/components/connector/ConnectorClusterUnavailableException.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.components.connector;
+
+/**
+ * Thrown when a clustered Connector request cannot be completed because no 
node is connected yet.
+ */
+public class ConnectorClusterUnavailableException extends RuntimeException {

Review Comment:
   [Claude Opus 5.5] Minor: the replicator methods already declare 
`IOException`. Extending `IOException` here would make this a checked 
exception, which matches how we usually signal an expected condition.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -3051,6 +3909,57 @@ private List<DescribedValue> fetchAllowableValues(final 
String stepName, final S
         return allowableValues;
     }
 
+    /**
+     * Progress through a pending flow evolution. The drain status is kept 
when the evolution fails, so a retry knows whether
+     * this node has already drained the outgoing flow.
+     *
+     * @param drainStatus {@link FlowEvolutionStatus#INACTIVE} when no 
evolution is pending, otherwise this node's evolve-drain progress
+     * @param failureExplanation why the evolution failed, or null when it has 
not failed
+     */
+    private record FlowEvolutionProgress(FlowEvolutionStatus drainStatus, 
String failureExplanation) {
+        private static final FlowEvolutionProgress NONE = new 
FlowEvolutionProgress(FlowEvolutionStatus.INACTIVE, null);
+
+        FlowEvolutionStatus reportedStatus() {
+            return hasFailed() ? FlowEvolutionStatus.FAILED : drainStatus;
+        }
+
+        boolean isDrainComplete() {
+            return drainStatus == FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE;
+        }
+
+        boolean hasFailed() {
+            return failureExplanation != null;
+        }
+
+        FlowEvolutionProgress withDrainStatus(final FlowEvolutionStatus 
status) {
+            return new FlowEvolutionProgress(status, failureExplanation);
+        }
+
+        FlowEvolutionProgress replaceDrainStatus(final FlowEvolutionStatus 
expected, final FlowEvolutionStatus replacement) {
+            return drainStatus == expected ? withDrainStatus(replacement) : 
this;
+        }
+
+        FlowEvolutionProgress withFailureIfAbsent(final String explanation) {
+            return hasFailed() ? this : new FlowEvolutionProgress(drainStatus, 
explanation);
+        }
+    }
+
+    /**
+     * One attempt at a pending flow evolution. The futures are each set at 
most once per attempt, and are read and written
+     * only while holding the evolution lock, so closing the attempt cancels 
this attempt's work and never a newer attempt's.
+     */
+    private static final class FlowEvolutionAttempt {

Review Comment:
   [Claude Opus 5.5] Not a blocker, but worth considering: this PR adds a 
little over 900 lines to `StandardConnectorNode`, which is now about 4,000 
lines. Most of the new code is flow evolution:
   - this class and `FlowEvolutionProgress`;
   - `flowEvolutionLock`;
   - the prepare, evolve-drain, begin-update, fail, and retry-stop methods;
   - the snapshot handling.
   
   Pulling that into its own class, owned by the node and called by it, would 
put all of the evolution state in one place. That would make it much easier to 
see which thread changes what, and to unit test the evolution states directly. 
The node would keep its lifecycle methods and delegate the evolution parts.
   
   Curious whether you think that's worthwhile.
   



##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorChangeVersionDrainIT.java:
##########
@@ -0,0 +1,481 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.tests.system.connectors;
+
+import org.apache.nifi.components.connector.ConnectorState;
+import org.apache.nifi.components.connector.FlowEvolutionStatus;
+import org.apache.nifi.flow.VersionedConnector;
+import org.apache.nifi.flow.VersionedConnectorState;
+import org.apache.nifi.flow.VersionedConnectorValueReference;
+import org.apache.nifi.flow.VersionedParameter;
+import org.apache.nifi.security.encryption.PropertyEncryptionEncoder;
+import org.apache.nifi.tests.system.FlowDefinitionFiles;
+import org.apache.nifi.toolkit.client.NiFiClientException;
+import org.apache.nifi.web.api.dto.ConnectorValueReferenceDTO;
+import org.apache.nifi.web.api.dto.StateEntryDTO;
+import org.apache.nifi.web.api.entity.ComponentStateEntity;
+import org.apache.nifi.web.api.entity.ConnectionEntity;
+import org.apache.nifi.web.api.entity.ConnectorEntity;
+import org.apache.nifi.web.api.entity.ParameterProviderEntity;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Single-node upgrade-drain scenarios for a version change.
+ */
+public class ConnectorChangeVersionDrainIT extends 
AbstractChangeVersionDrainIT {
+
+    private static final String UPGRADE_SECRET = "Upgrade Secret";
+    private static final String PROVIDER_NAME = "PropertiesParameterProvider";
+    private static final String SECRET_NAME = "upgrade-secret";
+    private static final String SECRET_FULLY_QUALIFIED_NAME = PROVIDER_NAME + 
".Parameters." + SECRET_NAME;
+    private static final String GENERATOR = "GenerateFlowFile";
+    private static final String CANCELLED_EVOLUTION = "Flow evolution was 
cancelled";
+    private static final String REWIRED_ENDPOINT = "Rewired Endpoint";
+
+    @Test
+    @Timeout(360)
+    public void testPreUpgradeParameterSecretIsRefreshedAndCanBeRestored() 
throws Exception {
+        final SecretConnector secretConnector = 
createSecretConnector("original-secret");
+        changeVersion(secretConnector.connectorId(), DRAIN_V2);
+        
getClientUtil().updateParameterProviderProperties(secretConnector.parameterProvider(),
 Map.of("parameters", SECRET_NAME + "=rotated-secret"));
+
+        getNiFiInstance().stop();

Review Comment:
   [Claude Opus 5.5] Only four tests in this class actually check behavior 
across a restart:
   - `testPreUpgradeParameterSecretIsRefreshedAndCanBeRestored`
   - `testPendingUpgradeEditsThenStartsWithoutReEvolving`, which restarts twice
   - `testRestartDuringDrainingComesBackPending`
   - `testPendingUpgradeBlocksEndingTroubleshootingWithQueuedFlowFiles`
   
   Could these become one test with a single restart? Before restarting, set up 
one connector per scenario:
   - one with a pending upgrade, working-configuration edits and a secret;
   - one in troubleshooting with an upgrade pending;
   - one draining, with its own `Gate File Path` so it can be held and released 
separately from the others.
   
   Then restart once and check each connector. The second restart in 
`testPendingUpgradeEditsThenStartsWithoutReEvolving` checks the saved flow 
after the upgrade finishes. It can become a read of the flow file with NiFi 
still running.
   
   Together with the change I suggested for `stopAndAssertNotPending`, this 
takes the class from 14 restarts to 1.



##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/AbstractConnectorUpgradeDrainIT.java:
##########
@@ -0,0 +1,406 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.tests.system.connectors;
+
+import org.apache.nifi.components.connector.ConnectorState;
+import org.apache.nifi.flow.Bundle;
+import org.apache.nifi.flow.VersionedConfigurationStep;
+import org.apache.nifi.flow.VersionedConnector;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.nar.NarState;
+import org.apache.nifi.tests.system.AbstractNarSwapMigrationIT;
+import org.apache.nifi.tests.system.FlowDefinitionFiles;
+import org.apache.nifi.tests.system.NiFiInstance;
+import org.apache.nifi.tests.system.nar.NarUploadUtil;
+import org.apache.nifi.toolkit.client.ConnectorClient;
+import org.apache.nifi.toolkit.client.NiFiClientException;
+import org.apache.nifi.web.api.dto.BundleDTO;
+import org.apache.nifi.web.api.dto.ConnectorDTO;
+import org.apache.nifi.web.api.dto.NarSummaryDTO;
+import org.apache.nifi.web.api.entity.ConnectionEntity;
+import org.apache.nifi.web.api.entity.ConnectorEntity;
+import org.apache.nifi.web.api.entity.ProcessGroupFlowEntity;
+import org.apache.nifi.web.api.entity.ProcessorEntity;
+import org.junit.jupiter.api.BeforeEach;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Upgrade-drain scenarios that hold for every way a Connector can move to a 
new version. Each subclass supplies the
+ * trigger: a NAR swap followed by a restart, or a version change while the 
Connector is stopped.
+ */
+public abstract class AbstractConnectorUpgradeDrainIT extends 
AbstractNarSwapMigrationIT {
+
+    private static final String GATE_FILE = "upgrade-drain-gate.txt";
+    static final String PREPARATION_FAILED = "flow evolution preparation 
failed";
+    static final String REMOVED_CONNECTION = "Removed Connection";
+    static final String REMOVED_PROCESSOR_CHAIN = "Removed Processor Chain";
+    static final String RELAY_PROCESSOR = "RelayFlowFile";
+    static final String REMOVED_PROCESSOR = "RemovedFlowFile";
+    static final String SWAPPED_PROCESSOR = "SwappedFlowFile";
+
+    static final String DRAIN_CONNECTOR_TYPE = 
"org.apache.nifi.connectors.tests.system.UpgradeDrainConnector";
+    static final String DRAIN_BUNDLE_GROUP = "org.apache.nifi";
+    static final String DRAIN_BUNDLE_ARTIFACT = 
"nifi-connector-upgrade-drain-nar";
+    static final String DRAIN_V1 = "1.0.0";
+    static final String DRAIN_V2 = "2.0.0";
+    private static final String DRAIN_NARS_LOCATION = 
"target/nifi-connector-upgrade-drain-nars";
+    private static final Map<String, String> DRAIN_NAR_IDS = Map.of(
+        DRAIN_V1, "nifi-connector-upgrade-drain-v1-nar",
+        DRAIN_V2, "nifi-connector-upgrade-drain-v2-nar");
+
+    /**
+     * Makes the version 2 UpgradeDrainConnector misbehave. The Connector 
reads the named file from its node's working
+     * directory, and the file contents select the behavior. Faults that share 
a file replace each other.
+     */
+    protected enum UpgradeDrainFault {
+        
PREPARATION_RETURNS_NULL_FUTURE("upgrade-drain-preparation-result.txt", 
"NULL_FUTURE"),
+        PREPARATION_FAILS("upgrade-drain-preparation-result.txt", 
"FAILED_FUTURE"),
+        
PREPARATION_UPDATES_PRE_UPGRADE_FLOW("upgrade-drain-preparation-result.txt", 
"UPDATE_PRE_UPGRADE_FLOW"),
+        ACTIVE_FLOW_RETURNS_NULL("upgrade-drain-active-flow.txt", "NULL"),
+        ACTIVE_FLOW_THROWS("upgrade-drain-active-flow.txt", "THROW"),
+        APPLY_FAILS("upgrade-drain-apply-fail.txt", "");
+
+        private final String fileName;
+        private final String contents;
+
+        UpgradeDrainFault(final String fileName, final String contents) {
+            this.fileName = fileName;
+            this.contents = contents;
+        }
+    }
+
+    /**
+     * Identifies a Connector bundle in assertions. A null version matches any 
version of the artifact.
+     */
+    protected record ExpectedBundle(String artifact, String version) {
+
+        String recordPrefix() {
+            return "org.apache.nifi:" + artifact + ":" + (version == null ? "" 
: version);
+        }
+
+        void assertMatches(final Bundle bundle, final String description) {
+            assertNotNull(bundle, description);
+            assertTrue(bundle.getArtifact().contains(artifact), description + 
" " + bundle.getArtifact());
+            if (version != null) {
+                assertEquals(version, bundle.getVersion(), description);
+            }
+        }
+    }
+
+    /**
+     * Creates the Connector on its outgoing version and applies the given 
configuration.
+     */
+    protected abstract String createOutgoingConnector(Map<String, String> 
properties) throws Exception;
+
+    /**
+     * Moves the stopped Connector to its incoming version, leaving it STOPPED 
with the upgrade pending.
+     */
+    protected abstract void triggerUpgrade(String connectorId) throws 
Exception;
+
+    protected abstract ExpectedBundle outgoingBundle();
+
+    protected abstract ExpectedBundle incomingBundle();
+
+    @BeforeEach
+    public void clearFaults() {
+        for (final UpgradeDrainFault fault : UpgradeDrainFault.values()) {
+            clearFault(fault);
+        }
+    }
+
+    protected void injectFault(final UpgradeDrainFault fault) throws 
IOException {
+        for (int nodeIndex = 1; nodeIndex <= nodeCount(); nodeIndex++) {
+            injectFault(nodeIndex, fault);
+        }
+    }
+
+    protected void injectFault(final int nodeIndex, final UpgradeDrainFault 
fault) throws IOException {
+        writeToNodeDirectory(nodeIndex, fault.fileName, fault.contents);
+    }
+
+    protected void clearFault(final UpgradeDrainFault fault) {
+        deleteFromNodeDirectories(fault.fileName);
+    }
+
+    protected String createOutgoingConnector(final String mode, final boolean 
removeConnection, final String comment) throws Exception {
+        return createOutgoingConnector(properties(mode, removeConnection, 
comment));
+    }
+
+    protected String queueStoppedUpgrade(final String mode, final boolean 
removeConnection, final String comment) throws Exception {
+        return queueAtLeastOneFlowFileAndStop(createOutgoingConnector(mode, 
removeConnection, comment));
+    }
+
+    protected String queueAtLeastOneFlowFileAndStop(final String connectorId) 
throws Exception {
+        holdFlowFiles();
+        getClientUtil().startConnector(connectorId);
+        waitForConnectorMinQueueCount(connectorId, 1);
+        getClientUtil().stopConnector(connectorId);
+        return connectorId;
+    }
+
+    protected void restartKeepingIncomingVersion() {
+        getNiFiInstance().stop();
+        getNiFiInstance().start(true);
+        waitForCluster();
+    }
+
+    protected void uploadDrainNars(final String... versions) throws Exception {
+        final File narsLocation = new File(DRAIN_NARS_LOCATION);
+        if (!narsLocation.exists()) {
+            throw new IllegalStateException("NARs location does not exist at: 
" + narsLocation.getAbsolutePath());
+        }
+
+        final NarUploadUtil narUploadUtil = new NarUploadUtil(getNifiClient());
+        for (final String version : versions) {
+            final NarSummaryDTO nar = narUploadUtil.uploadNar(narsLocation, 
DRAIN_NAR_IDS.get(version));
+            
waitFor(narUploadUtil.getWaitForNarStateSupplier(nar.getIdentifier(), 
NarState.INSTALLED));
+        }
+    }
+
+    protected String createDrainConnector(final String version, final 
Map<String, String> properties) throws NiFiClientException, IOException, 
InterruptedException {
+        final ConnectorEntity connector = 
getClientUtil().createConnector(DRAIN_CONNECTOR_TYPE, DRAIN_BUNDLE_GROUP, 
DRAIN_BUNDLE_ARTIFACT, version);
+        getClientUtil().configureConnector(connector, "Upgrade", properties);
+        getClientUtil().applyConnectorUpdate(connector);
+        getClientUtil().waitForValidConnector(connector.getId());
+        return connector.getId();
+    }
+
+    /**
+     * Sends the version change the way the UI does: a PUT carrying only the 
new bundle.
+     */
+    protected ConnectorEntity changeVersion(final String connectorId, final 
String version) throws NiFiClientException, IOException {
+        final ConnectorClient connectorClient = 
getNifiClient().getConnectorClient();
+        final ConnectorEntity current = 
connectorClient.getConnector(connectorId);
+        final ConnectorDTO component = new ConnectorDTO();
+        component.setId(connectorId);
+        component.setBundle(new BundleDTO(DRAIN_BUNDLE_GROUP, 
DRAIN_BUNDLE_ARTIFACT, version));
+
+        final ConnectorEntity update = new ConnectorEntity();
+        update.setId(connectorId);
+        update.setRevision(current.getRevision());
+        update.setComponent(component);
+        return connectorClient.updateConnector(update);
+    }
+
+    /**
+     * Stops the generator and lets the terminator drop queued FlowFiles on 
every node.
+     */
+    protected void releaseFlowFiles() throws IOException {
+        for (int nodeIndex = 1; nodeIndex <= nodeCount(); nodeIndex++) {
+            releaseFlowFiles(nodeIndex);
+        }
+    }
+
+    /**
+     * Stops the generator and lets the terminator drop queued FlowFiles on 
one node.
+     */
+    protected void releaseFlowFiles(final int nodeIndex) throws IOException {
+        writeToNodeDirectory(nodeIndex, GATE_FILE, "");
+    }
+
+    /**
+     * Lets the generator run and makes the terminator hold every FlowFile on 
every node.
+     */
+    protected void holdFlowFiles() {
+        deleteFromNodeDirectories(GATE_FILE);
+    }
+
+    private void writeToNodeDirectory(final int nodeIndex, final String 
fileName, final String contents) throws IOException {
+        Files.writeString(new File(nodeDirectory(nodeIndex), 
fileName).toPath(), contents);
+    }
+
+    private void deleteFromNodeDirectories(final String fileName) {
+        for (final File directory : instanceDirectories()) {
+            final File file = new File(directory, fileName);
+            if (file.exists()) {
+                assertTrue(file.delete(), file.getAbsolutePath());
+            }
+        }
+    }
+
+    protected void stopAndAssertNotPending(final String connectorId) throws 
IOException {

Review Comment:
   [Claude Opus 5.5] The new system tests restart NiFi far too often, and this 
helper is the biggest reason. It stops NiFi only so the test can read 
`flow.json.gz`. In the classes that share one NiFi instance between tests 
(`ConnectorChangeVersionDrainIT` and `ClusteredConnectorChangeVersionDrainIT`), 
the `@AfterEach` in `AbstractNarSwapMigrationIT` then sees that NiFi is not 
running and starts it again. So every test that ends this way pays for a full 
restart, even though it never checks anything across a restart.
   
   Here is the count across the PR:
   - `ConnectorChangeVersionDrainIT`: 11 of its 13 tests restart NiFi, for 14 
restarts in total.
   - `ClusteredConnectorChangeVersionDrainIT`: 1 restart of the 2-node cluster, 
through this helper.
   - `ClusteredConnectorUpgradeDrainIT` and `ConnectorUpgradeDrainProviderIT`: 
each builds a brand-new environment and then restarts it once for the NAR swap.
   - `ConnectorParameterContextIT`: 1 added restart.
   
   About 10 of those restarts happen only to read the flow file at the end of a 
test. The flow is saved shortly after each change, and 
`assertPersistedOutgoing` already reads the file while NiFi is running. Can 
this helper, and the `getNiFiInstance().stop()` calls followed by 
`FlowDefinitionFiles.readConnector` at the end of tests in 
`ConnectorChangeVersionDrainIT`, read the file with NiFi still running? 
Re-reading it inside a `waitFor` until it shows the expected bundle would 
handle the short delay before the save. Where the REST API already shows the 
same thing (for example, a flow evolution status of INACTIVE), checking through 
REST is simpler still.
   
   Overall, I'd like the new tests to get down to two that restart NiFi: one 
test that checks what survives a restart (see my comment in 
`ConnectorChangeVersionDrainIT`), and the clustered NAR-swap startup test in 
`ClusteredConnectorUpgradeDrainIT`.



##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorUpgradeDrainProviderIT.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.tests.system.connectors;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import org.apache.nifi.components.connector.ConnectorState;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.web.api.dto.ConfigurationStepConfigurationDTO;
+import org.apache.nifi.web.api.dto.ConnectorConfigurationDTO;
+import org.apache.nifi.web.api.dto.PropertyGroupConfigurationDTO;
+import org.apache.nifi.web.api.entity.ConnectorEntity;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+/**
+ * A NAR-swap upgrade applies configuration supplied by the external provider.
+ */
+public class ConnectorUpgradeDrainProviderIT extends 
AbstractNarSwapUpgradeDrainIT {
+
+    private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+    private static final String COMMENT = "Comment";
+    private static final String LEGACY_NOTE = "Legacy Note";
+    private static final String DRAIN_LABEL = "Drain Label";
+    private static final String SUPPLIED_COMMENT = "supplied-by-provider";
+
+    private final Path providerState = createProviderState();
+
+    @Override
+    protected Map<String, String> getNifiPropertiesOverrides() {
+        return 
Map.of(NiFiProperties.CONNECTOR_CONFIGURATION_PROVIDER_IMPLEMENTATION, 
"org.apache.nifi.connectors.tests.system.FileConnectorConfigurationProvider", 
NiFiProperties
+            .CONNECTOR_CONFIGURATION_PROVIDER_PROPERTIES_PREFIX + 
"state.directory", providerState.toAbsolutePath().toString());
+    }
+
+    @Test
+    @Timeout(240)
+    public void testProviderSuppliedConfigurationIsAppliedByUpgrade() throws 
Exception {
+        final Map<String, String> initial = properties("DRAIN", true, 
"shared");
+        initial.put(LEGACY_NOTE, "v1-only");
+        final String connectorId = createConnector("UpgradeDrainConnector", 
initial);
+
+        getNiFiInstance().stop();

Review Comment:
   [Claude Opus 5.5] This test needs its own environment because of the 
provider properties, and the NAR swap adds a restart on top of that. Is the NAR 
swap important to what this test checks? If the provider-supplied configuration 
gets applied the same way when the version is changed through the REST API (as 
in `AbstractChangeVersionDrainIT`), this test could use a version change and 
skip the restart. If the NAR-swap startup path is the point of the test, could 
it be folded into `ClusteredConnectorUpgradeDrainIT` instead, so that we have 
one NAR-swap restart test instead of two?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -1619,6 +1775,691 @@ public boolean isExtensionMissing() {
         return extensionMissing;
     }
 
+    @Override
+    public boolean isPendingFlowEvolution() {
+        synchronized (flowEvolutionLock) {
+            if (extensionMissing) {
+                return false;
+            }
+
+            final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot;
+            final Bundle snapshotBundle = snapshot == null ? null : 
snapshot.bundle();
+            return snapshotBundle != null && !sameBundle(snapshotBundle, 
bundleCoordinate);
+        }
+    }
+
+    @Override
+    public FlowEvolutionStatus getReportedFlowEvolutionStatus() {
+        return flowEvolutionProgress.get().reportedStatus();
+    }
+
+    private boolean isEvolveDrainComplete() {
+        return flowEvolutionProgress.get().isDrainComplete();
+    }
+
+    private boolean isApplyingEvolvedFlow() {
+        return getReportedFlowEvolutionStatus() == 
FlowEvolutionStatus.EVOLVE_DRAIN_COMPLETE;
+    }
+
+    boolean hasFlowEvolutionFailed() {
+        return flowEvolutionProgress.get().hasFailed();
+    }
+
+    private VersionedProcessGroup getAuthoritativeManagedProcessGroup() {
+        final ConnectorFlowSnapshot snapshot = authoritativeFlowSnapshot;
+        return snapshot == null ? null : snapshot.flow();
+    }
+
+    @Override
+    public ConnectorFlowSnapshot getAuthoritativeFlowSnapshot() {
+        return authoritativeFlowSnapshot;
+    }
+
+    @Override
+    public boolean restorePendingFlowEvolution(final VersionedConnector 
persisted, final List<VersionedConfigurationStep> activeConfiguration,
+            final List<VersionedConfigurationStep> workingConfiguration) 
throws FlowUpdateException {
+        final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted);
+        final VersionedProcessGroup persistedManagedGroup = 
persisted.getManagedProcessGroup();
+        final Map<String, SecretReference> parameterSecretReferences = 
ParameterSecretReferences.convertManagedParameterSecretReferences(persisted);
+        // Components in the persisted group may name NAR versions that are no 
longer installed. A missing coordinate
+        // is rewritten only when exactly one compatible bundle is installed. 
A ghost has no initialization context,
+        // and its coordinates have to stay as persisted.
+        if (!extensionMissing) {
+            resolvePersistedBundles(persistedManagedGroup);
+        }
+
+        if (extensionMissing && persistedManagedGroup != null) {
+            // Adopt before inheritConfiguration. A flow save with no snapshot 
omits the managed group, and inherit
+            // rebuilds the live group before the persisted one is installed 
below.
+            adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+            return false;
+        }
+
+        if (!extensionMissing && !sameManagedBundle(managedFlowBundle) && 
persistedManagedGroup != null) {
+            // Skip inheritConfiguration. Its applyUpdate would replace the 
pre-upgrade group with the new bundle's
+            // flow. The restored group keeps the previous bundle, so the next 
start evolves it.
+            restorePersistedFlow(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, true);
+            loadConfigurationsWithoutRebuildingFlow(activeConfiguration, 
workingConfiguration);
+            adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+            return true;
+        }
+
+        return false;
+    }
+
+    @Override
+    public void restorePersistedManagedFlow(final VersionedConnector 
persisted) {
+        if (!extensionMissing) {
+            throw new IllegalStateException("Cannot restore the persisted 
managed flow for " + this + " because its extension is not missing");
+        }
+
+        final VersionedProcessGroup persistedManagedGroup = 
persisted.getManagedProcessGroup();
+        if (persistedManagedGroup == null) {
+            return;
+        }
+
+        final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted);
+        final Map<String, SecretReference> parameterSecretReferences = 
ParameterSecretReferences.convertManagedParameterSecretReferences(persisted);
+
+        // Install the persisted flow while ghosted so queues exist before the 
FlowFile repository recovers.
+        // Components stay stopped: a ghost cannot start, and resuming would 
run them under a missing extension.
+        try {
+            restorePersistedFlow(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, false);
+        } catch (final Exception e) {
+            logger.error("{} failed to restore the persisted managed flow 
while ghosted; "
+                    + "queued FlowFiles may not rejoin until NiFi restarts 
after the Connector becomes loadable again", this, e);
+            markManagedFlowRestoreFailed();
+        }
+        adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+    }
+
+    @Override
+    public void restoreTroubleshootingManagedFlow(final VersionedConnector 
persisted) throws FlowUpdateException {
+        final Bundle managedFlowBundle = resolveManagedFlowBundle(persisted);
+        final VersionedProcessGroup persistedManagedGroup = 
persisted.getManagedProcessGroup();
+        final Map<String, SecretReference> parameterSecretReferences = 
ParameterSecretReferences.convertManagedParameterSecretReferences(persisted);
+
+        if (extensionMissing && persistedManagedGroup != null) {
+            adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+            restoreTroubleshootingFlow(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences);
+            return;
+        }
+
+        if (sameManagedBundle(managedFlowBundle) && persistedManagedGroup != 
null) {
+            try {
+                restorePersistedFlow(persistedManagedGroup, null, Map.of(), 
true);
+                adoptAuthoritativeSnapshot(persistedManagedGroup, 
persisted.getManagedParameterContext(), parameterSecretReferences, 
managedFlowBundle);
+            } catch (final Exception e) {
+                logger.error("{} failed to restore the managed flow while 
restoring Troubleshooting", this, e);
+                throw new FlowUpdateException("Failed to restore Managed 
Process Group from persisted flow: " + e.getMessage(), e);
+            }
+        }
+    }
+
+    @Override
+    public void ensureAuthoritativeFlowSnapshot() {
+        if (getAuthoritativeManagedProcessGroup() != null || extensionMissing) 
{
+            return;
+        }
+
+        // Without a snapshot, a later NAR upgrade has no pre-upgrade flow to 
evolve.
+        verifyCanCaptureAuthoritativeSnapshot();
+        final ConnectorFlowSnapshot snapshot = createAuthoritativeSnapshot();
+
+        synchronized (flowEvolutionLock) {
+            // Another thread may have installed a snapshot, such as the 
pre-upgrade flow, while the live flow was being captured
+            if (getAuthoritativeManagedProcessGroup() != null) {
+                return;
+            }
+
+            verifyCanCaptureAuthoritativeSnapshot();
+            installAuthoritativeSnapshot(snapshot);
+        }
+    }
+
+    private void restoreTroubleshootingFlow(final VersionedProcessGroup 
persistedManagedGroup, final VersionedParameterContext parameterContext,
+            final Map<String, SecretReference> parameterSecretReferences) 
throws FlowUpdateException {
+        logger.info("{} was persisted in Troubleshooting mode; restoring 
Managed Process Group from persisted snapshot", this);
+        try {
+            restorePersistedFlow(persistedManagedGroup, parameterContext, 
parameterSecretReferences, false);
+        } catch (final Exception e) {
+            logger.error("{} failed to restore the managed flow while 
restoring Troubleshooting", this, e);
+            throw new FlowUpdateException("Failed to restore Managed Process 
Group from persisted flow: " + e.getMessage(), e);
+        }
+    }
+
+    /**
+     * A group saved before managedFlowBundle existed is attributed to the 
connector bundle stored in the flow definition.
+     */
+    private static Bundle resolveManagedFlowBundle(final VersionedConnector 
versionedConnector) {
+        if (versionedConnector.getManagedFlowBundle() != null) {
+            return versionedConnector.getManagedFlowBundle();
+        }
+        if (versionedConnector.getManagedProcessGroup() != null) {
+            return versionedConnector.getBundle();
+        }
+        return null;
+    }
+
+    private boolean sameManagedBundle(final Bundle managedFlowBundle) {
+        if (managedFlowBundle == null || bundleCoordinate == null) {
+            return true;
+        }
+        return sameBundle(managedFlowBundle, bundleCoordinate);
+    }
+
+    /**
+     * Rewrites processor and controller service bundles in a persisted group 
to the only installed alternative when
+     * the exact coordinate is missing. A ghost Connector keeps the snapshot 
exactly as persisted.
+     */
+    void resolvePersistedBundles(final VersionedProcessGroup group) {
+        if (extensionMissing || group == null || initializationContext == 
null) {
+            return;
+        }
+
+        initializationContext.resolveBundles(group, 
BundleCompatibility.RESOLVE_BUNDLE);
+    }
+
+    void adoptAuthoritativeSnapshot(final VersionedProcessGroup flow, final 
VersionedParameterContext parameterContext,
+            final Map<String, SecretReference> parameterSecretReferences, 
final Bundle managedFlowBundle) {
+        installAuthoritativeSnapshot(new ConnectorFlowSnapshot(flow, 
parameterContext, parameterSecretReferences, managedFlowBundle,
+                
activeFlowContext.getConfigurationContext().toConnectorConfiguration()));
+    }
+
+    void restorePersistedFlow(final VersionedProcessGroup flow, final 
VersionedParameterContext parameterContext,
+            final Map<String, SecretReference> parameterSecretReferences, 
final boolean resumeComponents) {
+        final ParameterSecretReferences.Resolution resolution = 
resolvePersistedParameterSecrets(parameterContext, parameterSecretReferences);
+        if (!resolution.undecryptableParameters().isEmpty()) {
+            logger.warn("{} could not decrypt the persisted values of 
sensitive parameters {}; components are left stopped", this, 
resolution.undecryptableParameters());
+        }
+
+        if (resumeComponents && unresolvedParameterSecretReferences.isEmpty() 
&& resolution.undecryptableParameters().isEmpty()) {
+            activeFlowContext.restorePersistedFlow(flow, 
resolution.parameterContext());
+        } else {
+            activeFlowContext.restorePersistedFlow(flow, 
resolution.parameterContext(), false);

Review Comment:
   This feels weird to call the same method but in one place rely on 'default 
values' while in the other passing them explicitly. Probably makes sense to 
extract the `resumeComponents && unresolvedParameterSecretReferences.isEmpty() 
&& resolution.undecryptableParameters().isEmpty()` into a boolean and just pass 
that as the last argument, no?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/flow/synchronization/StandardVersionedComponentSynchronizer.java:
##########
@@ -4112,18 +4112,27 @@ private void verifyCanRemoveMissingComponents(final 
ProcessGroup processGroup, f
             final Map<String, VersionedConnection> 
proposedConnectionsByVersionedId = proposedGroup.getConnections().stream()
                 .collect(Collectors.toMap(VersionedComponent::getIdentifier, 
Function.identity()));
 
-            // match group's current connections to proposed connections to 
determine if they've been removed
+            // A missing connection, or one whose source changed, is removed 
and recreated. Either change drops
+            // queued data, so reject it before synchronization begins. A 
destination change updates the connection in place.
             for (final Connection connection : processGroup.getConnections()) {
                 final String versionedId = 
connection.getVersionedComponentId().orElse(
                     
VersionedComponentFlowMapper.generateVersionedComponentId(connection.getIdentifier()));
                 final VersionedConnection proposedConnection = 
proposedConnectionsByVersionedId.get(versionedId);
+                final FlowFileQueue flowFileQueue = 
connection.getFlowFileQueue();
+                if (flowFileQueue.isEmpty()) {
+                    continue;
+                }
+
                 if (proposedConnection == null) {
-                    // connection doesn't exist in proposed connections, make 
sure it doesn't have any data in it
-                    final FlowFileQueue flowFileQueue = 
connection.getFlowFileQueue();
-                    if (!flowFileQueue.isEmpty()) {
-                        throw new IllegalStateException(processGroup + " 
cannot be updated to the proposed flow because the proposed flow "
-                            + "does not contain a match for " + connection + " 
and the connection currently has data in the queue.");
-                    }
+                    throw new IllegalStateException(processGroup + " cannot be 
updated to the proposed flow because the proposed flow "
+                        + "does not contain a match for " + connection + " and 
the connection currently has data in the queue.");
+                }
+
+                final String existingSourceId = 
connection.getSource().getVersionedComponentId().orElse(null);

Review Comment:
   [Claude Opus 5.5] This check applies to every versioned-flow 
synchronization, including registry version changes, but its four unit tests 
were removed. Could one combined test come back in 
`StandardVersionedComponentSynchronizerTest`? It would cover a source change 
that is rejected while the queue has data and allowed when the queue is empty.
   
   Also, the source comparison repeats the one in `removeMissingConnections`. A 
small shared helper such as `isSourceChanged(Connection, VersionedConnection)` 
would keep the two from drifting apart.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java:
##########
@@ -733,6 +938,74 @@ private void waitForState(final ConnectorNode connector, 
final Set<ConnectorStat
         }
     }
 
+    private void waitForFlowEvolutionStatus(final ConnectorNode connector) 
throws FlowUpdateException, IOException, InterruptedException {

Review Comment:
   [Claude Opus 5.5] This wait still has no time limit. A user can always end 
it with STOP or cancel drain, because the loop checks for a local failure on 
every poll. But it never resolves on its own, and the silent-START case on 
`startConnector` is a concrete way to get here.
   
   The UI way out can also be blocked. STOP isn't offered while DRAINING 
([#L3425-L3443](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L3425-L3443)),
 and cancel drain is rejected if any node is STOPPING 
([#L1209-L1218](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L1209-L1218)).
   
   A limit equal to the drain timeout, measured from when this node's drain 
finishes, would be enough. The admin guide line that describes the wait as 
unlimited would need updating.
   



##########
nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java:
##########
@@ -3870,11 +3892,17 @@ public ConnectorEntity scheduleConnector(final Revision 
revision, final String i
 
         final RevisionUpdate<ConnectorDTO> snapshot = 
revisionManager.updateRevision(claim, user, () -> {
             switch (state) {
-                case RUNNING -> connectorDAO.startConnector(id);
+                case RUNNING -> connectorDAO.startConnector(id, 
controllerFacade::saveImmediate);
                 case STOPPED -> connectorDAO.stopConnector(id);
                 default -> throw new IllegalArgumentException("Unsupported 
scheduled state for Connector: " + state);
             }
-            controllerFacade.save();
+            // Evolution sets the desired state to RUNNING as soon as it is 
claimed so a joining node starts too.
+            // Persisting that state before the hook succeeds makes a failed 
evolution auto-start on restart.
+            // The repository saves once the upgraded flow is applied, and a 
failed evolution puts the desired state back.
+            final ConnectorNode scheduled = connectorDAO.getConnector(id, 
ConnectorSyncMode.LOCAL_ONLY);
+            if (scheduled.getCurrentState() != ConnectorState.DRAINING || 
!scheduled.isPendingFlowEvolution()) {

Review Comment:
   [Claude Opus 5.5] I think automatically resuming the upgrade after a restart 
is the right behavior. Today it only happens sometimes, though:
   - Flow saves record the Connector's desired state, which is RUNNING during 
the drain.
   - This save is skipped, but any other save during the drain writes RUNNING. 
That could be an edit anywhere in the flow, or the pending save that is flushed 
at shutdown.
   - So whether a restart resumes the upgrade depends on unrelated activity.
   
   If resuming is what we want, I'd drop this special case so that the RUNNING 
state is always saved when the upgrade begins. The developer guide sentence 
"The failure is not saved, so after a restart the Connector comes back in the 
state that was last saved" would need to match.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java:
##########
@@ -586,10 +618,155 @@ private void syncFromProviderForRead(final ConnectorNode 
connector) {
     }
 
     @Override
-    public Future<Void> startConnector(final ConnectorNode connector) {
+    public Future<Void> startConnector(final ConnectorNode connector, final 
ConnectorUpdateContext persistenceContext) {
+        Objects.requireNonNull(persistenceContext, "Flow persistence context 
is required");
+
+        final String connectorId = connector.getIdentifier();
+        final CompletableFuture<Void> runningEvolution = 
evolveThenStartFuturesByConnectorId.get(connectorId);
+        if (runningEvolution != null) {
+            return runningEvolution;
+        }
+        if (!connector.isPendingFlowEvolution()) {
+            return connector.start(lifecycleExecutor);
+        }
+
+        final CompletableFuture<Void> evolutionFuture = new 
CompletableFuture<>();
+        final CompletableFuture<Void> existingEvolution = 
evolveThenStartFuturesByConnectorId.putIfAbsent(connectorId, evolutionFuture);
+        if (existingEvolution != null) {
+            return existingEvolution;
+        }
+
+        try {
+            connector.prepareFlowEvolution();
+        } catch (final Throwable failure) {
+            evolutionFuture.completeExceptionally(failure);
+            evolveThenStartFuturesByConnectorId.remove(connectorId, 
evolutionFuture);
+            if (failure instanceof final Error error) {
+                throw error;
+            }
+            return evolutionFuture;
+        }
+
+        try {
+            flowEvolutionCoordinatorExecutor.submit(() -> 
evolveThenStart(connector, evolutionFuture, persistenceContext));
+        } catch (final Throwable failure) {
+            try {
+                connector.abortUpdate(failure);
+            } catch (final Throwable cleanupFailure) {
+                failure.addSuppressed(cleanupFailure);
+            }
+            evolutionFuture.completeExceptionally(failure);
+            evolveThenStartFuturesByConnectorId.remove(connectorId, 
evolutionFuture);
+            if (failure instanceof final Error error) {
+                throw error;
+            }
+        }
+
+        return evolutionFuture;
+    }
+
+    private void evolveThenStart(final ConnectorNode connector, final 
CompletableFuture<Void> evolutionFuture, final ConnectorUpdateContext 
persistenceContext) {
+        boolean startAttempted = false;
+        try {
+            final Future<Void> startFuture = runFlowEvolution(connector, 
persistenceContext);
+            startAttempted = true;
+            // The evolution is complete once its start is issued. A later 
START or STOP then acts on the connector directly.
+            
evolveThenStartFuturesByConnectorId.remove(connector.getIdentifier(), 
evolutionFuture);
+            startFuture.get();
+            evolutionFuture.complete(null);
+        } catch (final Throwable failure) {
+            if (failure instanceof InterruptedException) {
+                Thread.currentThread().interrupt();
+            }
+
+            if (startAttempted) {
+                logger.error("{} was evolved but did not start", connector, 
failure);
+            } else {
+                logger.error("Flow evolution failed for {}", connector, 
failure);
+            }
+
+            handleFlowEvolutionFailure(connector, evolutionFuture, failure, 
startAttempted);
+
+            if (failure instanceof final Error error) {
+                throw error;
+            }
+        } finally {
+            
evolveThenStartFuturesByConnectorId.remove(connector.getIdentifier(), 
evolutionFuture);
+        }
+    }
+
+    /**
+     * Runs the evolve-drain phase of the evolution, then applies the upgraded 
flow as a configuration update.
+     *
+     * @return the future for starting the evolved connector
+     */
+    private Future<Void> runFlowEvolution(final ConnectorNode connector, final 
ConnectorUpdateContext persistenceContext) throws Throwable {
+        awaitFlowEvolutionDrain(connector, 
connector.evolveFlow(flowEvolutionCoordinatorExecutor));
+
+        // No node applies the upgraded flow until every node that still has 
the outgoing flow has drained it.
+        waitForFlowEvolutionStatus(connector);
+
+        // Every node has drained, so as at startup the preparation is local 
and no cluster-wide wait for UPDATING is needed.
+        // A peer that has not yet observed the drain barrier may still be 
DRAINING or preparing.
+        connector.beginEvolvedFlowUpdate();
+        connector.prepareForUpdate();
+        completeUpdate(connector, persistenceContext, 
Set.of(ConnectorState.PREPARING_FOR_UPDATE, ConnectorState.UPDATING, 
ConnectorState.DRAINING));
+
+        // The connector stayed desired RUNNING throughout the evolution, so 
the flow saved after the apply already records it as running.
+        logger.info("{} has been successfully evolved; starting", connector);
         return connector.start(lifecycleExecutor);
     }
 
+    private void awaitFlowEvolutionDrain(final ConnectorNode connector, final 
Future<Void> future) throws Throwable {
+        try {
+            future.get(flowEvolutionDrainTimeout.toNanos(), 
TimeUnit.NANOSECONDS);
+        } catch (final TimeoutException e) {
+            final FlowUpdateException timedOut = new FlowUpdateException("Flow 
evolution drain timed out for " + connector, e);
+            connector.abortUpdate(timedOut);
+            throw timedOut;
+        } catch (final InterruptedException e) {
+            connector.abortUpdate(e);
+            throw e;
+        } catch (final CancellationException e) {
+            throw new FlowUpdateException("Flow evolution was cancelled for " 
+ connector, e);
+        } catch (final ExecutionException e) {
+            final Throwable cause = e.getCause() == null ? e : e.getCause();
+            if (cause instanceof final CancellationException cancelled) {
+                throw new FlowUpdateException("Flow evolution was cancelled 
for " + connector, cancelled);
+            }
+            throw cause;
+        }
+    }
+
+    private static Duration canonicalFlowEvolutionDrainTimeout(final Duration 
timeout) {
+        if (timeout == null || timeout.isZero() || timeout.isNegative()) {
+            return DEFAULT_FLOW_EVOLUTION_DRAIN_TIMEOUT;
+        }
+
+        return timeout;
+    }
+
+    private void handleFlowEvolutionFailure(final ConnectorNode connector, 
final CompletableFuture<Void> evolutionFuture,
+            final Throwable failure, final boolean startAttempted) {
+        // A failed evolution is not saved; the persisted flow stays as it was 
last saved until the connector is corrected.
+        // A failed start of the evolved connector is an ordinary failed 
start, because the upgraded flow has already been saved.
+        if (!startAttempted) {
+            try {
+                // While DRAINING this fails the evolution. Once preparation 
for the upgraded flow has begun it leaves UPDATE_FAILED.
+                connector.abortUpdate(failure);
+                if (connector.getCurrentState() == ConnectorState.STOPPING && 
connector instanceof final StandardConnectorNode standardConnector) {

Review Comment:
   [Claude Opus 5.5] Minor: the cast to `StandardConnectorNode` to call 
`retryPreUpgradeStop` suggests this retry belongs inside the node.
   



##########
nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java:
##########
@@ -979,7 +1032,7 @@ private void start(final FlowEngine scheduler, final 
CompletableFuture<Void> sta
                         
stateTransition.addPendingStartFuture(startCompleteFuture);
                         return;
                     }
-                    case STOPPED, PREPARING_FOR_UPDATE, UPDATED -> {
+                    case STOPPED, PREPARING_FOR_UPDATE, UPDATED, UPDATE_FAILED 
-> {

Review Comment:
   [Claude Opus 5.5] Only `StandardConnectorRepository.startConnector` checks 
for a pending upgrade. `start()` itself doesn't, and at least one caller goes 
around the repository:
   - On reconnect, `StandardConnectorAssetSynchronizer` restarts Connectors 
that are desired RUNNING and whose assets it just downloaded 
([#L124-L128](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/asset/StandardConnectorAssetSynchronizer.java#L124-L128)).
   - A draining Connector is [desired 
RUNNING](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorNode.java#L2014).
   - `restartConnector` stops it, which cancels the upgrade, then [calls 
`connector.start()` 
directly](https://github.com/apache/nifi/blob/53ffb1d9b528561f46ae7b3dfd7975e01b57c8db/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/components/connector/StandardConnectorRepository.java#L787).
 That runs the new Connector code on the pre-upgrade flow.
   
   I'd put the guard in the node, either by refusing to start while an upgrade 
is pending or by having `start()` handle that case, so that every current and 
future caller is covered.
   



##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/connectors/ConnectorParameterContextIT.java:
##########
@@ -108,6 +109,20 @@ public void 
testParameterContextInheritanceWithSensitiveAndAssetParameters() thr
 
         assertEquals(SENSITIVE_SECRET_VALUE, sensitiveContent, "Sensitive 
output file should contain the secret value");
         assertEquals(ASSET_FILE_CONTENT, assetContent, "Asset output file 
should contain the asset file contents");
+
+        // A same-bundle restart resumes the Connector with its sensitive and 
asset parameters still resolved
+        getNiFiInstance().stop();

Review Comment:
   [Claude Opus 5.5] This restart adds time to the suite but doesn't protect 
against the earlier bug where a normal restart restored the saved flow over the 
newly built one. The sensitive value here comes straight from the secret, so 
the saved flow and the newly built flow are the same, and the test would pass 
with or without that bug. I'd either drop this restart and cover that case with 
a unit test in `TestStandardConnectorRepository`, or use a value that the 
connector derives from the secret so that the restart actually checks something.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to