[ 
https://issues.apache.org/jira/browse/NIFI-16477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148346#comment-18148346
 ] 

ASF subversion and git services commented on NIFI-16477:
--------------------------------------------------------

Commit c229d1910a6089416950d7eb95d4bc9df8e08813 in nifi's branch 
refs/heads/main from Bob Paulin
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c229d1910a6 ]

NIFI-16477: Implement Flow Evolution Drain Hooks and Flow persistence (#11872)

* NIFI-16477: Implement Flow Evolution Drain Hooks and Flow persistence

* Implement hooks to allow the Connector to drain the flow when its
loaded version changes

* When the Connector version differs from the version that built the
authoritative flow load the persisted flow from the managedProcessGroup.

* NIFI-16477: Address review feedback on Connector flow evolution

- Keep the managed flow and configurations in flow.json when restoring
  the pre-upgrade flow fails at startup; an unreachable Secrets Provider
  counts as unresolved secrets and leaves components stopped
- Complete pending stop futures exceptionally when the Connector moves
to
  a state other than STOPPING or STOPPED
- Reject START of a pending Connector that is STOPPING, PURGING or in a
  user-requested drain, and refuse direct start() while an upgrade is
pending
- Always save RUNNING when an evolution begins so a restart during the
  drain resumes the upgrade
- Replace restore overloads and booleans with ResumeComponentState
- Restore with FlowMappingOptions.forPersistedFlow, matching capture
- Move flow evolution state and steps into ConnectorFlowEvolution
- Make ConnectorClusterUnavailableException an IOException, move the
  pre-upgrade stop retry into the node, and document FlowEvolutionStatus
- Add isSourceChanged helper and restore its synchronizer test
- Add unit tests and a capture/restore round-trip test
- Reduce system test restarts to two by reading flow.json.gz while NiFi
  runs, merging restart tests, and folding
ConnectorUpgradeDrainProviderIT
  into ClusteredConnectorUpgradeDrainIT

* NIFI-16477: Address further review feedback on Connector flow evolution

- Remove the flow evolution drain timeout; log a WARN every 10 minutes
with the queued FlowFile count
- Report an unreachable secrets provider separately from a missing
secret
- Make mapConnectorState exhaustive using
ConnectorNode#getPreUpdateDesiredState
- Fold AbstractNarSwapUpgradeDrainIT into
ClusteredConnectorUpgradeDrainIT

* NIFI-16477: Address latest review feedback on Connector flow evolution

- Refresh the authoritative snapshot when a Connector migration is
committed
- Wait for joining cluster nodes only at the flow evolution drain
barrier
- Adopt the pre-upgrade snapshot when restoring a pending evolution
fails to load configuration
- Fill the managed Parameter Context for a legacy Troubleshooting save
restored after an upgrade

* NIFI-16477: Read bundle from the coordinator's local connector

- Add NiFiServiceFacade.getConnectorBundle, which reads the loaded
bundle
  with LOCAL_ONLY and without filtering by permissions
- Fill expectedBundle on every clustered START, including requests from
  users who can only operate the Connector
- Stop START from syncing the coordinator alone with the configuration
  provider

> Implement Flow Evolution Drain Hooks and Flow persistence
> ---------------------------------------------------------
>
>                 Key: NIFI-16477
>                 URL: https://issues.apache.org/jira/browse/NIFI-16477
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Bob Paulin
>            Assignee: Bob Paulin
>            Priority: Major
>          Time Spent: 14h 10m
>  Remaining Estimate: 0h
>
> Framework changes needed for NIP-49
>  
> h3. *Persistence*
> VersionedComponentFlowMapper.mapConnector writes:
>  * {{{}managedParameterContext{}}}, {{{}managedFlowBundle{}}}, and 
> {{activeFlowConfiguration}} from one immutable in-memory 
> {{{}ConnectorFlowSnapshot{}}}, never by combining values from different 
> lifecycle phases. The record is a complete Connector flow-persistence value 
> that can be prepared, published as authoritative, or retained for rollback.
>  * {{managedProcessGroup}} from that snapshot, except while the connector is 
> in Troubleshooting, when it is written from the live managed group.
>  * {{bundle}} from the loaded Connector coordinate and {{managedFlowBundle}} 
> from the snapshot. While an upgrade is pending, these fields differ.
> The authoritative snapshot is set after every successful apply, after an 
> upgraded snapshot is committed, and when a persisted authoritative flow is 
> loaded. {{getAuthoritativeFlowSnapshot()}} names the currently committed 
> instance; prepared and rollback snapshots use the same 
> {{ConnectorFlowSnapshot}} value type. It is captured on entering 
> troubleshooting only when none is held. While troubleshooting, the persisted 
> managed group is the live edits and the snapshot bundle stays the version 
> that built the flow. On exiting troubleshooting, the snapshot is refreshed 
> from the installed Active flow, including when an upgrade was pending.
> h3. *Detection*
> During synchronization, the framework resolves the snapshot bundle from 
> {{{}managedFlowBundle{}}}, or infers it once from the persisted {{bundle}} 
> when a managed group exists, including a group persisted while 
> Troubleshooting. The inferred value is frozen before the loaded version can 
> change. No value is inferred when there is no managed group.
> Troubleshooting is restored from {{managedProcessGroup}} and defers evolution 
> until the Connector exits troubleshooting. The same detection rule applies 
> whenever the loaded version changes.
> h3. *Replacing the flow*
> Each node waits for the {{prepareForFlowEvolution}} future before applying 
> the working configuration and preparing the incoming snapshot. A timeout or 
> cancellation cancels the future, retains the authoritative pre-upgrade 
> snapshot, and leaves the Connector stopped and pending. After cluster-wide 
> commit authorization, every node atomically publishes the incoming snapshot 
> and persists its own flow, and START waits for cluster-wide persistence 
> completion before proceeding to running. Publishing the new snapshot makes 
> its managed-flow bundle equal the loaded bundle, so pending becomes false 
> without clearing a stored flag.
> If preparing or committing the upgraded flow fails, the outgoing snapshot 
> remains authoritative or is restored, and the node stays stopped and pending. 
> The existing flow-update verification remains as a backstop for a FlowFile 
> that arrives after the check, or for an {{applyUpdate}} that installs 
> something other than {{getActiveFlow}} on the working context.
> *Exiting troubleshooting while an upgrade is pending*
> {{prepareForFlowEvolution}} is not run. Every queue in the managed flow, 
> including child groups, must be empty. The exit is rejected while any 
> FlowFiles remain, and the Connector stays in Troubleshooting with the upgrade 
> still pending. When the queues are empty, the same synchronous exit used 
> otherwise installs {{{}getActiveFlow{}}}, captures the snapshot with the 
> loaded bundle, and leaves the Connector {{STOPPED}} and no longer pending. 
> This covers a connector that was in troubleshooting when the runtime was 
> upgraded, and a connector whose drain was cancelled so the user could enter 
> troubleshooting from {{{}STOPPED{}}}.
> The hook is skipped because the empty-queue check already makes the 
> replacement safe, and the shipped hooks only drain. Draining on exit would 
> start components the user added and stopped, against a flow whose structure 
> the hook cannot rely on after those edits. The user can empty the queues 
> before exiting, and the exit stays a synchronous verify-then-apply across the 
> cluster.
> h2. *New configuration*
> nifi.components.connectors.flow.evolution.timeout bounds 
> prepareForFlowEvolution. The default is 30 mins.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to