[
https://issues.apache.org/jira/browse/NIFI-16477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-16477.
-------------------------------
Fix Version/s: 2.13.0
Resolution: Fixed
> Implement Flow Evolution Drain Hooks and Flow persistence
> ---------------------------------------------------------
>
> Key: NIFI-16477
> URL: https://issues.apache.org/jira/browse/NIFI-16477
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Bob Paulin
> Assignee: Bob Paulin
> Priority: Major
> Fix For: 2.13.0
>
> Time Spent: 14h 10m
> Remaining Estimate: 0h
>
> Framework changes needed for NIP-49
>
> h3. *Persistence*
> VersionedComponentFlowMapper.mapConnector writes:
> * {{{}managedParameterContext{}}}, {{{}managedFlowBundle{}}}, and
> {{activeFlowConfiguration}} from one immutable in-memory
> {{{}ConnectorFlowSnapshot{}}}, never by combining values from different
> lifecycle phases. The record is a complete Connector flow-persistence value
> that can be prepared, published as authoritative, or retained for rollback.
> * {{managedProcessGroup}} from that snapshot, except while the connector is
> in Troubleshooting, when it is written from the live managed group.
> * {{bundle}} from the loaded Connector coordinate and {{managedFlowBundle}}
> from the snapshot. While an upgrade is pending, these fields differ.
> The authoritative snapshot is set after every successful apply, after an
> upgraded snapshot is committed, and when a persisted authoritative flow is
> loaded. {{getAuthoritativeFlowSnapshot()}} names the currently committed
> instance; prepared and rollback snapshots use the same
> {{ConnectorFlowSnapshot}} value type. It is captured on entering
> troubleshooting only when none is held. While troubleshooting, the persisted
> managed group is the live edits and the snapshot bundle stays the version
> that built the flow. On exiting troubleshooting, the snapshot is refreshed
> from the installed Active flow, including when an upgrade was pending.
> h3. *Detection*
> During synchronization, the framework resolves the snapshot bundle from
> {{{}managedFlowBundle{}}}, or infers it once from the persisted {{bundle}}
> when a managed group exists, including a group persisted while
> Troubleshooting. The inferred value is frozen before the loaded version can
> change. No value is inferred when there is no managed group.
> Troubleshooting is restored from {{managedProcessGroup}} and defers evolution
> until the Connector exits troubleshooting. The same detection rule applies
> whenever the loaded version changes.
> h3. *Replacing the flow*
> Each node waits for the {{prepareForFlowEvolution}} future before applying
> the working configuration and preparing the incoming snapshot. A timeout or
> cancellation cancels the future, retains the authoritative pre-upgrade
> snapshot, and leaves the Connector stopped and pending. After cluster-wide
> commit authorization, every node atomically publishes the incoming snapshot
> and persists its own flow, and START waits for cluster-wide persistence
> completion before proceeding to running. Publishing the new snapshot makes
> its managed-flow bundle equal the loaded bundle, so pending becomes false
> without clearing a stored flag.
> If preparing or committing the upgraded flow fails, the outgoing snapshot
> remains authoritative or is restored, and the node stays stopped and pending.
> The existing flow-update verification remains as a backstop for a FlowFile
> that arrives after the check, or for an {{applyUpdate}} that installs
> something other than {{getActiveFlow}} on the working context.
> *Exiting troubleshooting while an upgrade is pending*
> {{prepareForFlowEvolution}} is not run. Every queue in the managed flow,
> including child groups, must be empty. The exit is rejected while any
> FlowFiles remain, and the Connector stays in Troubleshooting with the upgrade
> still pending. When the queues are empty, the same synchronous exit used
> otherwise installs {{{}getActiveFlow{}}}, captures the snapshot with the
> loaded bundle, and leaves the Connector {{STOPPED}} and no longer pending.
> This covers a connector that was in troubleshooting when the runtime was
> upgraded, and a connector whose drain was cancelled so the user could enter
> troubleshooting from {{{}STOPPED{}}}.
> The hook is skipped because the empty-queue check already makes the
> replacement safe, and the shipped hooks only drain. Draining on exit would
> start components the user added and stopped, against a flow whose structure
> the hook cannot rely on after those edits. The user can empty the queues
> before exiting, and the exit stays a synchronous verify-then-apply across the
> cluster.
> h2. *New configuration*
> nifi.components.connectors.flow.evolution.timeout bounds
> prepareForFlowEvolution. The default is 30 mins.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)