[
https://issues.apache.org/jira/browse/NIP-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18061960#comment-18061960
]
Joe Witt commented on NIP-21:
-----------------------------
A couple higher level thoughts:
This goes beyond strictly the definition of the flow. Should it be considered
that we generalize this API to 'Flow export/import' and then there are a couple
modes:
1. flow definition w external CS
2. flow definition wo external CS
3. flow definition w external CS and component state
We do want to be clear that this still does not include carriage of sensitive
parameter or property values. It does not include flowfiles in the flow. And
the carriage of state for components *could* arguably include sensitive
environment specific data. I don't think it rises to the same level as
sensitive params/properties but we still want to clearly document/express that.
> Flow Definition Export/Import with Components State
> ---------------------------------------------------
>
> Key: NIP-21
> URL: https://issues.apache.org/jira/browse/NIP-21
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Assignee: Pierre Villard
> Priority: Major
>
> *Motivation*
> It can happen that it is necessary, for many reasons, to move a flow
> definition from one NiFi environment to another. When that happens, we
> currently don't have any way to easily move the state (if any) of the
> components included in the flow definition. This can make things particularly
> annoying when it means re-ingesting a lot of previously ingested data.
> We always provided some workarounds: some components expose properties
> allowing to set an initial state, or it could be possible to temporarily
> modify the flow to "skip" the data until components get back to the same
> state as in the source environment. But these workarounds are error-prone,
> time-consuming, and not always available for every component.
> It would make things much easier if we expose the possibility to export the
> flow definition and include component state in it. And, when importing such a
> flow definition, properly initialize the component state with the information
> from the JSON file.
> This feature is scoped exclusively to the JSON-based "Download flow
> definition" / "Upload flow definition" workflow. It does not apply to flow
> versioning via a registry client (registry snapshots are meant to be portable
> flow definitions, not point-in-time runtime snapshots, and including state in
> a registry would not make sense).
> *NiFi API Changes*
> The changes to the NiFi API are limited to two additions and are fully
> backward compatible.
> * A new class *VersionedComponentState* :
> {code:java}
> public class VersionedComponentState {
> // The cluster-scoped state of the component, or null if the component
> has no cluster state.
> private Map<String, String> clusterState;
> // The local-scoped state of the component for each node, keyed by
> ordinal node index
> // ("0", "1", ...), or null if the component has no local state.
> // In a standalone instance, there is a single entry with key "0".
> private Map<String, Map<String, String>> localNodeStates;
> }
> {code}
> * A new optional field *componentState* of type *VersionedComponentState* on
> the existing *VersionedConfigurableExtension* class:
> {code:java}
> private VersionedComponentState componentState;
> @Schema(description = "The state of the component, included when the flow
> definition is exported with state")
> public VersionedComponentState getComponentState() { return componentState; }
> public void setComponentState(VersionedComponentState componentState) {
> this.componentState = componentState; }
> {code}
> When not exporting with state, this field is null and omitted from the
> serialized JSON. When a NiFi instance that does not yet support this feature
> receives a flow definition containing component state, Jackson will silently
> ignore the unknown *componentState* field. This ensures full backward
> compatibility.
> *REST API Changes*
> The existing download endpoint *GET /process-groups/\{id}/download* gains an
> additional optional query parameter:
> - includeComponentState (boolean, default false): When set to true, the
> exported flow definition will include the state of all stateful processors
> and controller services in the process group (recursively).
> No new endpoints are required. The existing upload/import endpoint handles
> the new componentState field automatically.
> *Behavioral Constraints*
> * Components must be stopped for export with state: When
> includeComponentState=true, all processors in the process group must be
> stopped and all controller services must be disabled. If any component is
> running, the request is rejected with a 409 Conflict. This guarantees state
> consistency at the time of export.
> * This feature never applies to registry versioning: The componentState
> field is only populated during JSON export. The flow mapper used for registry
> versioning does not include state. On the import side, componentState is only
> acted upon if non-null (flow snapshots from a registry will never have this
> field, so no state initialization occurs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
