[
https://issues.apache.org/jira/browse/NIP-21?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18061965#comment-18061965
]
Pierre Villard commented on NIP-21:
-----------------------------------
Thanks [~markap14] - I agree with your suggestion, it makes total sense.
Thanks [~joewitt] -
Yeah so right now, in the UI, we have two sub-menus to select w or wo external
CS. I think we should remove the sub-menus and instead expose a view to the
user to select the different options for the export. This will provide a better
UX depending on the expected combinations for the user. This will also give us
the possibility to add some textual details to explain what those options mean
(a lot of users are still confused by what "with external controller services"
means).
Regarding the data sensitivity, I definitely agree that this should be made
clear to the user (and the view for selecting the export options will help with
that). Besides the export with state will need to properly check the
authorizations as accessing the state of a component requires WRITE access on
the component while exporting its definition only requires READ access.
> Flow Definition Export/Import with Components State
> ---------------------------------------------------
>
> Key: NIP-21
> URL: https://issues.apache.org/jira/browse/NIP-21
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Pierre Villard
> Assignee: Pierre Villard
> Priority: Major
>
> *Motivation*
> It can happen that it is necessary, for many reasons, to move a flow
> definition from one NiFi environment to another. When that happens, we
> currently don't have any way to easily move the state (if any) of the
> components included in the flow definition. This can make things particularly
> annoying when it means re-ingesting a lot of previously ingested data.
> We always provided some workarounds: some components expose properties
> allowing to set an initial state, or it could be possible to temporarily
> modify the flow to "skip" the data until components get back to the same
> state as in the source environment. But these workarounds are error-prone,
> time-consuming, and not always available for every component.
> It would make things much easier if we expose the possibility to export the
> flow definition and include component state in it. And, when importing such a
> flow definition, properly initialize the component state with the information
> from the JSON file.
> This feature is scoped exclusively to the JSON-based "Download flow
> definition" / "Upload flow definition" workflow. It does not apply to flow
> versioning via a registry client (registry snapshots are meant to be portable
> flow definitions, not point-in-time runtime snapshots, and including state in
> a registry would not make sense).
> *NiFi API Changes*
> The changes to the NiFi API are limited to two additions and are fully
> backward compatible.
> * A new class *VersionedComponentState* :
> {code:java}
> public class VersionedComponentState {
> // The cluster-scoped state of the component, or null if the component
> has no cluster state.
> private Map<String, String> clusterState;
> // The local-scoped state of the component for each node, keyed by
> ordinal node index
> // ("0", "1", ...), or null if the component has no local state.
> // In a standalone instance, there is a single entry with key "0".
> private Map<String, Map<String, String>> localNodeStates;
> }
> {code}
> * A new optional field *componentState* of type *VersionedComponentState* on
> the existing *VersionedConfigurableExtension* class:
> {code:java}
> private VersionedComponentState componentState;
> @Schema(description = "The state of the component, included when the flow
> definition is exported with state")
> public VersionedComponentState getComponentState() { return componentState; }
> public void setComponentState(VersionedComponentState componentState) {
> this.componentState = componentState; }
> {code}
> When not exporting with state, this field is null and omitted from the
> serialized JSON. When a NiFi instance that does not yet support this feature
> receives a flow definition containing component state, Jackson will silently
> ignore the unknown *componentState* field. This ensures full backward
> compatibility.
> *REST API Changes*
> The existing download endpoint *GET /process-groups/\{id}/download* gains an
> additional optional query parameter:
> - includeComponentState (boolean, default false): When set to true, the
> exported flow definition will include the state of all stateful processors
> and controller services in the process group (recursively).
> No new endpoints are required. The existing upload/import endpoint handles
> the new componentState field automatically.
> *Behavioral Constraints*
> * Components must be stopped for export with state: When
> includeComponentState=true, all processors in the process group must be
> stopped and all controller services must be disabled. If any component is
> running, the request is rejected with a 409 Conflict. This guarantees state
> consistency at the time of export.
> * This feature never applies to registry versioning: The componentState
> field is only populated during JSON export. The flow mapper used for registry
> versioning does not include state. On the import side, componentState is only
> acted upon if non-null (flow snapshots from a registry will never have this
> field, so no state initialization occurs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
