[
https://issues.apache.org/jira/browse/NIP-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910839#comment-17910839
]
Matt Burgess commented on NIP-2:
--------------------------------
You can to some extent do manual "unit testing" now, as you can send any number
of files downstream using a copy of the incoming relationship. One example is
FetchFile with real data, you send it downstream under the success relationship
to both a downstream processor and a "holder" processor like UpdateAttribute
that will hold the queue. At the end of the test (whether it be after one
processor or a sub-flow) you can send the results to a verification sub-flow
until the data looks good, then attach the connection for the verification
sub-flow to the rest of the real flow and feed the queue from the holder
processor back into the intended flow.
Providing a "preview" on processors interacting with external systems is
absolutely a lot of work and not one I'm necessarily suggesting here, such a
processor would need to implement some interface that knows to rollback
whatever occurred. Even if possible (such as PutDatabaseRecord) you would only
know the data was transferred successfully, but you can't know if it made into
an external system correctly, I don't think we want to do that as part of this.
I envision maybe a drag-n-drop scenario to put multiple files as "data under
test", that gets rid of the cut-and-paste issue with binary data and you can
run multiple files through MergeRecord for example. With multiple files the
attributes would be tricky but perhaps it would support FlowFile ZIP files? I
just think if we only supported paste-able content and only one file with its
attributes it would take a lot of rework to do it more generally. I'm sure I'm
missing some angles here but I'm thinking we try to get as much value for as
little work and rework as possible.
For repeatability we could keep track of the file(s) added in some sort of
"preview" mode so if you ran Preview again it would already have everything set
up. If we don't want to support provenance for Preview events then I agree
there are a lot of changes to be made. If the manual unit testing scenario is
more appealing then I'm ok with that for now, but this NIP is here because it
is a significant amount of work hopefully for a broad a powerful payoff.
> Preview capability for processors
> ---------------------------------
>
> Key: NIP-2
> URL: https://issues.apache.org/jira/browse/NIP-2
> Project: NiFi Improvement Proposal
> Issue Type: New Feature
> Reporter: Matt Burgess
> Priority: Normal
>
> *Motivation*
> The ability to "send" a sample FlowFile through a processor in an existing
> flow, without being associated with the flow itself (provenance, I/O metrics,
> e.g.), rather by executing the configured processor using the provided input
> and returning the output could be highly beneficial for Flow Designers. It's
> somewhat analogous to the "Run Once" feature but aims to operate outside the
> actual flow framework so as not to potentially contaminate the various NiFi
> repositories keeping track of the flow of "real" data.
> *Scope*
> This would involve several changes and additions to the API and codebase, see
> the Description section for details.
> *Compatibility*
> This feature would be backwards-compatible as is it is new and can only be
> executed on a flow in a NiFi instance that supports this capability.
> *Description*
> This capability would provide a user to test a processor already in the flow
> during design before starting the flow with "real" data which can help debug
> flows while preventing data loss during development.
> Tasks:
> * REST API call(s) to POST data to endpoint(s) which respond with machine
> readable output (JSON, XML, e.g.)
> * An annotation on processors such as Previewable, perhaps the
> SideEffectFree annotation suffices but you cannot extend Java annotations so
> SideEffectFree processors may just be a good starting point
> * An established format for both the POST data and the output data (content
> and attributes, multiple FlowFiles, relationships, etc.)
> * UI changes to include:
> -- Context-sensitive (right-click) menu option to Preview
> -- Dialog to paste/upload/drag the input content
> -- Inject parameters (perhaps reuse the Verification capability here as
> prudent)
> -- Leverage the existing Content Viewer as prudent to show the output (both
> content and attributes in one place would be preferable, but code reuse in
> the UI helps with consistency and maintainability)
> * Documentation
> -- Add a section to the processors' documentation if they are Previewable, a
> common description could be provided to avoid the need for custom UIs
> -- Add a section to the User Guide to describe how to preview a Previewable
> processor and how to interpret the results
> -- Add a section as necessary to the Admin guide for allowing/disallowing
> Preview (see the Authorization section below)
> * Authorization
> -- Add a policy to allow/prevent previewing a processor (optional for this
> NIP as this is intended to be a "sandbox" operation)
> *Verification*
> As processors are augmented with the Previewable annotation, the pull
> requests should include unit tests and the overall documentation should
> describe how to use the Preview capability on a Previewable processor.
> *Alternatives*
> On development systems (or in development process groups), the GenerateXYZ
> processors can be used to inject data into the processor and the UI or the
> REST API can be used to inspect the output data from the connection. This can
> be quite cumbersome and requires a separate NiFi system or process group for
> testing flow development, then moving the tested flow into the target NiFi
> system.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
