[
https://issues.apache.org/jira/browse/NIFI-190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Gresock updated NIFI-190:
--------------------------------
Description:
Our team has developed a processor for the following use case:
* Format A needs to be sent to Endpoint A
* Format B needs to be sent to Endpoint B, but should not proceed until A has
reached Endpoint A. We most commonly have this restriction when Endpoint B
requires some output of Endpoint A.
The proposed Wait/Notify processors enable this functionality:
* Wait: routes files to the 'wait' relationship until a matching Release Signal
Identifier is found in the distributed map cache. Then routes them to
'success' (unless they have expired)
* Notify: stores a Release Signal Identifier in the distributed map cache,
optionally with attributes to copy to the outgoing matching Wait flow files.
An example:
Wait is configured with Release Signal Attribute = "${myId}". Its 'wait'
relationship routes back onto itself.
flowFile 1 { myId : "123" }
comes into Wait processor
Wait checks the distributed cache map for "123", doesn't find it, and is
routed to the 'wait' relationship
Notify is configured with Release Signal Attribute = "${myId}"
flowFile 2 { myId : "123" }
comes in to Notify processor
Notify puts an entry in the map for "123" with any other attributes from
flowFile2
Next time flowFile 1 is processed by Wait...
Finds an entry for "123"
Removes that entry from the map
Copies attributes to flowFile 1
Sends flowFile 1 out the success relationship
Signal flow files will also copy their attributes to matching held files, as
optionally configured by an attribute name regex property. This is what allows
the output of Endpoint A to pass to Endpoint B, above. Wait also allows
conflicting attributes to either be replaced or kept, depending on property
configuration.
was:
Our team has developed a processor for the following use case:
* Format A needs to be sent to Endpoint A
* Format B needs to be sent to Endpoint B, but should not proceed until A has
reached Endpoint A. We most commonly have this restriction when Endpoint B
requires some output of Endpoint A.
The proposed HoldFile processor takes 2 types of flow files as input:
* Files to be held
* Signal files that can release corresponding held files, based on the value of
a configurable "release" attribute
Signal files are distinguished from held files by the presence of the
"flow.file.release.value" attribute. The processor is configured with a
"Release Signal Attribute". Held files with this attribute whose value matches
a received signal value will be released.
An example:
HoldFile is configured with Release Signal Attribute = "myId". Its 'Hold'
relationship routes back onto itself.
1. flowFile 1 { myId : "123" } enters HoldFile. It is routed to the 'Hold'
relationship
2. flowFile 2 { flow.file.release.value : "123" } enters HoldFile. flowfile 1
is then routed to 'Release', and flow file 2 is removed from the session.
Signal flow files will also copy their attributes to matching held files,
unless otherwise indicated. This is what allows the output of Endpoint A to
pass to Endpoint B, above.
> Wait/Notify processors
> ----------------------
>
> Key: NIFI-190
> URL: https://issues.apache.org/jira/browse/NIFI-190
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Joseph Gresock
> Assignee: Bryan Bende
> Priority: Minor
> Fix For: 1.2.0
>
> Attachments: Wait_Notify_template.xml
>
>
> Our team has developed a processor for the following use case:
> * Format A needs to be sent to Endpoint A
> * Format B needs to be sent to Endpoint B, but should not proceed until A has
> reached Endpoint A. We most commonly have this restriction when Endpoint B
> requires some output of Endpoint A.
> The proposed Wait/Notify processors enable this functionality:
> * Wait: routes files to the 'wait' relationship until a matching Release
> Signal Identifier is found in the distributed map cache. Then routes them to
> 'success' (unless they have expired)
> * Notify: stores a Release Signal Identifier in the distributed map cache,
> optionally with attributes to copy to the outgoing matching Wait flow files.
> An example:
> Wait is configured with Release Signal Attribute = "${myId}". Its 'wait'
> relationship routes back onto itself.
> flowFile 1 { myId : "123" }
> comes into Wait processor
> Wait checks the distributed cache map for "123", doesn't find it, and is
> routed to the 'wait' relationship
> Notify is configured with Release Signal Attribute = "${myId}"
> flowFile 2 { myId : "123" }
> comes in to Notify processor
> Notify puts an entry in the map for "123" with any other attributes from
> flowFile2
> Next time flowFile 1 is processed by Wait...
> Finds an entry for "123"
> Removes that entry from the map
> Copies attributes to flowFile 1
> Sends flowFile 1 out the success relationship
> Signal flow files will also copy their attributes to matching held files, as
> optionally configured by an attribute name regex property. This is what
> allows the output of Endpoint A to pass to Endpoint B, above. Wait also
> allows conflicting attributes to either be replaced or kept, depending on
> property configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
