[
https://issues.apache.org/jira/browse/NIFI-190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Gresock updated NIFI-190:
--------------------------------
Description:
Our team has developed a set of processors for the following use case:
* Format A needs to be sent to Endpoint A
* Format B needs to be sent to Endpoint B, but should not proceed until A has
reached Endpoint A. We most commonly have this restriction when Endpoint B
requires some output of Endpoint A.
The proposed Wait/Notify processors enable this functionality:
* Wait: routes files to the 'wait' relationship until a matching Release Signal
Identifier is found in the distributed map cache. Then routes them to
'success' (unless they have expired)
* Notify: stores a Release Signal Identifier in the distributed map cache,
optionally with attributes to copy to the outgoing matching Wait flow files.
An example:
Wait is configured with Release Signal Attribute = "${myId}". Its 'wait'
relationship routes back onto itself.
flowFile 1 { myId : "123" }
comes into Wait processor
Wait checks the distributed cache map for "123", doesn't find it, and is
routed to the 'wait' relationship
Notify is configured with Release Signal Attribute = "${myId}"
flowFile 2 { myId : "123" }
comes in to Notify processor
Notify puts an entry in the map for "123" with any other attributes from
flowFile2
Next time flowFile 1 is processed by Wait...
Finds an entry for "123"
Removes that entry from the map
Copies attributes to flowFile 1
Sends flowFile 1 out the success relationship
Notify will optionally cache attributes in the distributed map, as determined
by a regex property. This is what allows the output of Endpoint A to pass to
Endpoint B, above. Wait also allows conflicting attributes from the cache to
either be replaced or kept, depending on property configuration.
was:
Our team has developed a processor for the following use case:
* Format A needs to be sent to Endpoint A
* Format B needs to be sent to Endpoint B, but should not proceed until A has
reached Endpoint A. We most commonly have this restriction when Endpoint B
requires some output of Endpoint A.
The proposed Wait/Notify processors enable this functionality:
* Wait: routes files to the 'wait' relationship until a matching Release Signal
Identifier is found in the distributed map cache. Then routes them to
'success' (unless they have expired)
* Notify: stores a Release Signal Identifier in the distributed map cache,
optionally with attributes to copy to the outgoing matching Wait flow files.
An example:
Wait is configured with Release Signal Attribute = "${myId}". Its 'wait'
relationship routes back onto itself.
flowFile 1 { myId : "123" }
comes into Wait processor
Wait checks the distributed cache map for "123", doesn't find it, and is
routed to the 'wait' relationship
Notify is configured with Release Signal Attribute = "${myId}"
flowFile 2 { myId : "123" }
comes in to Notify processor
Notify puts an entry in the map for "123" with any other attributes from
flowFile2
Next time flowFile 1 is processed by Wait...
Finds an entry for "123"
Removes that entry from the map
Copies attributes to flowFile 1
Sends flowFile 1 out the success relationship
Signal flow files will also copy their attributes to matching held files, as
optionally configured by an attribute name regex property. This is what allows
the output of Endpoint A to pass to Endpoint B, above. Wait also allows
conflicting attributes to either be replaced or kept, depending on property
configuration.
> Wait/Notify processors
> ----------------------
>
> Key: NIFI-190
> URL: https://issues.apache.org/jira/browse/NIFI-190
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
> Reporter: Joseph Gresock
> Assignee: Joseph Gresock
> Priority: Minor
> Fix For: 1.2.0
>
> Attachments: Wait_Notify_template.xml
>
>
> Our team has developed a set of processors for the following use case:
> * Format A needs to be sent to Endpoint A
> * Format B needs to be sent to Endpoint B, but should not proceed until A has
> reached Endpoint A. We most commonly have this restriction when Endpoint B
> requires some output of Endpoint A.
> The proposed Wait/Notify processors enable this functionality:
> * Wait: routes files to the 'wait' relationship until a matching Release
> Signal Identifier is found in the distributed map cache. Then routes them to
> 'success' (unless they have expired)
> * Notify: stores a Release Signal Identifier in the distributed map cache,
> optionally with attributes to copy to the outgoing matching Wait flow files.
> An example:
> Wait is configured with Release Signal Attribute = "${myId}". Its 'wait'
> relationship routes back onto itself.
> flowFile 1 { myId : "123" }
> comes into Wait processor
> Wait checks the distributed cache map for "123", doesn't find it, and is
> routed to the 'wait' relationship
> Notify is configured with Release Signal Attribute = "${myId}"
> flowFile 2 { myId : "123" }
> comes in to Notify processor
> Notify puts an entry in the map for "123" with any other attributes from
> flowFile2
> Next time flowFile 1 is processed by Wait...
> Finds an entry for "123"
> Removes that entry from the map
> Copies attributes to flowFile 1
> Sends flowFile 1 out the success relationship
> Notify will optionally cache attributes in the distributed map, as determined
> by a regex property. This is what allows the output of Endpoint A to pass to
> Endpoint B, above. Wait also allows conflicting attributes from the cache to
> either be replaced or kept, depending on property configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
