Ryan Hendrickson created NIFI-9027:
--------------------------------------
Summary: Integration of S3 buckets with Remote Process Groups for
outage events and data overruns
Key: NIFI-9027
URL: https://issues.apache.org/jira/browse/NIFI-9027
Project: Apache NiFi
Issue Type: New Feature
Reporter: Ryan Hendrickson
A frequent struggle is the occasional death of a server in a NiFi flow.
Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis and 2 NiFi
clusters. The stand-alone NiFi's are linearly chained together with Remote
Process Groups to perform the overall dataflow goals. If any one of the 15
experience an outage, data begins to backup on the previous NiFi, hitting
backpressure limits, etc. When the down server comes back online, it receives
a flood of data, often overwhelming it, necessitating careful flowfile limits
and backpressure points.
The design pattern we're exploring, but would like to see baked into NiFi, is
that when a Remote Process Group becomes unavailable due to server-to-server
comms failures AND the backpressure limit is reached on a relationship, the
Remote Process Group would begin sending data to an S3 bucket as an automatic
failover procedure. When the downed server comes back-online, the Remote
Process Group port would know to pull data from S3 AND receive data via the
port from the chained server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
