Ryan Hendrickson created NIFI-9026:
--------------------------------------
Summary: Integration with auto-scaling container environments
Key: NIFI-9026
URL: https://issues.apache.org/jira/browse/NIFI-9026
Project: Apache NiFi
Issue Type: New Feature
Reporter: Ryan Hendrickson
We often struggle with scaling custom processors in large flows. We've
implemented a recent design pattern where we write custom python code packaged
into a docker image, fronted with a web-service, and deployed into a container
environment. In NiFi we use InvokeHTTP to call out to the web-service. If the
load becomes too great, we can easily spin-up more than one external
web-service without sacrificing the NiFi's overall thruput and thread
contention. In this scenario, we're using NiFi to manage the overall dataflow,
routes, ETL services, etc, but not so much smarts around auto-scaling
processing, etc.
This makes it really easy to receive a dataflow analytic/enrichment from an
external team, wrap it in a docker image, and deploy it into an auto-scaling
container environment. This also lets us avoid conversations about how
performant the code is and if it's meeting a minimum throughput requirement -
typically XXGB/5minutes.
The request for 2.x would be to make interacting with container deployed
services a more seamless, streamlined process, possibly through a normalized
API or as a "Remote Container Environment" that can bind to a service with a
health endpoint and verify it's alive and well, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
