[
https://issues.apache.org/jira/browse/NIFI-13402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103852#comment-18103852
]
Matt Burgess commented on NIFI-13402:
-------------------------------------
Quoted from the PR (to bring the discussion back here), [~exceptionfactory]'s
comments:
bq. Although I appreciate the interest in pursuing these changes, I currently
remain opposed to further extension of the Python Processor API to support this
type of use case with multiple FlowFiles.
bq. Python Processors have inherent scalability challenges due to the decoupled
nature of the Python process itself, and the socket communication between
Python processes and the JVM.
bq. Extending the Python API introduces new maintenance concerns on top of the
foundational issues.
bq. I think it would be helpful to discuss this in Jira before proceeding with
code changes, so I'm going to vote against this addition for now.
TBH my PR was a direct approach to your comments on the previous PR from
[~pvillard]:
bq. I think it would be better to introduce a new type of interface that
explicitly expects to return a collection of FlowFile results. This would avoid
making the existing interfaces too generic, and also make it clear when a
Python Processor is expecting to return a collection of results.
That is what my PR is trying to do. Perhaps some use cases to illustrate:
- GenerateRecord.py
(nifi-extension-bundles/nifi-py4j-extension-bundle/nifi-python-test-extensions/src/main/resources/extensions/GenerateRecord.py):
Needs an upstream connection for no reason, can only output one FlowFile. If
1000 FlowFiles are needed, the upstream processor has to run exactly 1000 times
(no more no less). Sure you can output a FlowFile with 1000 records but then
you need a SplitRecord later (depending on the target system). This pattern is
quite common in my experience.
- Enrichment: If enrichment of an input FlowFile is needed (record-based or
not) and the system is not immediately accessible via an existing Controller
Service, you have to script the Controller Service or the Enrichment code in a
separate language, defeating the purpose of the Python SDK's claim to support
rapid development. If there's a Python library that can be brought in to do the
enrichment "in-house", so much the better.
- RouteRecord: If there is complex logic to route content or records, we have
QueryRecord which requires SQL, again taking you out of the Python
"environment" and impeding rapid development.
- Transform processors that generate content needing to go to different
relationships: For things like Object detection, you may want to send the image
to a model and return bounding boxes in JSON to one relationship, the original
image to another, and the individually separated object images to another.
Objection handling:
I too am wary of the scalabilty challenges and socket communication, and
perhaps there should be a clear warning / explanation of such in the Python
Developer's Guide if there isn't already. Returning 1 billion FlowFiles from a
Python processor is a terrible idea. With great(er) power comes great(er)
responsibility.
I also agree that extending the Python API introduces new maintenance concerns,
and I would hope (with the exception of a Many-to-Many addition?) it would not
be extended any further. Fortunately the existing interface implementations are
~100 lines of code apiece and pretty straightforward, IMO the bugs might show
up in 1 or 2 of them but not all of them at the same time.
Hopefully we can get some more input from the community before any decision is
made. Thank you for the discussion!
> Multiple flowfiles as output for Python Processors
> --------------------------------------------------
>
> Key: NIFI-13402
> URL: https://issues.apache.org/jira/browse/NIFI-13402
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
> Reporter: Asif Arman Rahman
> Assignee: Matt Burgess
> Priority: Major
> Labels: features
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> This is a request to add the feature, where we can split an incoming flowfile
> to multiple flowfiles as output for Python Processor in NiFi 2.0.0.
> Use case: Input is a single flowfile which contains an excel file, and output
> would be multiple flowfiles, where each flowfile will contain one sheet from
> the excel file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
