[
https://issues.apache.org/jira/browse/NIFI-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422198#comment-17422198
]
Joe Witt commented on NIFI-9260:
--------------------------------
Wow. Interesting to see that this type of impact can happen with some systems.
Definitely a good idea to make other behaviors an option. The dot file
write/rename construct was purely an idiomatic type thing and particularly
biased toward linux notions of hidden files. This kind of optionality would
be useful on any processors that write files most likely (not necessary to do
all in one)
> Making "write and rename" behaviour optional in PutHDFS
> -------------------------------------------------------
>
> Key: NIFI-9260
> URL: https://issues.apache.org/jira/browse/NIFI-9260
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Simon Bence
> Assignee: Simon Bence
> Priority: Major
>
> In some cases the current "write and rename" behaviour of PutHDFS
> undesirable. Particularly if the the server is backed with AWS S3 which has
> no "native" rename but copies file content when renaming, serious perfomance
> issues might arise.
> In order to avoid this, I propose to make this behaviour optional and make it
> possible for the PutHDFS to write directly to the final destination. Of
> course, depending on the use case, there is the risk of reading
> "half-written" files in this manner, but in some scenarios this is not a
> danger. Also, in case of S3 this should not be an issue as files in S3 are
> visible only after the "write" operation is finished
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
