GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/1181
NIFI-1322 - PutHDFS - allow file append resolution
Two remarks:
- LocalFileSystem object used for unit tests does not implement the append
method.
- Based on HDFS-4600 (https://issues.apache.org/jira/browse/HDFS-4600),
appending to a file is only possible if the replication factor is lesser or
equal than the number of data nodes in the cluster. For live testing with a
standalone instance, consider changing the replication factor to 1 for the file
you are appending to.
Also, I considered that it would be up to the user to handle the demarcator
between two appended messages if such a demarcator is expected.
Finally, if two processors are appending to the same file, errors may be
raised due to concurrent access to the file. However, if redirecting the
failure relationship back to the processor, the processors should, in the end,
be able to append the messages. Obviously, in such a case, message ordering
shall not be a requirement. Local testing suggests that users should use a
MergeContent processor as upstream processor in order to ensure acceptable
bandwidth.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-1322
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1181.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1181
----
commit a42ed1d547762b51dbe27275aa54e0736e7046d4
Author: Pierre Villard <[email protected]>
Date: 2016-11-02T19:38:52Z
PutHDFS - allow file append resolution
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
