Hello How does nifi grab the data from the file system? It sounds like it is doing partial reads due to a competing consumer (data still being written) scenario.
Thanks On Mon, Oct 11, 2021 at 10:36 PM Jens M. Kofoed <[email protected]> wrote: > Dear Developers > > We have a situation where we see corrupted file after using PutSFTP and > FetchSFTP in NIFI 1.13.2 with openjdk version "1.8.0_292", OpenJDK Runtime > Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10), OpenJDK 64-Bit > Server VM (build 25.292-b10, mixed mode) running on a Ubuntu Server 20.04 > > We have a flow between 2 separated systems where we use a PUTSFTP to export > data from one NIFI instance to a datadiode and use FetchSFTP to grep data > on the other end. To be sure data is not corrupted we calculate a SHA256 on > each side, and transfer the flowfile metadata in a seperate file. In rare > cases have see that the SHA256 doesn't match on both sides and are > investigation where the errors happens. We see 2 errors. Manually > calculation a SHA256 on both side of the diodes the file is OK and we have > found that the errors at happens between NIFI and the SFTP servers. And it > can happens at both sides. > So for testing I created this little flow: > GeneratingFlowFile (size 100MB) (Run once) -> > CryptographicHashContent (SHA256) -> > UpdateAttribute ( hash.root = ${content_SHA-256} , iteration=1) -> > PutSFTP -> > FetchSFTP -> > CryptographicHashContent (SHA256) -> > routeOnAttribute (compare root.hash vs.content_SHA-256) > If unmatch -> > Going to a disabled process for placeholding the corrupted file in > a file queue > If match -> > UpdateAttribute ( iteration= ${iteration:plus(1)} ) -> looping back > to PutSFTP > > After 8992 iteration the file is corrupted. To test if the errors are in > the calculation of the SHA256 I have a copy of the flow without the > PUT/FETCH SFTP processors which haven't got any errors yet. > > It is very rare that we see these errors, millions of files are going > through without any issues but some time it happens which is not good. > > Can any one please help? Maybe trying to setup the same test and see if you > also have a corrupted file after some days. > > Kind regards > Jens M. Kofoed >
