Hi,
> Would then the assumption that this possibility ( part reported length >
> part file size ( reported by FileStatus on NN) ) is only attributable to
> this edge case be correct ?
Yes, I think so.
> Or do you see a case where in though the above is true, the part file would
> need tru
Thank you for your email.
Would then the assumption that this possibility ( part reported length >
part file size ( reported by FileStatus on NN) ) is only attributable to
this edge case be correct ?
Or do you see a case where in though the above is true, the part file would
need truncation as a
Hi Vishal,
I’ve come across the same problem. The problem is that by default the file
length is not updated when the output stream is not closed properly.
I modified the writer to update file lengths on each flush, but it comes with
some overhead, so this approach should be used when strong con
This seems strange. When I pull the ( copyToLocal ) the part file to local
FS, it has the same length as reported by the length file. The fileStatus
from hadoop seems to have a wrong length.
This seems to be true for all these type of discrepancies. It might be that
the block information did not g
Hello folks,
I have flink 1.7.2 working with hadoop 2.6 and b'coz there
is no in build truncate ( in hadoop 2.6 ) I am writing a method to cleanup
( truncate ) part files based on the length in the valid-length files
dropped by flink during restore. I see some thing very strange
