n3nash commented on a change in pull request #1332: [HUDI -409] Match header
and footer block length to improve corrupted block detection
URL: https://github.com/apache/incubator-hudi/pull/1332#discussion_r379556420
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -239,6 +239,15 @@ private boolean isBlockCorrupt(int blocksize) throws
IOException {
return true;
}
+ // check if the blocksize mentioned in the footer is the same as the
header; by seeking back the length of a long
+ // the backward seek does not incur additional IO as {@link
org.apache.hadoop.hdfs.DFSInputStream#seek()}
+ // only moves the index. actual IO happens on the next read operation
+ inputStream.seek(inputStream.getPos() - Long.BYTES);
Review comment:
Also, we need some kind of abstraction here, feel like the reading of the
block size based on long bytes is leaking here and will open to other issues
when trying to evolve the code. For now, can you move this to a method and add
a test case just for this ? Make that method @VisibleForTesting so it's
available to be called in the test class - this way if the footer is evolved,
we have a test to catch issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
