BalaMahesh opened a new issue #2230: URL: https://github.com/apache/hudi/issues/2230
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** When the compaction job starts, HoodieLogFileReader starts reading the log files generated by delta streamer sync process that are stored in the s3a file system. While reading the each and every log file , it fails with EOF Exception and reports each block as Corrupt block and at the end compaction does nothing. If I download the same file into my local and read the local file with Reader, it is able to do without any issue. So HoodieLogFileReader is failing when it is trying to seek(curPosition+blocksize) from S3AInputStream with EOFException, but seek() on other InputStream is fine. Issue is with seek on len(file). https://issues.apache.org/jira/browse/HADOOP-11270 https://issues.apache.org/jira/browse/HADOOP-11417 **To Reproduce** Steps to reproduce the behavior: 1. Ingest the records using delta streamer in continuous mode and use s3a as file system. 2. CREATE MERGE ON READ HUDI TABLE . 3. Enable inline compaction and check the logs of compaction process **Expected behavior** HoodieLogFileReader should be able to read the log file from s3 **Environment Description** * Hudi version : 0.6.1 * Spark version : 2.4.5 * Hive version : 1.2 * Hadoop version : 2.7.3 * Storage (HDFS/S3/GCS..) : s3a * Running on Docker? (yes/no) : no **Additional context** Add any other context about the problem here. **Stacktrace** ``` 20/11/03 13:15:19 INFO log.HoodieLogFileReader: Log HoodieLogFile{pathStr='s3a://xxx/test/hudi/data/yyyyyy/zzzzz/dt=2020-10-27/.07e2f735-aae4-45aa-b77e-2891c7c793dc-0_20201103122936.log.3_3-463-162543', fileLen=0} has a corrupted block at 14 20/11/03 13:15:20 INFO log.HoodieLogFileReader: Next available block in HoodieLogFile{pathStr='s3a://xxx/test/hudi/data/yyyyyy/zzzzz/dt=2020-10-27/.07e2f735-aae4-45aa-b77e-2891c7c793dc-0_20201103122936.log.3_3-463-162543', fileLen=0} starts at 209548 20/11/03 13:15:20 INFO log.AbstractHoodieLogRecordScanner: Found a corrupt block in s3a://xxx/test/hudi/data/yyyyyy/zzzzz/dt=2020-10-27/.07e2f735-aae4-45aa-b77e-2891c7c793dc-0_20201103122936.log.3_3-463-162543``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
