txwyy123 opened a new issue, #19929:
URL: https://github.com/apache/hudi/issues/19929
### Bug description
After upgrading from Hudi 0.14.x to 1.0.2, the default
`HoodieNativeAvroHFileReader` (enabled via
`hoodie.metadata.reader.use.native.reader=true`, which is the 1.0.2 default)
throws a `java.util.zip.ZipException` when reading certain `column_stats`
partition HFiles in the metadata table (MDT) during compaction. The same HFiles
read successfully when using the legacy `HoodieAvroHFileReader`
(`hoodie.metadata.reader.use.native.reader=false`).
The failure occurs deterministically on specific HFiles — not all
column_stats HFiles trigger it. The affected HFiles are byte-valid (not
corrupted on storage) and were written by Hudi 1.0.2 writers with no errors at
write time. The crash surfaces during the next MDT compaction that reads the
affected HFile block.
### Impact
The `ZipException` crash-loops the writer process on every subsequent write
attempt, because each write triggers MDT compaction which re-reads the same
corrupted block. This makes the affected table unwritable until the workaround
is applied.
### Steps to reproduce
1. Use Hudi 1.0.2 with `column_stats` metadata partition enabled (default).
2. Write to a COW table with multiple concurrent writers using OCC
(DynamoDB-based lock provider) over an extended period.
3. At some point, an MDT compaction reads a `column_stats` HFile block that
triggers the `ZipException`.
**Note:** The issue is intermittent — it depends on the specific data/block
layout written to the HFile. Not every `column_stats` HFile triggers it.
### Expected behavior
`HoodieNativeAvroHFileReader` should read all HFiles that
`HoodieAvroHFileReader` can read without error. Both readers operate on the
same byte-identical files.
### Actual behavior
```
java.util.zip.ZipException: <decompression error>
at java.base/java.util.zip.Inflater.inflateBytesBytes(Native Method)
at java.base/java.util.zip.Inflater.inflate(Inflater.java:378)
at org.apache.hadoop.hbase.io.compress.HFileDecompressor.decompress(...)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(...)
at org.apache.hudi.io.storage.HoodieNativeAvroHFileReader.readBlock(...)
...
```
The legacy `HoodieAvroHFileReader` reads the same file cleanly.
### Diagnosis
A/B testing confirmed the issue is isolated to the reader implementation:
- **`hoodie.metadata.reader.use.native.reader=true`** (native, 1.0.2
default): `ZipException` on the affected HFile block
- **`hoodie.metadata.reader.use.native.reader=false`** (legacy): reads all
records successfully (confirmed >9M records read without error from the same
file)
The native reader's block decompression logic in
`HoodieNativeAvroHFileReader` mis-handles certain HFile block encodings that
the HBase-backed legacy reader handles correctly. The root cause appears to be
in the native reader's block unpacking/decompression path, not in the data
written to the HFile.
### Workaround
Set `hoodie.metadata.reader.use.native.reader=false` in the writer
configuration to fall back to the legacy `HoodieAvroHFileReader`. This avoids
the crash with no functional difference (same data, same MDT behavior).
### Environment
- **Hudi version:** 1.0.2
- **Spark version:** 3.5.x
- **Java version:** JDK 17
- **Storage:** S3
- **Table type:** COW
- **MDT partitions enabled:** `files`, `column_stats`
- **Lock provider:** DynamoDB-based OCC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
