databird07 opened a new pull request, #20187:
URL: https://github.com/apache/hudi/pull/20187
### Describe the issue this Pull Request addresses
Dependabot flags `org.apache.parquet:parquet-avro` < 1.15.1 ("Apache Parquet
Avro Module Vulnerable to Arbitrary Code Execution"). The flagged artifact
comes from `${hive.parquet.version}` (1.10.1), and it is shaded into
`hudi-hadoop-mr-bundle`.
We can't just bump the version. The bundle runs on Hive's own parquet
(1.10.x on Hive 3.1.3), and parquet-avro 1.15.x needs `LogicalTypeAnnotation`,
`ParquetConfiguration` and `HadoopParquetConfiguration`, which that runtime
doesn't have. The Dependabot bump https://github.com/apache/hudi/pull/17654
failed `validate-bundles` for this reason and was closed.
The vulnerable path is parquet-avro reading the `java-class` /
`java-key-class` schema properties from the file footer and instantiating that
class (`AvroRecordConverter.getStringableClass` → `ClassUtils.forName`). It
only does this when the Avro data model is `SpecificData`/`ReflectData`. When
no model is set, parquet-avro falls back to `SpecificDataSupplier`, and that is
the default every Hudi reader was getting.
### Summary and Changelog
Hudi's parquet-avro readers now use `GenericData` explicitly.
Schema-supplied class names are never resolved and stringable fields are read
as `Utf8`/`String`. Hudi only works with `GenericRecord`/`IndexedRecord`, so
the results are unchanged.
- `HoodieAvroParquetReaderBuilder`: `model` defaults to `GenericData.get()`
instead of `null`. This covers `io.storage.hadoop.HoodieAvroParquetReader`.
Callers can still override it with `withDataModel`.
- `HoodieAvroReadSupport()`: the no-arg constructor now calls
`super(GenericData.get())`.
- `ParquetUtils`: all four `AvroParquetReader.builder(...)` call sites pass
`.withDataModel(GenericData.get())`. In parquet-avro,
`withDataModel(GenericData)` keeps `isReflect = true`, so `AVRO_COMPATIBILITY`
behavior is unchanged.
- `hadoop.avro.HoodieAvroParquetReader` (MR): `new
AvroReadSupport<>(GenericData.get())`.
- `HDFSParquetImporterUtils`: sets
`AvroReadSupport.setAvroDataSupplier(conf, GenericDataSupplier.class)` for the
`ParquetInputFormat` job.
- New `TestAvroStringableClassResolution`: writes a parquet file whose Avro
schema has `java-class` pointing at a probe class, then reads it with
`HoodieAvroParquetReaderBuilder` and `ParquetUtils.readAvroRecords`. It asserts
the probe is never instantiated. Without the main-code changes, both tests
fail: the field comes back as the probe instance.
No dependency versions change. With this in place, the alert can be
dismissed as "vulnerable code is not actually used" until the Hive bundle can
move to a newer parquet.
### Impact
None on public APIs. Reads that relied on parquet-avro resolving
`java-class` into custom classes now get plain strings. Hudi doesn't write such
schemas.
### Risk Level
low. This only changes the Avro data model on read paths that already
produce generic records. Verified locally: built
`hudi-hadoop-common`/`hudi-hadoop-mr`, and `TestAvroStringableClassResolution`,
`TestHoodieAvroReadSupport` and `TestParquetUtils` all pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
