Gabriel39 opened a new pull request, #68856:
URL: https://github.com/apache/doris/pull/68856
### What problem does this PR solve?
Hudi JNI scans on branch-4.1 fail with `NoSuchMethodError:
org.apache.parquet.format.Statistics.isSetNan_count()` after the Parquet 1.18.0
upgrade (#68557).
The BE preload assembly unpacks both the explicitly pinned
`parquet-format:2.12.0` and `parquet-hadoop-bundle:1.18.0`. The former supplies
an older `Statistics` class, while the latter supplies a reader that calls a
method absent from that class. The scanner inherits these incompatible classes
from its parent classloader.
Remove the redundant standalone format dependency so the format classes and
readers come from the same Parquet bundle. Keep the existing Parquet version
and classloader behavior.
### Release note
Fix Hudi JNI scan failures caused by incompatible Parquet classes in the BE
preload jar.
### Check List (For Author)
- Test
- [x] Manual test (details below)
- Behavior changed:
- [x] Yes. Hudi JNI scans can convert Parquet statistics without the
missing-method error.
- Does this need documentation?
- [x] No.
### Validation
- Assembled the unmodified preload module and reproduced the missing-method
error by converting INT64 statistics.
- Cleanly reassembled the modified module with `mvn -pl
be-java-extensions/preload-extensions clean assembly:single@make-assembly-deps`
from `fe/`.
- Ran the same statistics conversion against the rebuilt jar, with Hadoop
Common and JTS Core runtime dependencies on the classpath; it passed. Both
`Statistics` and `ParquetMetadataConverter` loaded from the rebuilt preload jar.
- Compared all 1,795 `org/apache/parquet/**/*.class` entries from the bundle
against the rebuilt assembly; all matched byte for byte.
- Passed `mvn -pl be-java-extensions/preload-extensions -am
checkstyle:check` and checked the reactor dependency tree.
The statistics conversion probe uses:
```java
Statistics stats = new Statistics();
stats.setNull_count(0);
ParquetMetadataConverter.fromParquetStatistics(
"parquet-mr version 1.18.0", stats, PrimitiveTypeName.INT64);
```
Full Hudi integration tests were not run locally. Existing Hudi catalog,
incremental, time-travel, snapshot, and schema-evolution regression suites
cover the affected JNI path and should be rerun in CI.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
