ump45nose opened a new pull request, #7228: URL: https://github.com/apache/datafusion-comet/pull/7228
## Which issue does this PR close? Part of #7213. This fixes the corrupt-file case. The duplicate-field case from the issue is not changed here. ## Rationale for this change When the native Parquet scan can't read a file (for example a corrupt footer), the error reaches the JVM as `CannotReadFile` with an empty `file_path`. The JVM falls back to the task's file list, so the `FAILED_READ_FILE.NO_HINT` message names no file, or lists every file of the task. Spark names the file that failed. ## What changes are included in this PR? - `FileContext` in `file_error_context.rs` already knows which file each planner and stream belongs to, and already fills the path for `ParquetTimestampOverflow`. It now also converts file read failures into `CannotReadFile` with that file's path, both when the file is opened (the planner and its pending planner, where the footer is read) and while its batches are read. - The classification is the one the JNI conversion already uses: a new `file_read_error` in `jni-bridge/src/errors.rs` wraps `try_classify_file_read_error` and returns only `CannotReadFile`. Missing files stay `FileNotFound`, Spark errors the reader raised itself (for example `ParquetMissingFieldIds`) keep their type, and bare I/O errors are still not treated as file reads. - The JVM fallback to the task's file list is unchanged and still covers read errors raised outside the scan. ## How are these changes tested? New unit tests in `file_error_context.rs`: - `read_failures_name_the_failed_file`: a corrupt-footer `ParquetError::General` and a `Context`-wrapped `EOF` become `CannotReadFile` with the file's path, and the native message is kept. - `other_errors_keep_their_type`: `ParquetMissingFieldIds`, a missing file and a bare `IoError` are not turned into `CannotReadFile`. I ran the `parquet::` lib tests in the `datafusion-comet` crate (330 passed) and `cargo clippy -p datafusion-comet -p datafusion-comet-jni-bridge --all-targets -- -D warnings`. The `datafusion-comet-jni-bridge` lib tests have the same 11 JVM-backed failures on my machine with and without this change. I did not run the Spark/JVM suites, so an end-to-end check with the reproduction from the issue would be welcome. AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
