ump45nose commented on code in PR #7228:
URL: https://github.com/apache/datafusion-comet/pull/7228#discussion_r4381641144
##########
native/core/src/parquet/file_error_context.rs:
##########
@@ -299,4 +314,67 @@ mod tests {
assert!(matches!(error, DataFusionError::External(source)
if matches!(source.downcast_ref::<SparkError>(),
Some(SparkError::DivideByZero))));
}
+
+ fn cannot_read_path(error: &DataFusionError) -> Option<&str> {
+ let DataFusionError::External(source) = error else {
+ return None;
+ };
+ match source.downcast_ref::<SparkError>() {
+ Some(SparkError::CannotReadFile { file_path, .. }) =>
Some(file_path),
+ _ => None,
+ }
+ }
+
+ #[test]
+ fn read_failures_name_the_failed_file() {
Review Comment:
Good point, thanks. Added in 4b7d0757: the corrupt footer and page tests in
`CometTopKSuite` now also run with both files in one partition
(`LEAF_NODE_DEFAULT_PARALLELISM` 1, `FILES_MAX_PARTITION_BYTES` 128 MB) and
assert that the structured `path` equals only the corrupt file. In that variant
the corrupt file is the one holding the smallest keys, so the TopK dynamic
filter cannot prune its row group whichever file is read first. I could not run
the JVM suites locally, so this relies on CI. The same commit updates the
fallback comment in `SparkErrorConverter.scala` to say the native scan now
attaches the path and the fallback covers errors raised outside the scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
