ErikBPF opened a new pull request, #5786:
URL: https://github.com/apache/datafusion-comet/pull/5786
## Which issue does this PR close?
Closes #5783.
## Rationale for this change
Native Parquet scans can silently multiply rows when a struct contains
byte-identical sibling names. Resolving or rejecting duplicates in schema
conversion is too late because decoding has already combined the leaves. The
issue explicitly accepts a clear error instead of Spark-compatible duplicate
selection.
## What changes are included in this PR?
- Validate sibling names recursively when the native reader loads metadata,
before decoder construction. Validation also runs after cache hits and covers
structs inside arrays/maps.
- Reject the entire file, including unprojected duplicates and reads using
field IDs, in either case-sensitivity mode. This deliberately conservative
behavior is documented in the Parquet compatibility guide. Users can disable
Comet to use Spark's duplicate-name resolution with an explicit schema.
- Preserve the validation requirement in comments describing eventual
replacement of the page-index reader workaround.
- Add 12 regression/control cases to the existing native-reader suite.
Duplicate fixtures contain three rows in one file, covering batch sizes 1 and
4096. Other checks cover repeated reads, unprojected duplicates, distinct-case
names, and repeated names in separate groups.
## How are these changes tested?
Validated commit `513d6fc26` on Apollo with Spark 4.1, JDK 17, and Rust
1.97.1, after rebasing onto `17f54da8c` (#5751). The native library was rebuilt
from that commit.
- RED on unpatched `fefee03d9`: all 11 negative cases fail with the
single-file fixtures; the valid-name control passes.
- GREEN: 139 Scala tests passed across `CometNativeReaderSuite` and
`ParquetReadV1Suite`, including all 12 added cases and the newly merged
case-insensitive cases. One existing NullType test is canceled (#4199), and one
existing optional-struct-field test is ignored.
- All 18 `ParquetEncryptionITCase` tests passed.
- Two native reader/cache tests and the upstream structural-narrowing
compatibility test passed.
- Full Maven `verify`, including packaging, formatting/style, and Apache RAT
checks, passed. Changed Rust files pass rustfmt; `git diff --check` passes.
```sh
make core
./mvnw verify -Pspark-4.1 -Dtest=none \
-Dsuites=org.apache.comet.exec.CometNativeReaderSuite,org.apache.comet.parquet.ParquetReadV1Suite
export
LD_LIBRARY_PATH="$JAVA_HOME/lib/server${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
(cd native && cargo test -p datafusion-comet parquet::parquet_exec::tests
--lib)
(cd native && cargo test -p datafusion-comet
structural_narrowing_requires_unambiguous_exact_match --lib)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
