chenhao-db opened a new pull request, #50990: URL: https://github.com/apache/spark/pull/50990
### What changes were proposed in this pull request? In the vectorized Parquet reader, there are two classes to resolve the Parquet schema when reading a Parquet file: - `ParquetReadSupport`: it clips the Parquet schema to only include the necessary part used by the Spark requested schema. The matching considers both field name and ID. - `ParquetToSparkSchemaConverter`: it resolves the Parquet schema to a Spark type by connecting it to the Spark requested schema. The matching only considers field name. When the field ID matches but field name doesn't, the first step will clip the Parquet schema to the same structure as the Spark requested schema as expected. In the second step, the Parquet type cannot be connected to a Spark type in the requested schema, and it will be inferred as a Spark type. It will usually work as expected if the inferred type is the same as the requested type. But it is possible that they are different and the read is still valid. For example, if the Parquet type is `int` and the Spark type is `long`. In this case, the vectorized Parquet reader will produce `int` data in column vectors, which will be interpreted as `long` data by subsequent operations. This can happen in real user cases if an Iceberg table with both rename and change column type (int -> long) is converted into a Delta table. This situation may be very rare, though. This PR fixes by bug by matching field ID in `ParquetToSparkSchemaConverter` when the name cannot be matched. ### Why are the changes needed? It fixes a correctness issue. ### Does this PR introduce _any_ user-facing change? Yes, as stated above. ### How was this patch tested? Unit test. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
