andygrove commented on PR #7528:
URL: 
https://github.com/apache/datafusion-comet/pull/7528#issuecomment-6949583853

   I think the Spark 3.5 failure is Spark's own reader rather than Comet. The 
fallback works, which is why the error comes from Spark's vectorized reader. 
But Spark 3.5 can't read an INT32 column as `decimal(12,-2)` at all, because 
widening an integer to a decimal on read arrived with Parquet type widening in 
Spark 4.0. So there's no Spark answer to compare against on 3.5.
   
   Could the test check the fallback on every version and only compare answers 
where Spark can read it? For example, wrap the 
`checkSparkAnswerAndFallbackReason` call in `if (isSpark40Plus)`, and on 3.5 
assert that both Spark and Comet raise and that the plan falls back with the 
same reason.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to