parthchandra commented on PR #1229: URL: https://github.com/apache/datafusion-comet/pull/1229#issuecomment-2576305496
> > Also, can we (for the moment), simply call spark cast directly in parquet support instead of duplicating code. Then, we can override the cast operations that are parquet specific. > > One challenge is that we can't just override specific casts in the parquet code because of the recursive nature of casting with complex types, so we will end up adding a lot of specializations in the spark_cast code. It seems better to keep spark cast and parquet-to-spark conversion separate. Once we have it working we can then review and move any common code out to shared code? Makes sense. Also, `parquet/read/mod.rs` has traits called `PlainDecoding` and `PlainDictDecoding` which are implemented for all the combinations of parquet type - arrow type needed by Spark. All the implementations are in `parquet/read/values.rs` and use copious quantities of unsafe code encapsulated in macros. Not sure if our Parquet-Spark cast functions can leverage the macros here but it would be nice if we could. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
