dajiaohuang opened a new pull request, #7347: URL: https://github.com/apache/datafusion-comet/pull/7347
## Which issue does this PR close? Part of #7298. This addresses Comet cache scans; it does not change the type reporting for Spark's own `InMemoryTableScanExec` when Comet's native cache scan is disabled. ## Rationale for this change A Comet cache scan returns concrete Comet vectors, but reports no vector types to Spark's columnar-to-row code generation. Reporting the decoded types allows Spark to use the concrete classes. The serializer chooses its format from the **full cached schema**, whereas Spark's serializer `vectorTypes` hook receives only the projected attributes. A supported projected column alone cannot distinguish a Comet cache from a Spark fallback cache whose unselected columns contain an unsupported type. This change passes the full schema from the Comet scan rather than making that unsafe inference in the generic hook. ## What changes are included in this PR? - Add schema-aware vector-type reporting to `ArrowCachedBatchSerializer`, using Spark's fallback serializer when the full cached schema is unsupported. - Have `CometInMemoryTableScanExec` pass both its stored schema and its projected output to the serializer. Other serializers keep their existing hook. - Compare reported classes with actual decoded batches for primitive, decimal, date/time, binary, null, array, struct, and map columns in a reordered projection. Also check the unsupported-unselected-column and empty-projection cases. ## How are these changes tested? - Before the change, the new cache scan test failed because `vectorTypes` was `None`. - `make format` and `make` passed in an isolated Linux environment with JDK 17. - `CometInMemoryCacheSuite` passed: 83 tests, 0 failures, on Spark 4.1. - `git diff --check` passed. I did not run the upstream Spark SQL suite or other Spark profiles. The project's CI guidance calls for a Spark SQL verdict before queueing a scan/planner change; `run-spark-4.1-tests` can provide that verdict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
