sunchao opened a new pull request, #6036: URL: https://github.com/apache/datafusion-comet/pull/6036
## Which issue does this PR close? Follow-up to #5954, extending the same Spark-to-Comet conversion boundary to `MAP<STRING,STRING>`. ## Rationale for this change The conversion gate still rejects every map type. Consequently, an explicitly enabled RDD or Parquet input containing a string map cannot feed otherwise supported native filters and projections. For example, a row-backed input with `attributes MAP<STRING,STRING>` falls back even when `RDDScan` conversion is enabled. The existing Arrow map writer already supports this representation. Admit binary string keys and values through the conversion gate, including nullable maps and values and placement inside supported structs. Existing source opt-ins remain in effect. ## What changes are included in this PR? - Add the narrow `MapType(StringType, StringType, _)` admission case. - Document supported nullability and explicit `RDDScan` configuration. - Cover both reader paths, retained output ownership, input reuse and slices, encoding-failure cleanup, nested structs, native shuffle round trips, and disabled-conversion controls. Other collection types and non-binary collations remain rejected. ## How are these changes tested? Local validation used Linux x86_64, JDK 21, Spark 4.1.3 and Scala 2.13.17, with a freshly built native debug library. - Complete `CometArrowStreamSuite` plus the `SparkToColumnar` tests in `CometExecSuite`: **48 passed**, none failed or skipped. - Clean reactor test compilation, Spotless, Scalastyle, scoped Markdown formatting, and `git diff --check`: passed. The map integration regression checks Spark result/schema parity and the actual conversion, native filter/projection, and native shuffle operators for RDD and Parquet inputs. Source conversion disabled controls retain fallback. Spark's broader SQL suite is requested through `run-spark-4.1-tests`. AI assistance: adapted and validated with OpenAI Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
