sunchao commented on code in PR #6107:
URL: https://github.com/apache/datafusion-comet/pull/6107#discussion_r4393666379
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -1928,12 +1928,8 @@ object CometExplodeExec extends
CometOperatorSerde[GenerateExec] {
return Unsupported(Some(s"Unsupported generator:
${op.generator.nodeName}"))
}
op.generator.children.head.dataType match {
- case _: ArrayType =>
+ case _: ArrayType | _: MapType =>
Review Comment:
[P2] Bound one map's expansion before repeating carried strings
Could we keep this shape on Spark until the native operator can split an
individual map before repeating its carried columns? A single input row with a
2,049-entry `map<int,int>` and a 1 MiB `STRING` payload is enough to fail.
`PendingInput::next_chunk_rows` never splits one input row, so `repeat_arrs`
calls Arrow `take` for the entire map before `BatchSplitStream` runs. The
repeated string reaches the 32-bit offset limit even with a configured batch
size of 256.
I reproduced `Arrow error: Offset overflow error: 2147483648` on the first
output poll using this head's unchanged `MapEntriesExpr` and `ExplodeExec`,
Arrow 59.3.1 and DataFusion 55.2.0. A small control passes. With the same input
stored in Parquet, plain Spark 4.1.3 successfully computes `SELECT
sum(CAST(length(concat(payload, CAST(key AS STRING))) AS BIGINT)) FROM wide_map
LATERAL VIEW explode(m) e AS key, value`. Its executed plan keeps `payload`
through `Generate` and consumes the rows in the aggregate.
The base and `branch-1.2` leave map generators on Spark, so this admission
change exposes a query failure even though array explode already has the
underlying limitation. Please add a wide carried-column regression and split
within a map before materialization, or retain a safe fallback for this shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
