[
https://issues.apache.org/jira/browse/SPARK-59375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59375:
-----------------------------------
Labels: pull-request-available (was: )
> flatten raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in
> codegen
> -------------------------------------------------------------------------------------
>
> Key: SPARK-59375
> URL: https://issues.apache.org/jira/browse/SPARK-59375
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 4.1.3, 4.0.4
> Reporter: sepuri sai krishna
> Priority: Minor
> Labels: pull-request-available
>
> {{Flatten.genCodeForNumberOfElements}} does not check the result element count
> against {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}}, while the interpreted
> path
> in {{Flatten.nullSafeEval}} does. The two paths therefore report different
> error
> conditions for the same input: the interpreted path raises
> {{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}}, and codegen reaches
> {{ArrayData.allocateArrayData}} and raises the internal
> {{_LEGACY_ERROR_TEMP_2176}}.
> Same pattern as SPARK-58631, which fixed it for {{array_repeat}}.
> Priority is Minor because the limit is not cheaply reachable here: {{flatten}}
> needs its input arrays to hold that many elements for real, so an executor
> will
> normally run out of memory first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
