[
https://issues.apache.org/jira/browse/SPARK-50686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-50686:
-----------------------------------
Labels: memory-saving optimization pull-request-available (was:
memory-saving optimization)
> Memory footprint reduction in the fallback from hash aggregation to the sort
> ----------------------------------------------------------------------------
>
> Key: SPARK-50686
> URL: https://issues.apache.org/jira/browse/SPARK-50686
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.4
> Reporter: Artem Kupchinskiy
> Priority: Major
> Labels: memory-saving, optimization, pull-request-available
>
> Currently, under the memory pressure, spark uses fallback from a hash-based
> aggregation to the sort-based one. The logic reuses internal hash map
> components for sorting to avoid unnecessary allocation. Yet, there is still
> [a
> spot|https://github.com/apache/spark/blob/b309db092cbb67fedb7cec9ab0ebcd07d8d95b04/sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java#L118]
> when a new allocation is initiated. Basically, every time we have duplicates
> in key aggregates we can't reuse the internal array and have to allocate a
> new one. The natural approach here would be invalidation of the redundant old
> array before a new allocation. That would reduce the overall memory footprint
> and decrease the chance of the oom error and subsequent task failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
