[
https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DjvuLee updated SPARK-21082:
----------------------------
Description: Spark Scheduler do not consider the memory usage during
dispatch tasks, this can lead to Executor OOM if the RDD is cached sometimes,
because Spark can not estimate the memory usage enough well(especially when the
RDD type is not flatten). We can offer a configuration for user to decide
whether scheduler will consider the memory usage to relief the OOM. (was: When
we cache the )
> Consider Executor's memory usage when scheduling task
> ------------------------------------------------------
>
> Key: SPARK-21082
> URL: https://issues.apache.org/jira/browse/SPARK-21082
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Affects Versions: 2.2.1
> Reporter: DjvuLee
>
> Spark Scheduler do not consider the memory usage during dispatch tasks, this
> can lead to Executor OOM if the RDD is cached sometimes, because Spark can
> not estimate the memory usage enough well(especially when the RDD type is not
> flatten). We can offer a configuration for user to decide whether scheduler
> will consider the memory usage to relief the OOM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
