Re: Spark driver reducing total executors count even when Dynamic Allocation is disabled.

h AM, AM will try to cancel some pending container > requests when current expected executor number is less. The actual > container killing command is issued by "doRequestTotalExecutors". > > Not sure what is your actual problem? is it unexpected? > > Thanks > Sais

Spark driver reducing total executors count even when Dynamic Allocation is disabled.

Hey all, Thanks in advance. I ran into a situation where spark driver reduced the total executors count for my job even with dynamic allocation disabled, and caused the job to hang for ever. Setup: Spark-1.3.1 on hadoop-yarn-2.4.0 cluster. All servers in cluster running Linux version 2.6.32.

Re: Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

hey all, Thanks in advance. I am facing this issue in production, where due to increased container request RM is reserving memory and hampering cluster utilization. Thus the fix needs to be patched on spark 1.2. Has any one looked in the removeContainerRequest part for allocated containers in sp

Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

This is because Yarn's AM client does not remove fulfilled container request from its MAP until the application's AM specifically calls removeContainerRequest for fulfilled container requests. Spark-1.2 : Spark's yarn AM does not call removeContainerRequest for fulfilled container request. Spark-