Are you using Yarn to run spark jobs only ?. Are you configuring spark properties in spark-submit parameters? . If so did you try with --no - of - executors x*53 (where x is no of nodes ) --spark executor-memory 1g --spark-driver-memory 1g.
You might see yarn allocating resources to all executors except one because driver draws that memory . Trade off is that if there were not much data to process then many executors may run empty wasting up resources . In that case probably you might need to settle down with dynamic allocation enabled . Sent from Samsung Mobile. Sent from Samsung Mobile. <div>-------- Original message --------</div><div>From: Alexander Pivovarov <apivova...@gmail.com> </div><div>Date:09/02/2016 10:33 (GMT+05:30) </div><div>To: dev@spark.apache.org </div><div>Cc: </div><div>Subject: spark on yarn wastes one box (or 1 GB on each box) for am container </div><div> </div>Lets say that yarn has 53GB memory available on each slave spark.am container needs 896MB. (512 + 384) I see two options to configure spark: 1. configure spark executors to use 52GB and leave 1 GB on each box. So, some box will also run am container. So, 1GB memory will not be used on all slaves but one. 2. configure spark to use all 53GB and add additional 53GB box which will run only am container. So, 52GB on this additional box will do nothing I do not like both options. Is there a better way to configure yarn/spark? Alex
