You can set custom per-instance-group configurations (e.g.,
["classification":"yarn-site",properties:{"yarn.nodemanager.labels":"SPARKAM"}])
using the Configurations parameter of
http://docs.aws.amazon.com/ElasticMapReduce/latest/API/API_InstanceGroupConfig.html.
Unfortunately, it's not currently possible to specify per-instance-group
configurations via the CLI though, only cluster wide configurations.~ Jonathan On Tue, Feb 9, 2016 at 12:36 PM Alexander Pivovarov <apivova...@gmail.com> wrote: > Thanks Jonathan > > Actually I'd like to use maximizeResourceAllocation. > > Ideally for me would be to add new instance group having single small box > labelled as AM > I'm not sure "aws emr create-cluster" supports setting custom LABELS , the > only settings awailable are: > > InstanceCount=1,BidPrice=0.5,Name=sparkAM,InstanceGroupType=TASK,InstanceType=m3.xlarge > > > How can I specify yarn label AM for that box? > > > > On Tue, Feb 9, 2016 at 12:16 PM, Jonathan Kelly <jonathaka...@gmail.com> > wrote: > >> Interesting, I was not aware of spark.yarn.am.nodeLabelExpression. >> >> We do use YARN labels on EMR; each node is automatically labeled with its >> type (MASTER, CORE, or TASK). And we do >> set yarn.app.mapreduce.am.labels=CORE in yarn-site.xml, but we do not set >> spark.yarn.am.nodeLabelExpression. >> >> Does Spark somehow not actually honor this? It seems weird that Spark >> would have its own similar-sounding property >> (spark.yarn.am.nodeLabelExpression). If spark.yarn.am.nodeLabelExpression >> is used and yarn.app.mapreduce.am.labels ignored, I could be wrong about >> Spark AMs only running on CORE instances in EMR. >> >> I'm guessing though that spark.yarn.am.nodeLabelExpression would simply >> override yarn.app.mapreduce.am.labels, so yarn.app.mapreduce.am.labels >> would be treated as a default when it is set and >> spark.yarn.am.nodeLabelExpression is not. Is that correct? >> >> In short, Alex, you should not need to set any of the label-related >> properties yourself if you do what I suggested regarding using small CORE >> instances and large TASK instances. But if you want to do something >> different, it would also be possible to add a TASK instance group with >> small nodes and configured with some new label. Then you could set >> spark.yarn.am.nodeLabelExpression to that label. >> >> Thanks, Marcelo, for pointing out spark.yarn.am.nodeLabelExpression! >> >> ~ Jonathan >> >> On Tue, Feb 9, 2016 at 9:54 AM Marcelo Vanzin <van...@cloudera.com> >> wrote: >> >>> You should be able to use spark.yarn.am.nodeLabelExpression if your >>> version of YARN supports node labels (and you've added a label to the >>> node where you want the AM to run). >>> >>> On Tue, Feb 9, 2016 at 9:51 AM, Alexander Pivovarov >>> <apivova...@gmail.com> wrote: >>> > Am container starts first and yarn selects random computer to run it. >>> > >>> > Is it possible to configure yarn so that it selects small computer for >>> am >>> > container. >>> > >>> > On Feb 9, 2016 12:40 AM, "Sean Owen" <so...@cloudera.com> wrote: >>> >> >>> >> If it's too small to run an executor, I'd think it would be chosen for >>> >> the AM as the only way to satisfy the request. >>> >> >>> >> On Tue, Feb 9, 2016 at 8:35 AM, Alexander Pivovarov >>> >> <apivova...@gmail.com> wrote: >>> >> > If I add additional small box to the cluster can I configure yarn to >>> >> > select >>> >> > small box to run am container? >>> >> > >>> >> > >>> >> > On Mon, Feb 8, 2016 at 10:53 PM, Sean Owen <so...@cloudera.com> >>> wrote: >>> >> >> >>> >> >> Typically YARN is there because you're mediating resource requests >>> >> >> from things besides Spark, so yeah using every bit of the cluster >>> is a >>> >> >> little bit of a corner case. There's not a good answer if all your >>> >> >> nodes are the same size. >>> >> >> >>> >> >> I think you can let YARN over-commit RAM though, and allocate more >>> >> >> memory than it actually has. It may be beneficial to let them all >>> >> >> think they have an extra GB, and let one node running the AM >>> >> >> technically be overcommitted, a state which won't hurt at all >>> unless >>> >> >> you're really really tight on memory, in which case something might >>> >> >> get killed. >>> >> >> >>> >> >> On Tue, Feb 9, 2016 at 6:49 AM, Jonathan Kelly < >>> jonathaka...@gmail.com> >>> >> >> wrote: >>> >> >> > Alex, >>> >> >> > >>> >> >> > That's a very good question that I've been trying to answer >>> myself >>> >> >> > recently >>> >> >> > too. Since you've mentioned before that you're using EMR, I >>> assume >>> >> >> > you're >>> >> >> > asking this because you've noticed this behavior on emr-4.3.0. >>> >> >> > >>> >> >> > In this release, we made some changes to the >>> >> >> > maximizeResourceAllocation >>> >> >> > (which you may or may not be using, but either way this issue is >>> >> >> > present), >>> >> >> > including the accidental inclusion of somewhat of a bug that >>> makes it >>> >> >> > not >>> >> >> > reserve any space for the AM, which ultimately results in one of >>> the >>> >> >> > nodes >>> >> >> > being utilized only by the AM and not an executor. >>> >> >> > >>> >> >> > However, as you point out, the only viable fix seems to be to >>> reserve >>> >> >> > enough >>> >> >> > memory for the AM on *every single node*, which in some cases >>> might >>> >> >> > actually >>> >> >> > be worse than wasting a lot of memory on a single node. >>> >> >> > >>> >> >> > So yeah, I also don't like either option. Is this just the price >>> you >>> >> >> > pay >>> >> >> > for >>> >> >> > running on YARN? >>> >> >> > >>> >> >> > >>> >> >> > ~ Jonathan >>> >> >> > >>> >> >> > On Mon, Feb 8, 2016 at 9:03 PM Alexander Pivovarov >>> >> >> > <apivova...@gmail.com> >>> >> >> > wrote: >>> >> >> >> >>> >> >> >> Lets say that yarn has 53GB memory available on each slave >>> >> >> >> >>> >> >> >> spark.am container needs 896MB. (512 + 384) >>> >> >> >> >>> >> >> >> I see two options to configure spark: >>> >> >> >> >>> >> >> >> 1. configure spark executors to use 52GB and leave 1 GB on each >>> box. >>> >> >> >> So, >>> >> >> >> some box will also run am container. So, 1GB memory will not be >>> used >>> >> >> >> on >>> >> >> >> all >>> >> >> >> slaves but one. >>> >> >> >> >>> >> >> >> 2. configure spark to use all 53GB and add additional 53GB box >>> which >>> >> >> >> will >>> >> >> >> run only am container. So, 52GB on this additional box will do >>> >> >> >> nothing >>> >> >> >> >>> >> >> >> I do not like both options. Is there a better way to configure >>> >> >> >> yarn/spark? >>> >> >> >> >>> >> >> >> >>> >> >> >> Alex >>> >> > >>> >> > >>> >>> >>> >>> -- >>> Marcelo >>> >> >
