I thought I answered this ... you can easily accomplish this with YARN by just telling YARN how much memory / CPU each machine has. This can be configured in groups too rather than per machine. I don't think you actually want differently-sized executors, and so don't need ratios. But you can have differently-sized containers which can fit different numbers of executors as appropriate.
On Wed, Sep 17, 2014 at 8:35 AM, Victor Tso-Guillen <[email protected]> wrote: > I'm supposing that there's no good solution to having heterogenous hardware > in a cluster. What are the prospects of having something like this in the > future? Am I missing an architectural detail that precludes this > possibility? > > Thanks, > Victor > > On Fri, Sep 12, 2014 at 12:10 PM, Victor Tso-Guillen <[email protected]> > wrote: >> >> Ping... >> >> On Thu, Sep 11, 2014 at 5:44 PM, Victor Tso-Guillen <[email protected]> >> wrote: >>> >>> So I have a bunch of hardware with different core and memory setups. Is >>> there a way to do one of the following: >>> >>> 1. Express a ratio of cores to memory to retain. The spark worker config >>> would represent all of the cores and all of the memory usable for any >>> application, and the application would take a fraction that sustains the >>> ratio. Say I have 4 cores and 20G of RAM. I'd like it to have the worker >>> take 4/20 and the executor take 5 G for each of the 4 cores, thus maxing >>> both out. If there were only 16G with the same ratio requirement, it would >>> only take 3 cores and 12G in a single executor and leave the rest. >>> >>> 2. Have the executor take whole number ratios of what it needs. Say it is >>> configured for 2/8G and the worker has 4/20. So we can give the executor >>> 2/8G (which is true now) or we can instead give it 4/16G, maxing out one of >>> the two parameters. >>> >>> Either way would allow me to get my heterogenous hardware all >>> participating in the work of my spark cluster, presumably without >>> endangering spark's assumption of homogenous execution environments in the >>> dimensions of memory and cores. If there's any way to do this, please >>> enlighten me. >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
