Hi, Martin: I think a better solution would be to set the number of cores of each container equals to that of a physical server if this mesos cluster is dedicated to your flink cluster.
On Mon, Sep 17, 2018 at 5:28 AM Martin Eden <martineden...@gmail.com> wrote: > Hi Till, > > I was able to use mesos.constraints.hard.hostattribute to run all task > managers on a particular host in my cluster. > > However, after looking a bit at the code, I'm not sure we can use > mesos.constraints.hard.hostattribute for load balancing Flink task managers > evenly across hosts in a Mesos cluster. > > This is because under the hood it uses the fenzo host attribute value > constraint while we would need the fenzo balanced host attribute constraint. > > The LaunchableMesosWorker sets the constraints via > the com.netflix.fenzo.TaskRequest and all of these hard constraints must be > satisfied by a host for the task scheduler to assign this task to that > host. Since the current implementation always return the static constraint > value configured i.e. what is after ":", see > org.apache.flink.mesos.runtime.clusterframework.MesosTaskManagerParameters#addHostAttrValueConstraint, > I don't see how we can use it to load balance unless the constraint value > would be dynamic based on the some property of the mesos task request. > > Am I correct in my assumptions? > > Any other way of load balancing? > Maybe by not even using the DCOS Flink package (mesos flink framework) at > all? > Any plans to add support for the fenzo balanced host attribute constraint? > > Thanks, > M > > > > > On Fri, Sep 14, 2018 at 5:46 PM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Martin, >> >> Flink supports the mesos.constraints.hard.hostattribute to specify task >> constraints based on agent attributes [1]. I think you could use them to >> control the task placement. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#mesos-constraints-hard-hostattribute >> >> Cheers, >> Till >> >> On Fri, Sep 14, 2018 at 3:08 PM Martin Eden <martineden...@gmail.com> >> wrote: >> >>> Thanks Vino! >>> >>> On Fri, Sep 14, 2018 at 3:37 AM vino yang <yanghua1...@gmail.com> wrote: >>> >>>> Hi Martin, >>>> >>>> Till has done most of the work of Flink on Mesos. Ping Till for you. >>>> >>>> Thanks, vino. >>>> >>>> Martin Eden <martineden...@gmail.com> 于2018年9月12日周三 下午11:21写道： >>>> >>>>> Hi all, >>>>> >>>>> We're using Flink 1.3.2 with DCOS / Mesos. >>>>> >>>>> We have a 3 node cluster and are running the Flink DCOS package (Flink >>>>> Mesos framework) configured with 3 Task Managers. >>>>> >>>>> Our goal is to run each of them on separate hosts for better load >>>>> balancing but it seems the task managers end up running on the same host. >>>>> >>>>> Looked around the docs and DCOS Flink package but could not find any >>>>> placement policy or anything of the sorts. >>>>> >>>>> Is there anything like that? >>>>> >>>>> We are also planning to upgrade to the latest Flink version. Is >>>>> something like that supported in this newer version? >>>>> >>>>> Thanks, >>>>> M >>>>> >>>> -- Liu, Renjie Software Engineer, MVAD
