Re: external shuffle service in mesos

2018-01-23 Thread igor.berman
Hi Susan, yes, agree with you regarding resource accounting. Imho, in this case shuffle service must run on node no matter what resources are available(same as we don't account for resources that "system" takes - mesos agent, OS itself and any other process that is running on same machine) One add

Re: external shuffle service in mesos

2018-01-22 Thread Susan X. Huynh
Hi Igor, You made a good point about the tradeoffs. I think the main thing you would get with Marathon is the accounting for resources (the memory and cpus specified in the config file). That allows Mesos to manage the resources properly. I don't think the other tools mentioned would reserve resou

Re: external shuffle service in mesos

2018-01-21 Thread igor.berman
Hi Susan In general I can get what I need without Marathon, with configuring external-shuffle-service with puppet/ansible/chef + maybe some alerts for checks. I mean in companies that don't have strong Devops teams and want to install services as simple as possible just by config - Marathon might

Re: external shuffle service in mesos

2018-01-20 Thread Susan X. Huynh
Hi Igor, The best way I know of is with Marathon. * Placement constraint: you could combine constraints in Marathon. Like: "constraints": [ ["hostname", "UNIQUE"], ["hostname", "LIKE", "host1|host2|host3"] ] https://groups.google.com/forum/#!topic/marathon-framework/hfLUw3TIw2I *