Hi, wanted to get some advice regarding managing external shuffle service in mesos environments
In spark documentation the Marathon is mentioned, however there is very limited documentation. I've tried to search for some documentation and it's seems not too difficult to configure it under Marathon(e.g. https://github.com/NBCUAS/dcos-spark-shuffle-service/blob/master/marathon/mesos-shuffle-service.json), however I see few problems: There is no clear way to deploy some application in mesos on every node see https://jira.mesosphere.com/browse/MARATHON-3791 * it's not possible to guarantee on which nodes shuffle service application will be placed(it's possible to guarantee with mesos unique constrain that only 1 shuffle service instance will be placed on some node) * cluster that has dynamic nodes joining/leaving - the config of shuffle service must be adjusted(specifically number of instances config) So any production ops advices will be welcome Igor -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
