When running Spark in Mesos cluster mode, the driver program runs in one of the cluster nodes, like the other Spark processes that are spawned. You won't need a special node for this purpose. I'm not very familiar with Chronos, but its UI or the regular Mesos UI should show you where the driver is running, then you can use the Spark web UI on that machine to see what the Spark job is doing.
dean Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Fri, Jul 24, 2015 at 4:47 PM, boci <boci.b...@gmail.com> wrote: > Thanks, but something is not clear... > I have the mesos cluster. > - I want to submit my application and scheduled with chronos. > - For cluster mode I need a dispatcher, this is another container (machine > in the real world)? What will this do? It's needed when I using chronos? > - How can I access to my spark job from chronos? > > I think submit in client mode is not fit to my condition, that's right? > > Thanks > b0c1 > > > ---------------------------------------------------------------------------------------------------------------------------------- > Skype: boci13, Hangout: boci.b...@gmail.com > > On Wed, Jul 22, 2015 at 4:51 PM, Dean Wampler <deanwamp...@gmail.com> > wrote: > >> This page, http://spark.apache.org/docs/latest/running-on-mesos.html, >> covers many of these questions. If you submit a job with the option >> "--supervise", it will be restarted if it fails. >> >> You can use Chronos for scheduling. You can create a single streaming job >> with a 10 minute batch interval, if that works for your every 10-min. need. >> >> Dean Wampler, Ph.D. >> Author: Programming Scala, 2nd Edition >> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >> Typesafe <http://typesafe.com> >> @deanwampler <http://twitter.com/deanwampler> >> http://polyglotprogramming.com >> >> On Wed, Jul 22, 2015 at 3:53 AM, boci <boci.b...@gmail.com> wrote: >> >>> Hi guys! >>> >>> I'm a new in mesos. I have two spark application (one streaming and one >>> batch). I want to run both app in mesos cluster. Now for testing I want to >>> run in docker container so I started a simple redjack/mesos-master, but I >>> think a lot of think unclear for me (both mesos and spark-mesos). >>> >>> If I have a mesos cluster (for testing it will be some docker container) >>> i need a separate machine (container) to run my spark job? Or can I submit >>> the cluster and schedule (chronos or I dunno)? >>> How can I run the streaming job? What happened if the "controller" died? >>> Or if I call spark-submit with master=mesos my application started and I >>> can forget? How can I run in every 10 min without submit in every 10 min? >>> How can I run my streaming app in HA mode? >>> >>> Thanks >>> >>> b0c1 >>> >>> >>> ---------------------------------------------------------------------------------------------------------------------------------- >>> Skype: boci13, Hangout: boci.b...@gmail.com >>> >> >> >
