On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <[email protected]> wrote: > I have a long running backend server where I will create a short-lived Spark > job in response to each user request, base on the fact that by default > multiple Spark Context cannot be created in the same JVM, looks like I have > 2 choices > > 2) run my jobs in yarn-cluster mode instead yarn-client
There's nothing in yarn-client mode that prevents you from doing what you describe. If you write some server for users to submit jobs to, it should work whether you start the context in yarn-client or yarn-cluster mode. It just might be harder to find out where it's running if you do it in cluster mode. -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
