Thanks Marcelo, But I have a single server(JVM) that is creating SparkContext, are you saying Spark supports multiple SparkContext in the same JVM? Could you please clarify on this?
Thanks Anfernee On Tue, Dec 1, 2015 at 8:14 PM, Marcelo Vanzin <[email protected]> wrote: > On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <[email protected]> wrote: > > I have a long running backend server where I will create a short-lived > Spark > > job in response to each user request, base on the fact that by default > > multiple Spark Context cannot be created in the same JVM, looks like I > have > > 2 choices > > > > 2) run my jobs in yarn-cluster mode instead yarn-client > > There's nothing in yarn-client mode that prevents you from doing what > you describe. If you write some server for users to submit jobs to, it > should work whether you start the context in yarn-client or > yarn-cluster mode. It just might be harder to find out where it's > running if you do it in cluster mode. > > -- > Marcelo > -- --Anfernee
