Re: Changing Spark configuration midway through application.

2016-08-10 Thread Andrew Ehrlich
If you're changing properties for the SparkContext, then I believe you will have to start a new SparkContext with the new properties. On Wed, Aug 10, 2016 at 8:47 AM, Jestin Ma wrote: > If I run an application, for example with 3 joins: > > [join 1] > [join 2] > [join 3] > > [final join and save

Changing Spark configuration midway through application.

2016-08-10 Thread Jestin Ma
If I run an application, for example with 3 joins: [join 1] [join 2] [join 3] [final join and save to disk] Could I change Spark properties in between each join? [join 1] [change properties] [join 2] [change properties] ... Or would I have to create a separate application with different proper