Re: Configuration for unit testing and sql.shuffle.partitions

you can create a Super class "FunSuiteWithSparkContext" that's going to create a Spark sessions, Spark context, and SQLContext with all the desired properties. Then you add the class to all the relevant test suites, and that's pretty much it. The other option can be is to pass it as a VM parameter

Re: Configuration for unit testing and sql.shuffle.partitions

How are you specifying it, as an option to spark-submit ? On Sat, Sep 16, 2017 at 12:26 PM, Akhil Das wrote: > spark.sql.shuffle.partitions is still used I believe. I can see it in the > code >

Re: Configuration for unit testing and sql.shuffle.partitions

spark.sql.shuffle.partitions is still used I believe. I can see it in the code and in the documentation page