Re: Spark SQL JDBC Source data skew

2015-06-25 Thread Sathish Kumaran Vairavelu
Can some one help me here? Please On Sat, Jun 20, 2015 at 9:54 AM Sathish Kumaran Vairavelu < vsathishkuma...@gmail.com> wrote: > Hi, > > In Spark SQL JDBC data source there is an option to specify upper/lower > bound and num of partitions. How Spark handles data distribution, if we do > not give

Spark SQL JDBC Source data skew

2015-06-20 Thread Sathish Kumaran Vairavelu
Hi, In Spark SQL JDBC data source there is an option to specify upper/lower bound and num of partitions. How Spark handles data distribution, if we do not give the upper/lower/num of parititons ? Will all data from the external data source skewed up in one executor? In many situations, we do not