ok. after reading some documentation, it would appear the issue is the default
number of partitions for a join (200).
After doing something like the following, I was able to change the value.
From: Darin McBeath
To: User
Sent: Wednesday, October 29, 2014 1:55 PM
Subject: Spark SQL and
Sorry, hit the send key a bitt too early.
Anyway, this is the code I set.
sqlContext.sql("set spark.sql.shuffle.partitions=10");
From: Darin McBeath
To: Darin McBeath ; User
Sent: Wednesday, October 29, 2014 2:47 PM
Subject: Re: Spark SQL and confused about number of partit
