cartesian(resourceScoresRDD).filter(arg =>
> arg._1._1 > arg._2._1).map(arg => (arg._1._1, arg._2._1, 0.8))
>
> simRDD.saveAsTextFile("/home/deployer/sim")
> }
>
> I ran the program through "java -jar myjar.jar", it crashed quickly,
eAsTextFile("/home/deployer/sim")}
I ran the program through "java -jar myjar.jar", it crashed quickly, but it
succeed when the size of the data file is small.
Thanks for your help!
qinwei
From: Andre Bois-Crettez [via Apache Spark User List]Date: 2014-04-16 17:50To:
Qin WeiSubj
Seem you have not enough memory on the spark driver. Hints below :
On 2014-04-15 12:10, Qin Wei wrote:
val resourcesRDD = jsonRDD.map(arg =>
arg.get("rid").toString.toLong).distinct
// the program crashes at this line of code
val bcResources = sc.broadcast(resourcesRDD.collect.to
