cartesian(resourceScoresRDD).filter(arg =>
> arg._1._1 > arg._2._1).map(arg => (arg._1._1, arg._2._1, 0.8))
>
> simRDD.saveAsTextFile("/home/deployer/sim")
> }
>
> I ran the program through "java -jar myjar.jar", it crashed quickly,
eAsTextFile("/home/deployer/sim")}
I ran the program through "java -jar myjar.jar", it crashed quickly, but it
succeed when the size of the data file is small.
Thanks for your help!
qinwei
From: Andre Bois-Crettez [via Apache Spark User List]Date: 2014-04-16 17:50To:
Qin WeiSubj
Seem you have not enough memory on the spark driver. Hints below :
On 2014-04-15 12:10, Qin Wei wrote:
val resourcesRDD = jsonRDD.map(arg =>
arg.get("rid").toString.toLong).distinct
// the program crashes at this line of code
val bcResources = sc.broadcast(resourcesRDD.collect.to
park.akka.logLifecycleEvents=true -Djava.library.path= -Xms512m -Xmx512m
org.apache.spark.deploy.worker.Worker spark://192.168.2.184:7077
Is there anybody who can help me? Thanks very much!!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-program-thows-OutOfMemoryError-tp4268.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
