My application is used for LDA(a topic model, with gibbs sampling), it's hard for me to explain LDA, so you need to search it on google if any. I did increase spark.akka.frameSize to 1GB(even 5GB), both in master/workers's spark-defaults.conf and SparkConf, but it has no effect at all. I'm not sure what leads to disconnection between master/executors and driver, maybe the CPU load is too high. Doing gibbs-sampling on 800,000,000 words takes 300s for each iteration on 30 cores. I will try what you have suggested, Thanks a lot!
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/why-does-driver-connects-to-master-fail-tp16758p16898.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
