Got any solution for this?
On Tuesday 04 October 2016 05:37 AM, Punit Naik wrote:
Hi All
I am trying to run a program for a large dataset (~ 1TB). I have
already tested the code for low size of data and it works fine. But
what I noticed is that he job fails if the size of input is large. It
was giving me errors regarding akkka actor disassociation which I
fixed by increasing the timeouts.
But now I am getting errors like "execuyor lost" and "executor lost
failure" which I can't seem to figure out. These are my current set of
configs:
|--conf ||"spark.network.timeout=30000"|
|--conf ||"spark.core.connection.ack.wait.timeout=30000"|
|--conf ||"spark.akka.timeout=30000"|
|--conf ||"spark.akka.askTimeout=30000"|
|--conf ||"spark.akka.frameSize=1000"|
|--conf ||"spark.storage.blockManagerSlaveTimeoutMs=600000"|
|--conf ||"spark.network.timeout=600"|
|--conf ||"spark.shuffle.memoryFraction=0.8"|
|--conf ||"spark.driver.maxResultSize=16g"|
|--conf ||"spark.driver.cores=10"|
|--conf ||"spark.driver.memory=10g"|
|Can anyone tell me any more configs to circumvent this "executor
lost" and "executor lost failure" error?|
--
Thank You
Regards
Punit Naik
