Which Spark version are you using? What are you trying to do exactly and what is the input data? As far as I know, akka has been dropped in recent Spark versions.
> On 30 Jan 2017, at 00:44, aravasai <arava...@gmail.com> wrote: > > I have a spark job running on 2 terabytes of data which creates more than > 30,000 partitions. As a result, the spark job fails with the error > "Map output statuses were 170415722 bytes which exceeds spark.akka.frameSize > 52428800 bytes" (For 1 TB data) > However, when I increase the akka.frame.size to around 500 MB, the job hangs > with no further progress. > > So, what is the ideal or maximum limit that i can assign akka.frame.size so > that I do not get the error of map output statuses exceeding limit for large > chunks of data ? > > Is coalescing the data into smaller number of partitions the only solution > to this problem? Is there any better way than coalescing many intermediate > rdd's in program ? > > My driver memory: 10G > Executor memory: 36G > Executor memory overhead : 3G > > > > > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Maximum-limit-for-akka-frame-size-be-greater-than-500-MB-tp20793.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
