You can set spark.local.dir to put this data somewhere other than /tmp if /tmp is full. Actually it’s recommended to have multiple local disks and set to to a comma-separated list of directories, one per disk.
Matei On Mar 23, 2014, at 3:35 PM, Aaron Davidson <ilike...@gmail.com> wrote: > On some systems, /tmp/ is an in-memory tmpfs file system, with its own size > limit. It's possible that this limit has been exceeded. You might try running > the "df" command to check to free space of "/tmp" or root if tmp isn't listed. > > 3 GB also seems pretty low for the remaining free space of a disk. If your > disk size is in the TB range, it's possible that the last couple GB have > issues when being allocated due to fragmentation or reclamation policies. > > > On Sun, Mar 23, 2014 at 3:06 PM, Ognen Duzlevski <og...@nengoiksvelzud.com> > wrote: > Hello, > > I have a weird error showing up when I run a job on my Spark cluster. The > version of spark is 0.9 and I have 3+ GB free on the disk when this error > shows up. Any ideas what I should be looking for? > > [error] (run-main-0) org.apache.spark.SparkException: Job aborted: Task > 167.0:3 failed 4 times (most recent failure: Exception failure: > java.io.FileNotFoundException: > /tmp/spark-local-20140323214638-72df/31/shuffle_31_3_127 (No space left on > device)) > org.apache.spark.SparkException: Job aborted: Task 167.0:3 failed 4 times > (most recent failure: Exception failure: java.io.FileNotFoundException: > /tmp/spark-local-20140323214638-72df/31/shuffle_31_3_127 (No space left on > device)) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619) > at scala.Option.foreach(Option.scala:236) > at > org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207) > > Thanks! > Ognen >
