Hi
We are using Spark with Cassandra. While using rdd.saveAsTextFile("/tmp/dr"),
we are getting following error when we run the application with root access.
Spark is able to create two level of directories but fails after that with
Exception:
16/03/01 22:59:48 WARN TaskSetManager: Lost task 73.3 in stage 0.0 (TID 144,
host1): java.io.IOException: Mkdirs failed to create
file:/tmp/dr/_temporary/0/_temporary/attempt_201603012259_0000_m_000073_144
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:799)
at
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1056)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1047)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Permissions on /tmp:
chmod -R 777 /tmp has been executed and permissions look like:
drwxrwxrwx. 31 root root 1.2K Mar 1 22:54 tmp
Forgive me for raising this question in Cassandra Mailing list. I think Spark &
Cassandra user base is overlapping, so I expected help here.
I am not yet part of Spark mailing list.
Thanks
Anuj
