Spark stream job is take up /TMP with 100%

[Sutanu Das]

We have a Spark steaming job and when running in LOCAL mode, it takes up /TMP 
at 100% and Fails with error below, this doesn't happen in YARN Mode but in 
YARN we have performance issues.

How can I re-direct Spark Local Subffle from /TMP to /other_filesystem_location 
(where we have lots of space) ?


Please help us, thanks again community




16/02/19 09:22:52 ERROR Executor: Exception in task 1.0 in stage 4336.0 (TID 
470495)

java.io.FileNotFoundException: 
/tmp/spark-348b864b-1f78-4109-8649-6ebf32fca3ff/blockmgr-a4aeabb9-f52a-460c-98e7-ec57b523055b/2f/temp_shuffle_900e97fe-b9d3-4d28-b59b-68954a8d81d5
 (No such file or directory)

        at java.io.FileOutputStream.open0(Native Method)

        at java.io.FileOutputStream.open(FileOutputStream.java:270)

        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)