We have a Spark steaming job and when running in LOCAL mode, it takes up /TMP at 100% and Fails with error below, this doesn't happen in YARN Mode but in YARN we have performance issues.
How can I re-direct Spark Local Subffle from /TMP to /other_filesystem_location (where we have lots of space) ? Please help us, thanks again community 16/02/19 09:22:52 ERROR Executor: Exception in task 1.0 in stage 4336.0 (TID 470495) java.io.FileNotFoundException: /tmp/spark-348b864b-1f78-4109-8649-6ebf32fca3ff/blockmgr-a4aeabb9-f52a-460c-98e7-ec57b523055b/2f/temp_shuffle_900e97fe-b9d3-4d28-b59b-68954a8d81d5 (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213)