Hi All, Does number of worker threads bear any relationship to setting executor memory ?. I have a 16 GB RAM, with an 8-core processor. I had set SPARK_MEM to 12g and was running locally with default 1 thread. So this means there can be maximum one executor in one node scheduled at any point of time. If I increase the number of worker threads to say 4, do I need to reduce SPARK_MEM to 3g or I need not ???
Is there any performance difference between running in interactive spark shell to non-interactive standalone spark apart from building time(sbt…) ??? THANKS !!
