Thanks, Matei.
I will give it a try.
Larry
On Wed, Dec 17, 2014 at 1:01 PM, Matei Zaharia
wrote:
>
> I see, you may have something else configured weirdly then. You should
> look at CPU and disk utilization while your Spark job is reading from NFS
> and, if you see high CPU use, run jstack to s
I see, you may have something else configured weirdly then. You should look at
CPU and disk utilization while your Spark job is reading from NFS and, if you
see high CPU use, run jstack to see where the process is spending time. Also
make sure Spark's local work directories (spark.local.dir) are
Hi, Matei
Thanks for your response.
I tried to copy the file (1G) from NFS and took 10 seconds. The NFS mount
is a LAN environment and the NFS server is running on the same server that
Spark is running on. So basically I mount the NFS on the same bare metal
machine.
Larry
On Wed, Dec 17, 2014 a
The problem is very likely NFS, not Spark. What kind of network is it mounted
over? You can also test the performance of your NFS by copying a file from it
to a local disk or to /dev/null and seeing how many bytes per second it can
copy.
Matei
> On Dec 17, 2014, at 9:38 AM, Larryliu wrote:
>
