I see, you may have something else configured weirdly then. You should look at CPU and disk utilization while your Spark job is reading from NFS and, if you see high CPU use, run jstack to see where the process is spending time. Also make sure Spark's local work directories (spark.local.dir) are not on NFS. They shouldn't be though, that should be /tmp.
Matei > On Dec 17, 2014, at 11:56 AM, Larry Liu <larryli...@gmail.com> wrote: > > Hi, Matei > > Thanks for your response. > > I tried to copy the file (1G) from NFS and took 10 seconds. The NFS mount is > a LAN environment and the NFS server is running on the same server that Spark > is running on. So basically I mount the NFS on the same bare metal machine. > > Larry > > On Wed, Dec 17, 2014 at 11:42 AM, Matei Zaharia <matei.zaha...@gmail.com > <mailto:matei.zaha...@gmail.com>> wrote: > The problem is very likely NFS, not Spark. What kind of network is it mounted > over? You can also test the performance of your NFS by copying a file from it > to a local disk or to /dev/null and seeing how many bytes per second it can > copy. > > Matei > > > On Dec 17, 2014, at 9:38 AM, Larryliu <larryli...@gmail.com > > <mailto:larryli...@gmail.com>> wrote: > > > > A wordcounting job for about 1G text file takes 1 hour while input from a > > NFS > > mount. The same job took 30 seconds while input from local file system. > > > > Is there any tuning required for a NFS mount input? > > > > Thanks > > > > Larry > > > > > > > > -- > > View this message in context: > > http://apache-spark-user-list.1001560.n3.nabble.com/wordcount-job-slow-while-input-from-NFS-mount-tp20747.html > > > > <http://apache-spark-user-list.1001560.n3.nabble.com/wordcount-job-slow-while-input-from-NFS-mount-tp20747.html> > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > <mailto:user-unsubscr...@spark.apache.org> > > For additional commands, e-mail: user-h...@spark.apache.org > > <mailto:user-h...@spark.apache.org> > > >
