Depending on the Alluxio version you are running, e..g, for 2.0, the metrics of the local short-circuit read is not turned on by default. So I would suggest you to first turn on the metrics collecting local short-circuit reads by setting alluxio.user.metrics.collection.enabled=true
Regarding the generic question to achieve high data locality when running Spark on Alluxio, can you read this article https://www.alluxio.io/blog/top-10-tips-for-making-the-spark-alluxio-stack-blazing-fast/ and follow the suggests there. E.g., things can be weird on running Spark on YARN for this case. If you need more detailed instructions, feel free to join Alluxio community channel https://slackin.alluxio.io <https://www.alluxio.io/slack> - Bin Fan alluxio.io <http://bit.ly/2JctWrJ> | powered by <http://bit.ly/2JdD0N2> | Data Orchestration Summit 2019 <https://www.alluxio.io/data-orchestration-summit-2019/> On Wed, Aug 28, 2019 at 1:49 AM Jerry Yan <summer19890...@gmail.com> wrote: > Hi, > > We are running Spark jobs on an Alluxio Cluster which is serving 13 > gigabytes of data with 99% of the data is in memory. I was hoping to speed > up the Spark jobs by reading the in-memory data in Alluxio, but found > Alluxio local hit rate is only 1.68%, while Alluxio remote hit rate is > 98.32%. By monitoring the network IO across all worker nodes through > "dstat" command, I found that only two nodes had about 1GB of recv or send > in the whole precessand, and it is sending 1GB or receiving 1GB during > Spark Shuffle Stage. Is there any metrics I could check or configuration > to tune ? > > > Best, > > Jerry >
