What is directory "/path/_spark_metadata" for?

2019-08-28 Thread Mark Zhao
Hey, When running Spark on Alluxio-1.8.2, I encounter the following exception: “alluxio.exception.FileDoseNotExistException: Path “/test-data/_spark_metadata” does not exist” in Alluxio master.log. What exactly is the directory "_spark_metadata" used for? And how can I fix this problem? Thanks.

Can I set the Alluxio WriteType in Spark applications?

2019-09-17 Thread Mark Zhao
Hi, If Spark applications write data into alluxio, can WriteType be configured? Thanks, Mark