I'm not sure this is applied consistently across Spark, but I'm dealing with another change now where an unqualified path is assumed to be a local file. The method Utils.resolvePath implements this logic and is used several places. Therefore I think this is probably intended behavior and you can write hdfs:///tmp if you mean to reference /tmp on HDFS.
On Wed, Oct 12, 2016 at 7:55 PM Koert Kuipers <ko...@tresata.com> wrote: > i see this warning when running jobs on cluster: > > 2016-10-12 14:46:47 WARN spark.SparkContext: Spark is not running in local > mode, therefore the checkpoint directory must not be on the local > filesystem. Directory '/tmp' appears to be on the local filesystem. > > however the checkpoint "directory" that it warns about is a hadoop path. i > use an unqualified path, which means a path on the default filesystem by > hadoop convention. when running on the cluster my default filesystem is > hdfs (and it correctly uses hdfs). > > how about if we change the method that does this check > (Utils.nonLocalPaths) to be aware of the default filesystem instead of > incorrectly assuming its local if not specified? > >
