> Looking at https://github.com/apache/spark/pull/1222/files , > the following change may have caused what Stephen described: > > + if (!fileSystem.isDirectory(new Path(logBaseDir))) { > > When there is no schema associated with logBaseDir, local path > should be assumed.
Yes, that looks right. In branch-1.2, it looks like: logDir goes through getLogDirPath: https://github.com/apache/spark/blob/branch-1.2/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L61 Which calls resolveUri: https://github.com/apache/spark/blob/branch-1.2/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L185 Which prepends the file scheme if needed: https://github.com/apache/spark/blob/branch-1.2/core/src/main/scala/org/apache/spark/util/Utils.scala#L1588 So, raw/scheme-less "/some/dir" paths were previously supported. However, now on master, logBaseDir is passed to getHadoopFileSystem directly: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L64 Note that getLogDirPath was renamed to just getLogPath: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L264 But, right, per Ted Yu's comment, it's not used when creating the file system. I'd file a pull request but Eclipse/maven/etc. is being dumb. Can someone/a regular spark dev pick this up? Or else I can keep fighting Eclipse/m2e for awhile. - Stephen --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
