[
https://issues.apache.org/jira/browse/FLINK-9683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272668#comment-17272668
]
Jiang Xin edited comment on FLINK-9683 at 1/27/21, 8:35 AM:
------------------------------------------------------------
[~chesnay] You're right. I created a pr to remove the scheme check which seems
safe to me. But I didn't remove the public method
WebMonitorUtils#validateAndNormalizeUri in case of external calls, although
there's no use in Flink. Any idea?
was (Author: jiang xin):
[~chesnay] You're right. I created a pr to remove the scheme check which seems
safe to me. But I didn't remove the public method
WebMonitorUtils#validateAndNormalizeUri in case of external calls. Any idea?
> inconsistent behaviors when setting historyserver.archive.fs.dir
> ----------------------------------------------------------------
>
> Key: FLINK-9683
> URL: https://issues.apache.org/jira/browse/FLINK-9683
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.4.2
> Reporter: Ethan Li
> Priority: Major
> Labels: pull-request-available, starter
>
> I am using release-1.4.2,
>
> With fs.default-scheme and fs.hdfs.hadoopconf set correctly,
> when setting
> {code:java}
> historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive
> {code}
> I am seeing
> {code:java}
> 2018-06-27 18:51:12,692 WARN
> org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create
> Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'.
> Directory will not be monitored.
> java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is
> null. Please specify the file system scheme explicitly in the URI.
> at
> org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)
> {code}
>
> And then if I set
> {code:java}
> historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive{code}
> I am seeing:
>
> {code:java}
> java.io.IOException: The given file system URI
> (hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority
> (like for example HDFS NameNode address/port or S3 host). The attempt to use
> a configured default authority failed: Hadoop configuration for default file
> system ('fs.default.name' or 'fs.defaultFS') contains no valid authority
> component (like hdfs namenode, S3 host, etc)
> at
> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
> at
> org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
> {code}
> The only way it works is to provide the full path of hdfs like:
> {code:java}
> #historyserver.archive.fs.dir:
> hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive
> {code}
>
> Above situations are because there are two parts of code treating "scheme"
> differently.
> https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302
> https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338
> I believe the first case should be supported if users have set
> fs.default-scheme
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
