Ethan Li created FLINK-9683:
-------------------------------
Summary: inconsistent behaviors when setting
historyserver.archive.fs.dir
Key: FLINK-9683
URL: https://issues.apache.org/jira/browse/FLINK-9683
Project: Flink
Issue Type: Bug
Affects Versions: 1.4.2
Reporter: Ethan Li
I am using release-1.4.2,
With fs.default-scheme and fs.hdfs.hadoopconf set correctly,
when setting
{code:java}
historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive
{code}
I am seeing
{code:java}
2018-06-27 18:51:12,692 WARN
org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create
Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'. Directory
will not be monitored.
java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is null.
Please specify the file system scheme explicitly in the URI.
at
org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)
{code}
And then if I set
{code:java}
historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive{code}
I am seeing:
{code:java}
java.io.IOException: The given file system URI
(hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority (like
for example HDFS NameNode address/port or S3 host). The attempt to use a
configured default authority failed: Hadoop configuration for default file
system ('fs.default.name' or 'fs.defaultFS') contains no valid authority
component (like hdfs namenode, S3 host, etc)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
at
org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
{code}
The only way it works is to provide the full path of hdfs like:
{code:java}
#historyserver.archive.fs.dir:
hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive
{code}
Above situations are because there are two parts of code treating "scheme"
differently.
https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302
https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338
I believe the first case should be supported if users have set
fs.default-scheme
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
