Stephen Chu created HDFS-4841:
---------------------------------
Summary: FsShell commands using webhfds fail ClientFinalizer
shutdown hook
Key: HDFS-4841
URL: https://issues.apache.org/jira/browse/HDFS-4841
Project: Hadoop HDFS
Issue Type: Bug
Components: security, webhdfs
Affects Versions: 3.0.0
Reporter: Stephen Chu
Attachments: core-site.xml,
hadoop-root-namenode-hdfs-upgrade-pseudo.ent.cloudera.com.out, hdfs-site.xml,
jsvc.out
Hadoop version:
{code}
bash-4.1$ $HADOOP_HOME/bin/hadoop version
Hadoop 3.0.0-SNAPSHOT
Subversion git://github.com/apache/hadoop-common.git -r
d5373b9c550a355d4e91330ba7cc8f4c7c3aac51
Compiled by root on 2013-05-22T08:06Z
>From source with checksum 8c4cc9b1e8d6e8361431e00f64483f
This command was run using
/var/lib/hadoop-hdfs/hadoop-3.0.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.0.0-SNAPSHOT.jar
{code}
I'm seeing a problem when issuing FsShell commands using the webhdfs:// URI
when security is enabled. The command completes but leaves a warning that
ShutdownHook 'ClientFinalizer' failed.
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/
2013-05-22 09:46:55,710 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46
webhdfs://hdfs-upgrade-pseudo.ent.cloudera.com:50070/user
2013-05-22 09:46:58,660 WARN [Thread-3] util.ShutdownHookManager
(ShutdownHookManager.java:run(56)) - ShutdownHook 'ClientFinalizer' failed,
java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2400)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2372)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1001)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1013)
at org.apache.hadoop.security.token.Token.cancel(Token.java:382)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152)
at
org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58)
at
org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:822)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2446)
at
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2463)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
{code}
I've checked that FsShell + hdfs:// commands and WebHDFS operations through
curl work successfully:
{code}
bash-4.1$ hadoop-3.0.0-SNAPSHOT/bin/hadoop fs -ls /
2013-05-22 09:46:43,663 INFO [main] util.Shell
(Shell.java:isSetsidSupported(311)) - setsid exited with exit code 0
Found 3 items
drwxr-xr-x - hbase supergroup 0 2013-05-22 09:46 /hbase
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /tmp
drwxr-xr-x - hdfs supergroup 0 2013-05-22 09:46 /user
bash-4.1$ curl -i --negotiate -u :
"http://hdfs-upgrade-pseudo.ent.cloudera.com:50070/webhdfs/v1/?op=GETHOMEDIRECTORY";
HTTP/1.1 401
Cache-Control: must-revalidate,no-cache,no-store
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=;Path=/;Expires=Thu, 01-Jan-1970 00:00:00 GMT
Content-Length: 1358
Server: Jetty(6.1.26)
HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 01-Jan-1970 00:00:00 GMT
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Date: Wed, 22 May 2013 16:47:14 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie:
hadoop.auth="u=hdfs&p=hdfs/hdfs-upgrade-pseudo.ent.cloudera....@ent.cloudera.com&t=kerberos&e=1369277234852&s=m3vJ7/pV831tBLkpOBb0Naa5N+g=";Path=/
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
{"Path":"/user/hdfs"}bash-4.1$
{code}
When I disable security, the warning goes away.
I'll attach my core-site.xml, hdfs-site.xml, NN and DN output logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
