[
https://issues.apache.org/jira/browse/HIVE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-5796:
----------------------------------------------------
Fix Version/s: 0.13.0
> percentjobcomplete returned by webhcat is null
> ----------------------------------------------
>
> Key: HIVE-5796
> URL: https://issues.apache.org/jira/browse/HIVE-5796
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
> Affects Versions: 0.12.0
> Reporter: Hari Sankar Sivarama Subramaniyan
> Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 0.13.0
>
> Attachments: HIVE-5796.1.patch.txt
>
>
> Currently the WebHCat E2E test TestMapReduce_1 fails when comparing the job
> status field percentComplete which is returned as null, the expected value is
> "map 100% reduce 100%".
> In the templeton.log I see the following message:
> {noformat}
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Using Hadoop
> Version: 0.23
> INFO | 07 Nov 2013 17:14:08,106 |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
> WEBHCAT_CONF_DIR=null
> WARN | 07 Nov 2013 17:14:08,106 |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Could not
> find
> D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\bin\null\override-container-log4j.properties.
> Monitoring of Hadoop jobs submitted through WebHCat may be affected.
> {noformat}
> To get past this problem I explicitly set the WEBHCAT_CONF_DIR environment
> variable to HCATALOG_HOME/etc/webhcat where
> override-container-log4j.properties is present.
> After fixing the above and rerunning, I still see the following error:
> {noformat}
> INFO | 07 Nov 2013 18:29:39,517 |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
> WEBHCAT_CONF_DIR=D:\hdp\\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat
> INFO | 07 Nov 2013 18:29:39,517 |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob | Found
> D:\hdp\hcatalog-0.12.0.2.0.6.0-1007\etc\webhcat\override-container-log4j.properties
> to use for job submission.
> ERROR | 07 Nov 2013 18:29:39,519 |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob |
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob is not
> properly initialized. Monitoring of Hadoop jobs submitted through WebHCat may
> be affected.
> java.lang.IllegalArgumentException: Pathname /d:/hadoop/hdfs/tmp from
> d:/hadoop/hdfs/tmp is not a valid DFS filename.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397)
> at
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:93)
> at
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob$1.run(TempletonControllerJob.java:82)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.copyLog4JtoFileSystem(TempletonControllerJob.java:82)
> at
> org.apache.hive.hcatalog.templeton.tool.TempletonControllerJob.<clinit>(TempletonControllerJob.java:126)
> at
> org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:104)
> at
> org.apache.hive.hcatalog.templeton.LauncherDelegator$1.run(LauncherDelegator.java:101)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at
> org.apache.hive.hcatalog.templeton.LauncherDelegator.queueAsUser(LauncherDelegator.java:101)
> at
> org.apache.hive.hcatalog.templeton.LauncherDelegator.enqueueController(LauncherDelegator.java:82)
> at
> org.apache.hive.hcatalog.templeton.JarDelegator.run(JarDelegator.java:55)
> at
> org.apache.hive.hcatalog.templeton.Server.mapReduceJar(Server.java:690)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1480)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1411)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1360)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1350)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1360)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at org.eclipse.jetty.server.Server.handle(Server.java:349)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:925)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> at java.lang.Thread.run(Unknown Source)
> {noformat}
> I think the problem is due to the reliance on the value of hadoop.tmp.dir as
> a temporary HDFS directory which is a temporary file location and applies to
> both local file system and HDFS. There are some derived parameters that build
> on top of this location assuming its local (eg. mapred.local.dir) and
> sometimes assuming its on hdfs (eg. mapred.system.dir), in short it is very
> hairy to rely on hadoop.tmp.dir.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
