Andy Isaacson created HADOOP-8447:
-------------------------------------
Summary: DN logs backtrace when running under jsvc and /jmx is
loaded
Key: HADOOP-8447
URL: https://issues.apache.org/jira/browse/HADOOP-8447
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Priority: Minor
Running the data node under jsvc and requesting /jmx falls victim to a kernel
bug http://marc.info/?l=linux-kernel&m=133788505209725&w=2 which results in
EACCES when open()ing /proc/self/fd to attempt to count the open
filedescriptors.
Hopefully someday we will have kernels with this bug fixed; in the meantime,
the log spew which results is unpleasant:
12270 2012-05-15 21:04:41,683 ERROR org.apache.hadoop.jmx.JMXJsonServlet:
getting attribute OpenFileDescriptorCount of java.lang:type=OperatingSystem
threw an exception
12271 javax.management.RuntimeErrorException: java.lang.InternalError: errno:
13 error: Unable to open directory /proc/self/fd
12272
12273 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
12274 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
12275 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:670)
12276 at
com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
12277 at
org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:314)
12278 at
org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:292)
12279 at
org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:192)
12280 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
12281 at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
12282 at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
12283 at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
12284 at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:932)
12285 at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
12286 at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
12287 at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
12288 at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
12289 at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
12290 at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
12291 at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
12292 at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
12293 at org.mortbay.jetty.Server.handle(Server.java:326)
12294 at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
12295 at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
12296 at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
12297 at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
12298 at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
12299 at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
12300 at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
12301 Caused by: java.lang.InternalError: errno: 13 error: Unable to open
directory /proc/self/fd
12302
12303 at
com.sun.management.UnixOperatingSystem.getOpenFileDescriptorCount(Native Method)
12304 at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
12305 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
12306 at java.lang.reflect.Method.invoke(Method.java:597)
12307 at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167)
12308 at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96)
12309 at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33)
12310 at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
12311 at
com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
12312 at
com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
12313 at
javax.management.StandardMBean.getAttribute(StandardMBean.java:358)
12314 at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
12315 ... 25 more
we should catch the RuntimeErrorException and make it a debug().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
