[
https://issues.apache.org/jira/browse/CASSANDRA-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945960#comment-13945960
]
Arya Goudarzi commented on CASSANDRA-5345:
------------------------------------------
Running 1.2.14 on Ubuntu 12.04 with HotSpot Java 1.7.0_21-b11.
For the past couple of days our nodes started to produce this exception every
second. This exception started to get produced after we decommissioned one DC,
and it continued to get produced until the nodes started producing client
errors as we got alerts. No suspicious other log entries were found and no full
GCs were recorded in the GC logs. The GC logs looked normal however system.log
was getting filled with this exception.
We decommissioned the DC by first altering our keyspace to not replication to
that DC as we are using NetworkTopologyStrategy. Then we issued nodetool
decommission on each node in that DC.
Why this exception started to get produced is not known. However, restarting
the nodes fixed the problem. Another observation was that the nodetool info
command which we use to collect heap size statistics was not working as well as
was tossing this exception:
Exception in thread "main" java.lang.IllegalArgumentException:
javax.management.InstanceNotFoundException: java.lang:type=Memory
at
java.lang.management.ManagementFactory.newPlatformMXBeanProxy(ManagementFactory.java:610)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:175)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:116)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1138)
Caused by: javax.management.InstanceNotFoundException: java.lang:type=Memory
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.isInstanceOf(DefaultMBeanServerInterceptor.java:1401)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.isInstanceOf(JmxMBeanServer.java:1082)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1492)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:96)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1327)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1419)
at
javax.management.remote.rmi.RMIConnectionImpl.isInstanceOf(RMIConnectionImpl.java:957)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
> Potential problem with GarbageCollectorMXBean
> ---------------------------------------------
>
> Key: CASSANDRA-5345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5345
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.7
> Environment: JVM:JVM vendor/version: Java HotSpot(TM) 64-Bit Server
> VM/1.6.0_30 typical 6 node 2 availability zone Mutli DC cluster on linux vms
> with
> and mx4j-tools.jar and jna.jar both on path. Default configuration bar token
> setup(equispaced), sensible cassandra-topology.properties file and use of
> said snitch.
> Reporter: Matt Byrd
> Priority: Trivial
>
> I am not certain this is definitely a bug, but I thought it might be worth
> posting to see if someone with more JVM//JMX knowledge could disprove my
> reasoning. Apologies if I've failed to understand something.
> We've seen an intermittent problem where there is an uncaught exception in
> the scheduled task of logging gc results in GcInspector.java:
> {code}
> ...
> ERROR [ScheduledTasks:1] 2013-03-08 01:09:06,335
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[ScheduledTasks:1,5,main]
> java.lang.reflect.UndeclaredThrowableException
> at $Proxy0.getName(Unknown Source)
> at
> org.apache.cassandra.service.GCInspector.logGCResults(GCInspector.java:95)
> at
> org.apache.cassandra.service.GCInspector.access$000(GCInspector.java:41)
> at org.apache.cassandra.service.GCInspector$1.run(GCInspector.java:85)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.management.InstanceNotFoundException:
> java.lang:name=ParNew,type=GarbageCollector
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at
> com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:106)
> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:148)
> at
> javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:248)
> ... 13 more
> ...
> {code}
> I think the problem, may be caused by the following reasoning:
> In GcInspector we populate a list of mxbeans when the GcInspector instance is
> instantiated:
> {code}
> ...
> List<GarbageCollectorMXBean> beans = new ArrayList<GarbageCollectorMXBean>();
> MBeanServer server = ManagementFactory.getPlatformMBeanServer();
> try
> {
> ObjectName gcName = new
> ObjectName(ManagementFactory.GARBAGE_COLLECTOR_MXBEAN_DOMAIN_TYPE + ",*");
> for (ObjectName name : server.queryNames(gcName, null))
> {
> GarbageCollectorMXBean gc =
> ManagementFactory.newPlatformMXBeanProxy(server, name.getCanonicalName(),
> GarbageCollectorMXBean.class);
> beans.add(gc);
> }
> }
> catch (Exception e)
> {
> throw new RuntimeException(e);
> }
> ...
> {code}
> Cassandra then periodically calls:
> {code}
> ...
> private void logGCResults()
> {
> for (GarbageCollectorMXBean gc : beans)
> {
> Long previousTotal = gctimes.get(gc.getName());
> ...
> {code}
> In the oracle javadocs, they seem to suggest that these beans could disappear
> at any time.(I'm not sure why when or how this might happen)
> http://docs.oracle.com/javase/6/docs/api/
> See: getGarbageCollectorMXBeans
> {code}
> ...
> public static List<GarbageCollectorMXBean> getGarbageCollectorMXBeans()
> Returns a list of GarbageCollectorMXBean objects in the Java virtual machine.
> The Java virtual machine may have one or more GarbageCollectorMXBean objects.
> It may add or remove GarbageCollectorMXBean during execution.
> Returns:
> a list of GarbageCollectorMXBean objects.
> ...
> {code}
> Correct me if I'm wrong, but do you think this might be causing the problem?
> That somehow the JVM decides to remove the GarbageCollectorMXBean temporarily
> or permanently (causing said exception) and if this is expected behaviour,
> should it be handled in some way?
> Also I'd like to point out that this may be an issue on other versions as
> well as I don't believe this code has changed in quite a long time.
> Unfortunately I haven't been able to reproduce this outside of the production
> environment, if you have any tips, questions or are able to explain//disprove
> my concerns, I'd be very grateful.
> Thanks,
> Matt
--
This message was sent by Atlassian JIRA
(v6.2#6252)
