Funs, Santhosh fixed this problem yesterday. https://issues.apache.org/jira/browse/CLOUDSTACK-7079 has more details.
Thanks, Likitha -----Original Message----- From: Funs Kessen [mailto:fkes...@schubergphilis.com] Sent: Thursday, July 10, 2014 2:10 PM To: dev@cloudstack.apache.org Cc: Daan Hoogland Subject: "Bug" introduced by part of commit a600d8408ea86782318139c17cf346c8497943d0 Hi Devs, I recently noticed on master that after starting, stopping and starting the management server again I get stacktraces about every second. After some digging with Daan we found that part of the a600d8408ea86782318139c17cf346c84979943d0 commit causes this. The problem is however deeper rooted, as the code is supposed to update the mshost table, but because another Session ID (actually _runId from ClusterManagerImpl.java from the previous run) is already in there the update fails. The Session ID is based on time in milliseconds, and offcourse changes when you stop and start the management server again. Prior to the commit it failed silently, and has done so since the initial checkin it seems. The real question is what the original idea is behind the _runId, and is it something that requires fixing ? I've pasted part of the commit, sql and stacktrace below. Cheers, Funs === mysql> select * from mshost; +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+ | id | msid | runid | name | state | version | service_ip | service_port | last_update | removed | alert_count | +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+ | 1 | 90520734207775 | 1404924979461 | cs-mgmt | Up | 4.5.0-SNAPSHOT | 127.0.0.1 | 9090 | 2014-07-09 16:58:07 | NULL | 0 | +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+ 1 row in set (0.00 sec) === commit a600d8408ea86782318139c17cf346c8497943d0 Author: Santhosh Edukulla <santhosh.eduku...@gmail.com> 2014-07-02 10:38:16 Committer: Santhosh Edukulla <santhosh.eduku...@gmail.com> 2014-07-04 12:47:58 Fixed Resource Leaks, null dereferences, few other issues reported by coverity - framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java diff --git a/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java b/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java index 3d0c3f5..89d7d27 100644 --- a/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java +++ b/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java @@ -53,15 +53,14 @@ @Override public void invalidateRunSession(long id, long runid) { TransactionLegacy txn = TransactionLegacy.currentTxn(); - PreparedStatement pstmt = null; - try { - pstmt = txn.prepareAutoCloseStatement("update mshost set runid=0, state='Down' where id=? and runid=?"); - pstmt.setLong(1, id); - pstmt.setLong(2, runid); - - pstmt.executeUpdate(); + try (PreparedStatement pstmt = txn.prepareStatement("update mshost set runid=0, state='Down' where id=? and runid=?");){ + if(pstmt != null) { + pstmt.setLong(1, id); + pstmt.setLong(2, runid); + pstmt.executeUpdate(); + } } catch (SQLException e) { - throw new CloudRuntimeException("DB exception on " + pstmt.toString(), e); + throw new CloudRuntimeException("invalidateRunSession:Exception:"+ e.getMessage(), e); } } === 2014-07-09 18:45:06,541 WARN [c.c.c.d.ManagementServerHostDaoImpl] (Cluster-Heartbeat-1:ctx-5f2f8ad5) update:Exception:Invalid cluster session detected com.cloud.utils.exception.CloudRuntimeException: Invalid cluster session detected at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy158.update(Unknown Source) at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:545) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: com.cloud.cluster.ClusterInvalidSessionException: runid 1404924278923 is no longer valid at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147) ... 26 more 2014-07-09 18:45:06,543 ERROR [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-5f2f8ad5) Unexpected exception in cluster heartbeat java.lang.RuntimeException: update:Exception:Invalid cluster session detected at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:155) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy158.update(Unknown Source) at com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:545) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: com.cloud.utils.exception.CloudRuntimeException: Invalid cluster session detected at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147) ... 26 more Caused by: com.cloud.cluster.ClusterInvalidSessionException: runid 1404924278923 is no longer valid at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147) ... 26 more
