Doesn’t seem related to that particular commit, the function
invalidateRunSession appears to be dead code.
I’ll do some digging.
Cheers,
Hugo
On 10 jul. 2014, at 10:40, Funs Kessen <fkes...@schubergphilis.com> wrote:
> Hi Devs,
>
> I recently noticed on master that after starting, stopping and starting the
> management server again I get stacktraces about every second.
> After some digging with Daan we found that part of the
> a600d8408ea86782318139c17cf346c84979943d0 commit causes this. The problem is
> however deeper rooted, as the code is supposed to update the mshost table,
> but because another Session ID (actually _runId from ClusterManagerImpl.java
> from the previous run) is already in there the update fails. The Session ID
> is based on time in milliseconds, and offcourse changes when you stop and
> start the management server again.
> Prior to the commit it failed silently, and has done so since the initial
> checkin it seems. The real question is what the original idea is behind the
> _runId, and is it something that requires fixing ?
>
> I've pasted part of the commit, sql and stacktrace below.
>
> Cheers,
>
> Funs
>
> ===
> mysql> select * from mshost;
> +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+
> | id | msid | runid | name | state | version |
> service_ip | service_port | last_update | removed | alert_count |
> +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+
> | 1 | 90520734207775 | 1404924979461 | cs-mgmt | Up | 4.5.0-SNAPSHOT |
> 127.0.0.1 | 9090 | 2014-07-09 16:58:07 | NULL | 0 |
> +----+----------------+---------------+---------+-------+----------------+------------+--------------+---------------------+---------+-------------+
> 1 row in set (0.00 sec)
>
> ===
> commit a600d8408ea86782318139c17cf346c8497943d0
> Author: Santhosh Edukulla <santhosh.eduku...@gmail.com> 2014-07-02 10:38:16
> Committer: Santhosh Edukulla <santhosh.eduku...@gmail.com> 2014-07-04 12:47:58
>
> Fixed Resource Leaks, null dereferences, few other issues reported by coverity
>
> -
> framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java
> diff --git
> a/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java
>
> b/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java
> index 3d0c3f5..89d7d27 100644
> ---
> a/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java
> +++
> b/framework/cluster/src/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java
> @@ -53,15 +53,14 @@
> @Override
> public void invalidateRunSession(long id, long runid) {
> TransactionLegacy txn = TransactionLegacy.currentTxn();
> - PreparedStatement pstmt = null;
> - try {
> - pstmt = txn.prepareAutoCloseStatement("update mshost set
> runid=0, state='Down' where id=? and runid=?");
> - pstmt.setLong(1, id);
> - pstmt.setLong(2, runid);
> -
> - pstmt.executeUpdate();
> + try (PreparedStatement pstmt = txn.prepareStatement("update mshost
> set runid=0, state='Down' where id=? and runid=?");){
> + if(pstmt != null) {
> + pstmt.setLong(1, id);
> + pstmt.setLong(2, runid);
> + pstmt.executeUpdate();
> + }
> } catch (SQLException e) {
> - throw new CloudRuntimeException("DB exception on " +
> pstmt.toString(), e);
> + throw new
> CloudRuntimeException("invalidateRunSession:Exception:"+ e.getMessage(), e);
> }
> }
>
> ===
> 2014-07-09 18:45:06,541 WARN [c.c.c.d.ManagementServerHostDaoImpl]
> (Cluster-Heartbeat-1:ctx-5f2f8ad5) update:Exception:Invalid cluster session
> detected
> com.cloud.utils.exception.CloudRuntimeException: Invalid cluster session
> detected
> at
> com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy158.update(Unknown Source)
> at
> com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:545)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: com.cloud.cluster.ClusterInvalidSessionException: runid
> 1404924278923 is no longer valid
> at
> com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147)
> ... 26 more
> 2014-07-09 18:45:06,543 ERROR [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-5f2f8ad5) Unexpected exception in cluster heartbeat
> java.lang.RuntimeException: update:Exception:Invalid cluster session detected
> at
> com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy158.update(Unknown Source)
> at
> com.cloud.cluster.ClusterManagerImpl$4.runInContext(ClusterManagerImpl.java:545)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Invalid cluster
> session detected
> at
> com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147)
> ... 26 more
> Caused by: com.cloud.cluster.ClusterInvalidSessionException: runid
> 1404924278923 is no longer valid
> at
> com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(ManagementServerHostDaoImpl.java:147)
> ... 26 more
>
>
>
