Sanjeev, I have opened a bug: https://issues.apache.org/jira/browse/CLOUDSTACK-8371
Thanks. On Tue, Apr 7, 2015 at 4:01 AM, Sanjeev N <[email protected]> wrote: > Looks like a bug to me. Please open a bug in https://issues.apache.org/ > > On Tue, Apr 7, 2015 at 1:57 AM, Grayson Head <[email protected]> > wrote: > >> One of my coworkers has created a VPC that we are unable to modify or >> remove. >> >> The coworker said he was testing the site-to-site VPN functionality, >> and ended up having to create a seperate VPC instance to get it >> working. Now we are unable to remove this VPC instance using the web >> interface or the API. I suspect based on the log outputs that the >> site-to-site VPN is what is causing this problem, but the VPC shows >> that it doesn't have any site-to-site connections configured. >> >> The VPC still has one external IP address allocated to it, which I >> think was the one he was using as the VPN gateway. I am also unable to >> remove this IP address, it fails with an "internal server" error. The >> following is the management-server.log when attempting to release this >> IP address. >> >> 2015-04-06 13:18:42,427 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Add job-6262 into job >> monitoring >> 2015-04-06 13:18:42,427 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Executing AsyncJobVO >> {id:6262, userId: 3, accountId: 3, instanceType: IpAddress, >> instanceId: 9, cmd: >> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd, >> cmdInfo: >> {"response":"json","id":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","sessionkey":"gSf69NNJJZ0SzrIEz1fwOQJpRTo\u003d","ctxDetails":"{\"com.cloud.network.IpAddress\":\"3d2205a4-41e9-4cf7-9261-d1ae63ce863f\"}","cmdEventType":"NET.IPRELEASE","ctxUserId":"3","httpmethod":"GET","_":"1428351522388","uuid":"3d2205a4-41e9-4cf7-9261-d1ae63ce863f","ctxAccountId":"3","ctxStartEventId":"7705"}, >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated: >> null, lastPolled: null, created: null} >> 2015-04-06 13:18:42,427 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-14:ctx-91e999f7 ctx-49efc96a) ===END=== 10.233.51.176 >> -- GET >> command=disassociateIpAddress&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&id=3d2205a4-41e9-4cf7-9261-d1ae63ce863f&_=1428351522388 >> 2015-04-06 13:18:42,451 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all >> Firewallrules as a part of public IP id=9 release... >> 2015-04-06 13:18:42,462 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0 >> firewall rules for ip id=9 >> 2015-04-06 13:18:42,464 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no >> firewall rules to apply >> 2015-04-06 13:18:42,466 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully >> released firewall rules for ip id=9 and # of rules now = 0 >> 2015-04-06 13:18:42,475 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all >> PortForwarding/StaticNat rules as a part of public IP id=9 release... >> 2015-04-06 13:18:42,477 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0 >> port forwarding rules for ip id=9 >> 2015-04-06 13:18:42,478 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Releasing 0 >> static nat rules for ip id=9 >> 2015-04-06 13:18:42,480 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no >> port forwarding rules to apply for ip id=9 >> 2015-04-06 13:18:42,482 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) There are no >> static nat rules to apply for ip id=9 >> 2015-04-06 13:18:42,483 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Source ip >> id=Ip[66.194.167.148-1] is not one to one nat >> 2015-04-06 13:18:42,487 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Successfully >> released rules for ip id=9 and # of rules now = 0 >> 2015-04-06 13:18:42,487 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Revoking all >> LoadBalancing rules as a part of public IP id=9 release... >> 2015-04-06 13:18:42,488 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Found 0 lb >> rules to cleanup >> 2015-04-06 13:18:42,488 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262 ctx-bbf8738d) Cleaning up >> remote access vpns as a part of public IP id=9 release... >> 2015-04-06 13:18:42,523 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Unexpected exception while >> executing >> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd >> java.lang.NullPointerException >> at >> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374) >> at >> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152) >> at >> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443) >> at >> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305) >> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> at >> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source) >> at >> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556) >> at >> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571) >> at >> com.cloud.network.NetworkServiceImpl.releaseIpAddressInternal(NetworkServiceImpl.java:935) >> at >> com.cloud.network.NetworkServiceImpl.releaseIpAddress(NetworkServiceImpl.java:895) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> at >> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> at com.sun.proxy.$Proxy156.releaseIpAddress(Unknown Source) >> at >> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd.execute(DisassociateIPAddrCmd.java:80) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) >> at >> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 2015-04-06 13:18:42,525 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Complete async job-6262, >> jobStatus: FAILED, resultCode: 530, result: >> >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530} >> 2015-04-06 13:18:42,534 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Done executing >> org.apache.cloudstack.api.command.user.address.DisassociateIPAddrCmd >> for job-6262 >> 2015-04-06 13:18:42,539 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-96:ctx-428fa98b job-6262) Remove job-6262 from job >> monitoring >> 2015-04-06 13:18:42,800 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-11:null) SeqA 9-147481: Processing Seq 9-147481: >> { Cmd , MgmtId: -1, via: 9, Ver: v1, Flags: 11, >> >> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":78,"_loadInfo":"{\n >> \"connections\": []\n}","wait":0}}] } >> 2015-04-06 13:18:42,808 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-11:null) SeqA 9-147481: Sending Seq 9-147481: { >> Ans: , MgmtId: 345050016868, via: 9, Ver: v1, Flags: 100010, >> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] >> } >> 2015-04-06 13:18:45,447 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-5:ctx-931b2c88) ===START=== 10.233.51.176 -- GET >> >> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444 >> 2015-04-06 13:18:45,478 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-5:ctx-931b2c88 ctx-7b984e22) ===END=== 10.233.51.176 >> -- GET >> command=queryAsyncJobResult&jobId=61329013-6b40-43d2-ad1b-8fba0325bae1&response=json&sessionkey=gSf69NNJJZ0SzrIEz1fwOQJpRTo%3D&_=1428351525444 >> >> Removing the VPC also gives an internal server error and outputs the >> following to the logs. >> >> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) submit async job-6257, >> details: AsyncJobVO {id:6257, userId: 3, accountId: 3, instanceType: >> None, instanceId: null, cmd: >> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd, cmdInfo: >> >> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"}, >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated: >> null, lastPolled: null, created: null} >> 2015-04-06 07:03:10,148 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Add job-6257 into job >> monitoring >> 2015-04-06 07:03:10,148 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Executing AsyncJobVO >> {id:6257, userId: 3, accountId: 3, instanceType: None, instanceId: >> null, cmd: org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd, >> cmdInfo: >> {"response":"json","id":"02158f27-88f7-4ffe-83bf-60cfea333dee","sessionkey":"68X4liCjVtaZUrDpChfoXD1a2Mo\u003d","ctxDetails":"{\"com.cloud.network.vpc.Vpc\":\"02158f27-88f7-4ffe-83bf-60cfea333dee\"}","cmdEventType":"VPC.DELETE","ctxUserId":"3","httpmethod":"GET","_":"1428328990097","uuid":"02158f27-88f7-4ffe-83bf-60cfea333dee","ctxAccountId":"3","ctxStartEventId":"7627"}, >> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >> result: null, initMsid: 345050016868, completeMsid: null, lastUpdated: >> null, lastPolled: null, created: null} >> 2015-04-06 07:03:10,149 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-12:ctx-1b8cdf03 ctx-97a67e82) ===END=== 10.233.51.176 >> -- GET >> command=deleteVPC&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&id=02158f27-88f7-4ffe-83bf-60cfea333dee&_=1428328990097 >> 2015-04-06 07:03:10,164 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Destroying >> vpc [VPC [9-VPC] >> 2015-04-06 07:03:10,169 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Shutting down >> vpc [VPC [9-VPC] >> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Vpc [VPC >> [9-VPC] has been shutdown succesfully >> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up >> resources for vpc id=9 >> 2015-04-06 07:03:10,172 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up >> existed site to site VPN connections >> 2015-04-06 07:03:10,174 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up >> existed site to site VPN gateways >> 2015-04-06 07:03:10,177 DEBUG [c.c.n.v.VpcManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing ips >> for vpc id=9 as a part of vpc cleanup >> 2015-04-06 07:03:10,177 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all >> Firewallrules as a part of public IP id=9 release... >> 2015-04-06 07:03:10,189 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0 >> firewall rules for ip id=9 >> 2015-04-06 07:03:10,190 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no >> firewall rules to apply >> 2015-04-06 07:03:10,192 DEBUG [c.c.n.f.FirewallManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully >> released firewall rules for ip id=9 and # of rules now = 0 >> 2015-04-06 07:03:10,202 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all >> PortForwarding/StaticNat rules as a part of public IP id=9 release... >> 2015-04-06 07:03:10,204 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0 >> port forwarding rules for ip id=9 >> 2015-04-06 07:03:10,206 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Releasing 0 >> static nat rules for ip id=9 >> 2015-04-06 07:03:10,208 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no >> port forwarding rules to apply for ip id=9 >> 2015-04-06 07:03:10,209 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) There are no >> static nat rules to apply for ip id=9 >> 2015-04-06 07:03:10,211 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Source ip >> id=Ip[66.194.167.148-1] is not one to one nat >> 2015-04-06 07:03:10,214 DEBUG [c.c.n.r.RulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Successfully >> released rules for ip id=9 and # of rules now = 0 >> 2015-04-06 07:03:10,214 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Revoking all >> LoadBalancing rules as a part of public IP id=9 release... >> 2015-04-06 07:03:10,216 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Found 0 lb >> rules to cleanup >> 2015-04-06 07:03:10,216 DEBUG [c.c.n.IpAddressManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257 ctx-499d812d) Cleaning up >> remote access vpns as a part of public IP id=9 release... >> 2015-04-06 07:03:10,254 ERROR [c.c.a.ApiAsyncJobDispatcher] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Unexpected exception while >> executing org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd >> java.lang.NullPointerException >> at >> com.cloud.network.NetworkModelImpl.getPhysicalNetworkId(NetworkModelImpl.java:1374) >> at >> com.cloud.network.element.VirtualRouterElement.canHandle(VirtualRouterElement.java:152) >> at >> com.cloud.network.element.VirtualRouterElement.stopVpn(VirtualRouterElement.java:443) >> at >> com.cloud.network.vpn.RemoteAccessVpnManagerImpl.destroyRemoteAccessVpnForIp(RemoteAccessVpnManagerImpl.java:305) >> at sun.reflect.GeneratedMethodAccessor670.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> at >> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> at com.sun.proxy.$Proxy231.destroyRemoteAccessVpnForIp(Unknown Source) >> at >> com.cloud.network.IpAddressManagerImpl.cleanupIpResources(IpAddressManagerImpl.java:556) >> at >> com.cloud.network.IpAddressManagerImpl.disassociatePublicIpAddress(IpAddressManagerImpl.java:571) >> at >> com.cloud.network.vpc.VpcManagerImpl.cleanupVpcResources(VpcManagerImpl.java:1429) >> at com.cloud.network.vpc.VpcManagerImpl.destroyVpc(VpcManagerImpl.java:938) >> at com.cloud.network.vpc.VpcManagerImpl.deleteVpc(VpcManagerImpl.java:899) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) >> at >> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) >> at >> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) >> at >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) >> at com.sun.proxy.$Proxy161.deleteVpc(Unknown Source) >> at >> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd.execute(DeleteVPCCmd.java:76) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) >> at >> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >> at >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >> at >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >> at >> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> 2015-04-06 07:03:10,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Complete async job-6257, >> jobStatus: FAILED, resultCode: 530, result: >> >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530} >> 2015-04-06 07:03:10,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Done executing >> org.apache.cloudstack.api.command.user.vpc.DeleteVPCCmd for job-6257 >> 2015-04-06 07:03:10,273 INFO [o.a.c.f.j.i.AsyncJobMonitor] >> (API-Job-Executor-91:ctx-9fe47d69 job-6257) Remove job-6257 from job >> monitoring >> 2015-04-06 07:03:11,046 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-1:null) Ping from 13 >> 2015-04-06 07:03:11,046 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) Process host VM state report from ping >> process. host: 13 >> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) Process VM state report. host: 13, >> number of records in report: 9 >> 2015-04-06 07:03:11,066 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 959, >> power state: PowerOn >> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 959 >> 2015-04-06 07:03:11,068 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 968, >> power state: PowerOn >> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 968 >> 2015-04-06 07:03:11,071 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 975, >> power state: PowerOn >> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 975 >> 2015-04-06 07:03:11,073 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 974, >> power state: PowerOn >> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 974 >> 2015-04-06 07:03:11,076 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 871, >> power state: PowerOn >> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 871 >> 2015-04-06 07:03:11,079 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 78, >> power state: PowerOn >> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 78 >> 2015-04-06 07:03:11,081 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 893, >> power state: PowerOn >> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 893 >> 2015-04-06 07:03:11,084 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 914, >> power state: PowerOn >> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 914 >> 2015-04-06 07:03:11,086 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM state report. host: 13, vm id: 912, >> power state: PowerOn >> 2015-04-06 07:03:11,089 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) VM power state does not change, skip DB >> writing. vm id: 912 >> 2015-04-06 07:03:11,095 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >> (AgentManager-Handler-1:null) Done with process of VM state report. >> host: 13 >> 2015-04-06 07:03:13,165 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-11:ctx-ad36fed0) ===START=== 10.233.51.176 -- GET >> >> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153 >> 2015-04-06 07:03:13,191 DEBUG [c.c.a.ApiServlet] >> (catalina-exec-11:ctx-ad36fed0 ctx-5af312b0) ===END=== 10.233.51.176 >> -- GET >> command=queryAsyncJobResult&jobId=85d5c719-6114-494d-bb52-26f43372976d&response=json&sessionkey=68X4liCjVtaZUrDpChfoXD1a2Mo%3D&_=1428328993153 >> >> >> I'm at a loss as to what could be causing this, and I don't want to >> hack the database to fix this without completely understanding how all >> of these components interact with each other. Any help would be >> appreciated. >>
