We have one more finding, issue occur because we did not wait for template creation process to finish, we have checked the logs and we restarted the management server even template creation process did not finished yet :) We need to find a way to rollback these template creation process like manually updating template table...
2018-01-15 11:41:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-55f871c2) (logid:5a3f1482) Task (job-1112) has been pending for 990 seconds 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1118) has been pending for 758 seconds 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1117) has been pending for 774 seconds 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1116) has been pending for 800 seconds 2018-01-15 11:42:14,778 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1115) has been pending for 881 seconds 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1114) has been pending for 892 seconds 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1113) has been pending for 905 seconds 2018-01-15 11:42:14,779 WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-988624e1) (logid:cae141bb) Task (job-1112) has been pending for 1050 seconds On Mon, Jan 15, 2018 at 12:21 PM, Özhan Rüzgar Karaman < [email protected]> wrote: > Hi; > We are using Cloudstack 4.9.3, we hit an error when trying to delete > template from cloudstack ui. Our developer send 7 template creation > requests for same snapshot each waiting 5-10 seconds between each requests. > > Now we have 7 templates all in Creating state and when we try to delete > them we get "Failed to update state" error from interface. The detailed > cloudstack management logs are attached. > > We also checked mysql db for templates status and all are in Creating > state, we restarted cloudstack management server to see if this template > creating process maybe rolled back but we again could not delete template > and we got same error. > > Because these templates remain on cloudstack our user could not take any > more actions because they hit to the resource limitation for user/domain. > > Is there a way to manually update Creating state to maybe Ready and try to > delete this templates after that? > > We only like to delete this templates in a proper way, does anyone could > give us an advice? > > Thanks > Özhan > > Mysql Records: > mysql> select * from template_store_ref where state = 'Creating'; > +----+----------+-------------+---------------------+------- > -------+--------+--------------+------+------------+-------- > -------+----------------+-----------+------------+---------- > -----------+------+----------+-----------+---------+-------- > ------+---------+---------------------+--------------------- > -+--------------+ > | id | store_id | template_id | created | last_updated | > job_id | download_pct | size | store_role | physical_size | download_state > | error_str | local_path | install_path | url | state | > destroyed | is_copy | update_count | ref_cnt | updated | > download_url_created | download_url | > +----+----------+-------------+---------------------+------- > -------+--------+--------------+------+------------+-------- > -------+----------------+-----------+------------+---------- > -----------+------+----------+-----------+---------+-------- > ------+---------+---------------------+--------------------- > -+--------------+ > | 23 | 1 | 213 | 2018-01-15 08:24:44 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/213 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:24:44 | NULL > | NULL | > | 24 | 1 | 214 | 2018-01-15 08:27:08 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/214 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:27:08 | NULL > | NULL | > | 25 | 1 | 215 | 2018-01-15 08:27:22 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/215 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:27:22 | NULL > | NULL | > | 26 | 1 | 216 | 2018-01-15 08:27:33 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/216 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:27:33 | NULL > | NULL | > | 27 | 1 | 217 | 2018-01-15 08:28:54 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/217 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:28:54 | NULL > | NULL | > | 28 | 1 | 218 | 2018-01-15 08:29:19 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/218 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:29:20 | NULL > | NULL | > | 29 | 1 | 219 | 2018-01-15 08:29:36 | NULL | NULL > | 0 | NULL | Image | 0 | NULL | > NULL | NULL | template/tmpl/4/219 | NULL | Creating | 0 > | 0 | 1 | 0 | 2018-01-15 08:29:36 | NULL > | NULL | > +----+----------+-------------+---------------------+------- > -------+--------+--------------+------+------------+-------- > -------+----------------+-----------+------------+---------- > -----------+------+----------+-----------+---------+-------- > ------+---------+---------------------+--------------------- > -+--------------+ > 7 rows in set (0.00 sec) > > mysql> > > mysql> select * from vm_template where display_text like '%api55'; > +-----+--------------------------------------------+-------- > ----------+--------------------------------------+--------+- > ---------+------+-----+------+------+--------+-------------- > -------+---------+------------+----------+------------------ > +-----------------+---------------+-------------+----------+ > -------------+-------------+-------------+-----------------+ > --------------------+--------------+----------+------+------ > --+--------------+---------+----------------------+ > | id | unique_name | name | > uuid | public | featured | type | hvm | > bits | url | format | created | removed | account_id | > checksum | display_text | enable_password | enable_sshkey | guest_os_id > | bootable | prepopulate | cross_zones | extractable | hypervisor_type | > source_template_id | template_tag | sort_key | size | state | update_count > | updated | dynamically_scalable | > +-----+--------------------------------------------+-------- > ----------+--------------------------------------+--------+- > ---------+------+-----+------+------+--------+-------------- > -------+---------+------------+----------+------------------ > +-----------------+---------------+-------------+----------+ > -------------+-------------+-------------+-----------------+ > --------------------+--------------+----------+------+------ > --+--------------+---------+----------------------+ > | 213 | 213-4-6c7694ef-cb29-3c06-878f-301300c71a21 | snaptovm - api55 | > 094de09d-0beb-4e4f-8f3c-bab492f47645 | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:24:44 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 214 | 214-4-c9071093-06cd-3351-a1e6-e5570b16c27a | snaptovm - api55 | > 57986622-f5fb-4200-a24a-95651ca086bc | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:27:08 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 215 | 215-4-ee8ec3d2-8880-35e0-b1c9-0365b9c1839e | snaptovm - api55 | > 95cb604b-e27c-4385-b2fc-9123c856549c | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:27:22 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 216 | 216-4-0ee2f1c7-ff28-3dbc-b58a-cae1e2f71a6b | snaptovm - api55 | > 533cea26-243d-43cf-ba2f-53c8daf0dbfe | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:27:32 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 217 | 217-4-3f5093c7-335c-3fc5-b5ff-7c0481dbf5e4 | snaptovm - api55 | > c3cc1614-0ac8-437d-ba1a-ce8253237c81 | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:28:53 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 218 | 218-4-d49b0a44-efef-337d-9bdc-5c859f67de32 | snaptovm - api55 | > e8f801e9-4c8a-420e-9df4-a26f98bb6c03 | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:29:19 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > | 219 | 219-4-9aece59e-c8d5-3eb8-9035-8db61072d007 | snaptovm - api55 | > c14dcd71-c001-408e-924f-f5cd9fc88d06 | 0 | 0 | USER | 1 | > 64 | NULL | RAW | 2018-01-15 08:29:35 | NULL | 4 | NULL > | snaptovm - api55 | 1 | 0 | 254 | > 1 | 0 | 0 | 1 | KVM | > 201 | NULL | 0 | NULL | Active | 0 | NULL > | 0 | > +-----+--------------------------------------------+-------- > ----------+--------------------------------------+--------+- > ---------+------+-----+------+------+--------+-------------- > -------+---------+------------+----------+------------------ > +-----------------+---------------+-------------+----------+ > -------------+-------------+-------------+-----------------+ > --------------------+--------------+----------+------+------ > --+--------------+---------+----------------------+ > 7 rows in set (0.01 sec) > > mysql> > > > Management Server Logs: > > 2018-01-15 12:02:35,069 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Unexpected > exception while executing org.apache.cloudstack.api.c > ommand.user.template.DeleteTemplateCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to update state > at org.apache.cloudstack.storage.image.store.TemplateObject. > processEvent(TemplateObject.java:174) > at org.apache.cloudstack.storage.image.TemplateServiceImpl. > deleteTemplateAsync(TemplateServiceImpl.java:698) > at com.cloud.template.HypervisorTemplateAdapter.delete( > HypervisorTemplateAdapter.java:401) > at com.cloud.template.TemplateManagerImpl.deleteTemplate( > TemplateManagerImpl.java:1192) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.springframework.aop.support.AopUtils. > invokeJoinpointUsingReflection(AopUtils.java:317) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:150) > at org.apache.cloudstack.network.contrail.management. > EventUtils$EventInterceptor.invoke(EventUtils.java:107) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:161) > at com.cloud.event.ActionEventInterceptor.invoke( > ActionEventInterceptor.java:51) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:161) > at org.springframework.aop.interceptor. > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:172) > at org.springframework.aop.framework.JdkDynamicAopProxy. > invoke(JdkDynamicAopProxy.java:204) > at com.sun.proxy.$Proxy183.deleteTemplate(Unknown Source) > at org.apache.cloudstack.api.command.user.template. > DeleteTemplateCmd.execute(DeleteTemplateCmd.java:113) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > at com.cloud.api.ApiAsyncJobDispatcher.runJob( > ApiAsyncJobDispatcher.java:108) > at org.apache.cloudstack.framework.jobs.impl. > AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at org.apache.cloudstack.managed.context. > ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl. > DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl. > DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl. > DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context. > ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at org.apache.cloudstack.framework.jobs.impl. > AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:473) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1152) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.cloud.utils.fsm.NoTransitionException: Unable to > transition to a new state from Creating via DestroyRequested > at com.cloud.utils.fsm.StateMachine2.getTransition( > StateMachine2.java:108) > at com.cloud.utils.fsm.StateMachine2.getNextState( > StateMachine2.java:94) > at com.cloud.utils.fsm.StateMachine2.transitTo( > StateMachine2.java:124) > ... 35 more > 2018-01-15 12:02:35,071 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-5:ctx-9135400d job-1133) (logid:ea158f63) Complete > async job-1133, jobStatus: FAILED, resultCode: 530, > result: org.apache.cloudstack.api.response.ExceptionResponse/ > null/{"uuidList":[],"errorcode":530,"errortext":"Failed to update state"} >
