[
https://issues.apache.org/jira/browse/CLOUDSTACK-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722690#comment-13722690
]
Kelven Yang commented on CLOUDSTACK-3534:
-----------------------------------------
Please do following configuration changes to help generate necessary log
information, without these configuration changes, it will be hard to trace down
the root cause
1) configure /etc/my.cnf for slow SQL query logs, after you've done changes in
my.cnf, please make sure to restart mysql service.
2) in db.properties files, append following to db.cloud.url.params
db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&includeThreadDumpInDeadlockExceptions=true&logSlowQueries=true
3) config log4j-cloud.xml, turning log level to TRACE and add following
additional log section
<category name="com.cloud.utils.db.Transaction">
<priority value="TRACE"/>
</category>
> SQL Exception while updating Async job status when trying to put cluster
> level primary storage into maintenance mode
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3534
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3534
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Sailaja Mada
> Assignee: Alena Prokharchyk
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: apilog.log, management-server.log
>
>
> Steps:
> 1. Configure Advance Zone configuration using VMWARE
> 2. Add two primary storages (1 - Cluster level during zone config 2 - Zone
> level PS )
> 3. Deploy VM and add one volume using User account on to the cluster level PS
> 4. Tried to put the cluster level PS into maintenance state
> Observation:
> SQL Exception while updating Async job status when trying to put the host
> into maintenance mode
> 2013-07-15 20:23:48,277 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-158:null) Seq 1-1271202483: Response Received:
> 2013-07-15 20:23:48,277 DEBUG [agent.manager.ClusteredAgentManagerImpl]
> (DirectAgent-158:null) Seq 1-1271202483: MgmtId 94838926819810: Resp: Routing
> to peer
> 2013-07-15 20:23:50,272 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
> 2013-07-15 20:23:51,284 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
> (consoleproxy-1:null) Skip capacity scan due to there is no Primary Storage
> UPintenance mode
> 2013-07-15 20:23:52,370 DEBUG [db.Transaction.Transaction]
> (catalina-exec-3:null) Rolling back the transaction: Time = 50962 Name =
> queryAsyncJobResult; called by
> -Transaction.rollback:890-Transaction.removeUpTo:833-Transaction.close:657-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:133-GenericDaoBase.update:775-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-GenericDaoBase.update:1256-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-AsyncJobManagerImpl.queryAsyncJobResult:441-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:125-AsyncJobManagerImpl.queryAsyncJobResult:401
> 2013-07-15 20:23:52,370 ERROR [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-3:null) Unexpected exception while querying async job-54 = [
> 375a5b3d-ad76-4367-a04b-7b2a94668cb1 ] status:
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.PreparedStatement@3f71962d: UPDATE async_job SET
> async_job.last_polled='2013-07-15 14:53:01' WHERE async_job.id = 54
> at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:822)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:775)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1256)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.async.AsyncJobManagerImpl.queryAsyncJobResult(AsyncJobManagerImpl.java:441)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.async.AsyncJobManagerImpl.queryAsyncJobResult(AsyncJobManagerImpl.java:401)
> at
> com.cloud.api.ApiResponseHelper.queryJobResult(ApiResponseHelper.java:1929)
> at
> org.apache.cloudstack.api.command.user.job.QueryAsyncJobResultCmd.execute(QueryAsyncJobResultCmd.java:66)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:529)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting
> transaction
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2450)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2371)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2355)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:814)
> ... 48 more
> 2013-07-15 20:23:52,379 DEBUG [cloud.api.ApiServlet] (catalina-exec-3:null)
> ===END=== 10.104.255.13 -- GET
> command=queryAsyncJobResult&jobId=375a5b3d-ad76-4367-a04b-7b2a94668cb1&response=json&sessionkey=Qao%2FstRUJUp6xhAbIH5OSE5XRNw%3D&_=1373900159912
> 2013-07-15 20:23:52,693 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null)
> ===START=== 10.104.255.13 -- GET
> command=queryAsyncJobResult&jobId=375a5b3d-ad76-4367-a04b-7b2a94668cb1&response=json&sessionkey=Qao%2FstRUJUp6xhAbIH5OSE5XRNw%3D&_=1373900210676
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
