Any idea when we can get fix for https://issues.apache.org/jira/browse/CLOUDSTACK-4825 ?
Regards, Rayees -----Original Message----- From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] Sent: Wednesday, October 09, 2013 7:42 AM To: dev@cloudstack.apache.org Subject: Re: Latest Master DB issue I think I'll solve this with a fourth option. I don't particularly like two thing. Threads that never access the DB before are now doing it (albeit just once) and if there is any error calling the DB it does exit(1). I'm going to make the CallingContext instance that gets registered for the system account do lazy loading of the User and Account objects. So the system CallContext will still be available on all thread, but if you never access it, it will never hit the database. Second, I'm going to remove the exit(1) that was already in the code. Its just a bad idea. If you were to ever exhauster your DB connection pool (or the number of connections on mysql), your entire mgmt stack will kill itself and by default we don't run ACS under process supervision. Darren On Wed, Oct 9, 2013 at 1:59 AM, Darren Shepherd <darren.s.sheph...@gmail.com> wrote: > Kelven, > > So the issue is the combination of my code with VmwareContextPool. > With the ManagedContext framework, what I've done is replace every > Runnable and TimerTask with ManagedContextRunnable and > ManagedContextTimerTask. Those classes will run the onEnter, onLeave > logic which will setup CallContext as a result. I purposely changed > every reference so that everything was consistent. I didn't want to > have developers have to consider when or when not to use > ManagedContext, so just always use it. So as a result even though > your code has nothing to do with the DB, the ManageContextListener for > CallContext does. > > So I'm sure your thinking that Resources shouldn't call the DB. The > ManagedContext framework only does things when deployed in a managed > JVM. The only managed JVM is the mgmt server. If it's AWSAPI, Usage, > or an Agent JVM, then that framework does nothing. > > So there are three possible solutions I see > > 1) Change VmwareContextPool to be initialized from the @PostConstruct > or start() method. > 2) Revert the change to VmwareContextPool to use TimerTask and not > ManageContextTimerTask > 3) Merge spring modularization. > > The simplest stop gap would be option 2. > > Darren > > On Tue, Oct 8, 2013 at 5:35 PM, Kelven Yang <kelven.y...@citrix.com> wrote: >> The problem seems to me is whether or not a background job that >> touches with database respects the bootstrap initialization order. As >> of VmwareContextPool itself, its background job does something fully >> within its own territory (no database, no reference outside). and >> vmware-base package was originally designed to be running on its own >> without assuming any container that offers unified lifecycle >> management. I don't think this type of background job has anything to >> do with the failure in this particular case. >> >> However, I do agree that we need to clean up and unify a few things >> inside the CloudStack, especially on life-cycle management and all >> background-jobs that their execution path touches with component >> life-cyle, auto-wiring, AOP etc. >> >> To live with the time before the spring modularization merge, we just >> need to figure out which background job that triggers all these and >> get it fixed, it used to work before even it is fragile, I don't >> think the fix of the problem is impossible. Is anyone working on this issue? >> >> Kelven >> >> >> >> >> On 10/8/13 2:35 PM, "Darren Shepherd" <darren.s.sheph...@gmail.com> wrote: >> >>>Some more info about this. What specifically is happening is that >>>the VmwareContextPool call is creating a Timer during the constructor >>>of the class which is being constructed in a static block from >>>VmwareContextFactory. So when the VmwareContextFactory class is >>>loaded by the class loader, the background thread is created. Which >>>is way, way before the Database upgrade happens. This will still be >>>fixed if we merge the spring modularization, but this vmware code >>>should change regardless. Background threads should only be launched >>>from a @PostConstruct or ComponentLifecycle.start() method. They >>>should not be started when a class is constructed or loaded. >>> >>>Darren >>> >>> >>>On Tue, Oct 8, 2013 at 2:22 PM, Darren Shepherd >>><darren.s.sheph...@gmail.com> wrote: >>>> Hey, I half way introduced this issue in a really long and round >>>> about way. I don't think there's a good simple fix unless we merge >>>> the spring-modularization branch. I'm going to look further into >>>> it. But here's the background of why we are seeing this. >>>> >>>> I introduced "Managed Context" framework that will wrap all the >>>> background threads and manage the thread locals. This was the >>>> union of CallContext, ServerContext, and AsyncJob*Context into one >>>> simple framework. The problem with ACS though is that A LOT of >>>> background threads are spawned at all different random times of the >>>> initialization. So what is happening is that during the >>>> initialization of some bean its kicking off a background thread >>>> that tries to access the database before the database upgrade has >>>> ran. Now the CallContext has a strange suicidal behaviour (this >>>> was already there, I didn't change this), if it can't find account >>>> 1, it does a System.exit(1). So since this one background thread >>>> is failing, the whole JVM shuts down. Before CallContext only >>>> existed on some threads, but the addition of the Managed Context >>>> framework, it is now on almost all threads. >>>> >>>> Now in the spring-modularization branch there is a very strict and >>>> (mostly) deterministic initialization order. The database upgrade >>>> class will be initialized and ran before any other bean in >>>> CloudStack is even initiated. So this works around all these DB >>>> problems. The current spring setup in master is very, very >>>> fragile. As I said before, it is really difficult to ensure >>>> certain aspects are initialized before others, and since we moved >>>> to (which I don't really agree with) doing DB schema upgrades >>>> purely on startup of the mgmt server, we now have to be extra careful >>>> about initialization order. >>>> >>>> Darren >>>> >>>> On Tue, Oct 8, 2013 at 11:34 AM, Rayees Namathponnan >>>> <rayees.namathpon...@citrix.com> wrote: >>>>> Here the defect created for this issue >>>>> >>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-4825 >>>>> >>>>> >>>>> Regards, >>>>> Rayees >>>>> >>>>> -----Original Message----- >>>>> From: Francois Gaudreault [mailto:fgaudrea...@cloudops.com] >>>>> Sent: Tuesday, October 08, 2013 11:15 AM >>>>> To: dev@cloudstack.apache.org >>>>> Subject: Re: Latest Master DB issue >>>>> >>>>> I guess in my case, it's fine. It was a fresh install... >>>>> >>>>> Francois >>>>> >>>>> On 10/8/2013, 2:08 PM, Darren Shepherd wrote: >>>>>> Deploy db from maven will drop all the tables. Not sure if this >>>>>> is fresh install or not. For master, running mvn will be your best bet. >>>>>> Otherwise you can look at running >>>>>> com.cloud.upgrade.DatabaseCreator manually if your adventurous. >>>>>> >>>>>> Darren >>>>>> >>>>>> On Tue, Oct 8, 2013 at 10:55 AM, Francois Gaudreault >>>>>> <fgaudrea...@cloudops.com> wrote: >>>>>>> Thanks Alena for the explaination. >>>>>>> >>>>>>> What is the better path to fix this on our setup? Should I wait >>>>>>> for a fix in master or should I manually run the deploydb with >>>>>>> mvn? I guess the second option won't work since I used RPM? >>>>>>> >>>>>>> Francois >>>>>>> >>>>>>> >>>>>>> On 10/8/2013, 1:47 PM, Alena Prokharchyk wrote: >>>>>>>> Ok, this is what going on - the DB upgrade procedure is >>>>>>>> different on developer's setup and when deployed using >>>>>>>> cloudstack-setup-databases >>>>>>>> >>>>>>>> On developers setup: >>>>>>>> >>>>>>>> 1) you deploy the code >>>>>>>> 2) Deploy the DB using 'mvn -P developer -pl developer -Ddeploydb'. >>>>>>>> As a part of this step, the DataBaseUpgradeChecker: >>>>>>>> >>>>>>>> * first deploys the base DB version - 4.0.0 >>>>>>>> * then checks the current version of the code, and performs the >>>>>>>> db upgrade if needed. So on master, version table looks like >>>>>>>> this after the db is >>>>>>>> deployed: >>>>>>>> >>>>>>>> mysql> select * from version; >>>>>>>> +----+---------+---------------------+----------+ >>>>>>>> | id | version | updated | step | >>>>>>>> +----+---------+---------------------+----------+ >>>>>>>> | 1 | 4.0.0 | 2013-10-08 10:34:47 | Complete | >>>>>>>> | 2 | 4.1.0 | 2013-10-08 17:35:22 | Complete | >>>>>>>> | 3 | 4.2.0 | 2013-10-08 17:35:22 | Complete | >>>>>>>> | 4 | 4.3.0 | 2013-10-08 17:35:22 | Complete | >>>>>>>> +----+---------+---------------------+----------+ >>>>>>>> 4 rows in set (0.00 sec) >>>>>>>> >>>>>>>> >>>>>>>> 3) Start management server. >>>>>>>> >>>>>>>> >>>>>>>> When deployed from rpm: >>>>>>>> >>>>>>>> 1) you deploy the code >>>>>>>> 2) run cloudstack-setup-databases. As the result of this step, >>>>>>>> 4.0.0 base version of the DB is deployed. Thats why you see >>>>>>>> only 4.0.0 record in the DB. >>>>>>>> 3) Start management server. DataBaseUpgradeChecker is being >>>>>>>> invoked as a part of it, and performs the db upgrade to the >>>>>>>> version of the current code. Only after that all the managers >>>>>>>> get invoked + system caller context get initialized. >>>>>>>> >>>>>>>> Looks like the load order for step 3) got broken recently, and >>>>>>>>system context gets initialized before the db upgrade is finished. >>>>>>>> So we either need to fix the order, or invoke >>>>>>>>DataBaseUpgradeChecker as a part of cloudstack-setup-databases >>>>>>>>so at the point when management server starts up, the DB >>>>>>>>already upgraded to the latest version. >>>>>>>> >>>>>>>> -Alena. >>>>>>>> >>>>>>>> >>>>>>>> On 10/8/13 10:25 AM, "Francois Gaudreault" >>>>>>>> <fgaudrea...@cloudops.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hmm... I just checked the DB version and it's 4.0??? It should >>>>>>>>> be >>>>>>>>> 4.3.0 no? >>>>>>>>> >>>>>>>>> mysql> select * from version; >>>>>>>>> +----+---------+---------------------+----------+ >>>>>>>>> | id | version | updated | step | >>>>>>>>> +----+---------+---------------------+----------+ >>>>>>>>> | 1 | 4.0.0 | 2013-10-08 10:58:49 | Complete | >>>>>>>>> +----+---------+---------------------+----------+ >>>>>>>>> 1 row in set (0.00 sec) >>>>>>>>> >>>>>>>>> I installed cloudstack-management-4.3.0: >>>>>>>>> [root@eng-testing-cstack_master ~]# rpm -qf >>>>>>>>> /usr/bin/cloudstack-setup-databases >>>>>>>>> cloudstack-management-4.3.0-SNAPSHOT.el6.x86_64 >>>>>>>>> >>>>>>>>> Francois >>>>>>>>> >>>>>>>>> On 10/8/2013, 1:04 PM, Francois Gaudreault wrote: >>>>>>>>>> It's a fresh master RPM install. >>>>>>>>>> >>>>>>>>>> Francois >>>>>>>>>> >>>>>>>>>> On 10/8/2013, 12:40 PM, Alena Prokharchyk wrote: >>>>>>>>>>> It is not a small issue. is_default filed was added to the >>>>>>>>>>> table as a part of the 41-42 db upgrade. Looks like the code >>>>>>>>>>> tries to retrieve system user before the db upgrade is completed. >>>>>>>>>>> >>>>>>>>>>> DB upgrade is a major part of system integrity check; no >>>>>>>>>>> queries to the DB should be made before its completed. >>>>>>>>>>> Francois, did you start seeing this problem just recently? >>>>>>>>>>> >>>>>>>>>>> -Alena. >>>>>>>>>>> >>>>>>>>>>> On 10/8/13 8:04 AM, "Francois Gaudreault" >>>>>>>>>>> <fgaudrea...@cloudops.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I compiled Master this morning, and there is a small DB issue. >>>>>>>>>>>> One field is missing in the account table (field default). >>>>>>>>>>>> CS will not start because of that. >>>>>>>>>>>> >>>>>>>>>>>> 2013-10-08 11:01:42,623 FATAL [o.a.c.c.CallContext] >>>>>>>>>>>> (Timer-2:null) Exiting the system because we're unable to >>>>>>>>>>>>register the system call context. >>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB >>>>>>>>>>>>Exception >>>>>>>>>>>>on: >>>>>>>>>>>> com.mysql.jdbc.JDBC4PreparedStatement@4c1aa2e9: SELECT >>>>>>>>>>>>account.id, account.account_name, account.type, >>>>>>>>>>>>account.domain_id, account.state, account.removed, >>>>>>>>>>>>account.cleanup_needed, account.network_domain, >>>>>>>>>>>>account.uuid, account.default_zone_id, account.default FROM >>>>>>>>>>>>account WHERE account.id = 1 AND account.removed IS NULL >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja >>>>>>>>>>>>va:98 >>>>>>>>>>>>6) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess >>>>>>>>>>>> or$In >>>>>>>>>>>> tercept >>>>>>>>>>>> orD >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.jav >>>>>>>>>>>>a:963 >>>>>>>>>>>>) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess >>>>>>>>>>>> or$In >>>>>>>>>>>> tercept >>>>>>>>>>>> orD >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja >>>>>>>>>>>>va:92 >>>>>>>>>>>>6) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> com.cloud.utils.component.ComponentInstantiationPostProcess >>>>>>>>>>>> or$In >>>>>>>>>>>> tercept >>>>>>>>>>>> orD >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>ispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>com.cloud.dao.EntityManagerImpl.findById(EntityManagerImpl.j >>>>>>>>>>>>ava:4 >>>>>>>>>>>>5) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.context.CallContext.register(CallCont >>>>>>>>>>>> ext.j >>>>>>>>>>>> ava:166 >>>>>>>>>>>> ) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.context.CallContext.registerSystemCal >>>>>>>>>>>> lCont >>>>>>>>>>>> extOnce >>>>>>>>>>>> Onl >>>>>>>>>>>> >>>>>>>>>>>> y(CallContext.java:141) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.context.CallContextListener.onEnterCo >>>>>>>>>>>> ntext >>>>>>>>>>>> (CallCo >>>>>>>>>>>> nte >>>>>>>>>>>> >>>>>>>>>>>> xtListener.java:36) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedCo >>>>>>>>>>>> ntext >>>>>>>>>>>> .callWi >>>>>>>>>>>> thC >>>>>>>>>>>> >>>>>>>>>>>> ontext(DefaultManagedContext.java:83) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedCo >>>>>>>>>>>> ntext >>>>>>>>>>>> .runWit >>>>>>>>>>>> hCo >>>>>>>>>>>> >>>>>>>>>>>> ntext(DefaultManagedContext.java:53) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnabl >>>>>>>>>>>> e.run >>>>>>>>>>>> (Manage >>>>>>>>>>>> dCo >>>>>>>>>>>> >>>>>>>>>>>> ntextRunnable.java:46) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTa >>>>>>>>>>>> sk.ru >>>>>>>>>>>> n(Manag >>>>>>>>>>>> edC >>>>>>>>>>>> >>>>>>>>>>>> ontextTimerTask.java:27) >>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:534) >>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:484) >>>>>>>>>>>> Caused by: >>>>>>>>>>>>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: >>>>>>>>>>>> Unknown column 'account.default' in 'field list' >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>sun.reflect.NativeConstructorAccessorImpl.newInstance0(Nativ >>>>>>>>>>>>e >>>>>>>>>>>> Method) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(Nativ >>>>>>>>>>>> eCons >>>>>>>>>>>> tructor >>>>>>>>>>>> Acc >>>>>>>>>>>> >>>>>>>>>>>> essorImpl.java:57) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(D >>>>>>>>>>>> elega >>>>>>>>>>>> tingCon >>>>>>>>>>>> str >>>>>>>>>>>> >>>>>>>>>>>> uctorAccessorImpl.java:45) >>>>>>>>>>>> at >>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:532) >>>>>>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) >>>>>>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386) >>>>>>>>>>>> at >>>>>>>>>>>> com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053) >>>>>>>>>>>> at >>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074) >>>>>>>>>>>> at >>>>>>>>>>>>com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006) >>>>>>>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468) >>>>>>>>>>>> at >>>>>>>>>>>>com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629) >>>>>>>>>>>> at >>>>>>>>>>>> com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedSt >>>>>>>>>>>> ateme >>>>>>>>>>>> nt.java >>>>>>>>>>>> :21 >>>>>>>>>>>> >>>>>>>>>>>> 55) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement. >>>>>>>>>>>> java:23 >>>>>>>>>>>> 18) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute >>>>>>>>>>>> Query >>>>>>>>>>>> (Delega >>>>>>>>>>>> tin >>>>>>>>>>>> >>>>>>>>>>>> gPreparedStatement.java:96) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute >>>>>>>>>>>> Query >>>>>>>>>>>> (Delega >>>>>>>>>>>> tin >>>>>>>>>>>> >>>>>>>>>>>> gPreparedStatement.java:96) >>>>>>>>>>>> at >>>>>>>>>>>> >>>>>>>>>>>>com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.ja >>>>>>>>>>>>va:98 >>>>>>>>>>>>3) >>>>>>>>>>>> ... 27 more >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Francois Gaudreault >>>>>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect >>>>>>>>>>>> fgaudrea...@cloudops.com >>>>>>>>>>>> 514-629-6775 >>>>>>>>>>>> - - - >>>>>>>>>>>> CloudOps >>>>>>>>>>>> 420 rue Guy >>>>>>>>>>>> Montréal QC H3J 1S6 >>>>>>>>>>>> www.cloudops.com >>>>>>>>>>>> @CloudOps_ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Francois Gaudreault >>>>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect >>>>>>>>> fgaudrea...@cloudops.com >>>>>>>>> 514-629-6775 >>>>>>>>> - - - >>>>>>>>> CloudOps >>>>>>>>> 420 rue Guy >>>>>>>>> Montréal QC H3J 1S6 >>>>>>>>> www.cloudops.com >>>>>>>>> @CloudOps_ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Francois Gaudreault >>>>>>> Architecte de Solution Cloud | Cloud Solutions Architect >>>>>>> fgaudrea...@cloudops.com >>>>>>> 514-629-6775 >>>>>>> - - - >>>>>>> CloudOps >>>>>>> 420 rue Guy >>>>>>> Montréal QC H3J 1S6 >>>>>>> www.cloudops.com >>>>>>> @CloudOps_ >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Francois Gaudreault >>>>> Architecte de Solution Cloud | Cloud Solutions Architect >>>>>fgaudrea...@cloudops.com >>>>> 514-629-6775 >>>>> - - - >>>>> CloudOps >>>>> 420 rue Guy >>>>> Montréal QC H3J 1S6 >>>>> www.cloudops.com >>>>> @CloudOps_ >>>>> >>
