H Miguel, Are these tools free for each dev. This might not be a restriction but would have my preference. what about redistributable parts, like classes that are part of the engine they implement?
On Wed, Mar 12, 2014 at 10:41 AM, Miguel Ferreira <mferre...@schubergphilis.com> wrote: > I did a quick scan of the two tools proposed by Rajani and here's what I > found: > > > - Liquibase: > > o With Liquibase we would need to specify the changes (in change sets) to > the database (using either XML, YAML, JSON or SQL). > > o Each change set would then be applied by the tool to a running DB , and > that would entail schema changes as well as data migration. > > o The data migration part seems especially simple, yet powerful. > > o The tool seems to offer the rollback capability for free as long as the > change sets are written using its own dialect. But if the change set is > specified directly in SQL, there is the option to define (also in SQL) the > rollback procedure. > > - Flyaway > > o Flyaway seems to be very much similar to the DB upgrade tool ACS > currently has, although more elaborate and feature rich. > > o It provides a java API, a command line tool, as well as maven, gradle, > ant and SBT support. > > o Compared with Liquibase it seems to me a more free-form-like approach > (again, similar to what exists now in ACS) > > o Incremental changes to the DB are programmed in Java and then applied by > the tool. > > o It supports upgrades to specific versions (say from 4.2 to 3.4 when the > latest is 4.0 for example), as proposed by Alex Huang. > > Both tools seem capable of improving the current DB upgrade procedure, but > they both require the change I already proposed in the way ACS developers > maintain DB related code. They both require that changes are made > incrementally and independently of each other. > > All-in-all I think Liquibase is a better tool for the job. > > Cheers, > Miguel > > From: Mike Tutkowski [mailto:mike.tutkow...@solidfire.com] > Sent: dinsdag 11 maart 2014 6:52 > To: dev@cloudstack.apache.org > Cc: Miguel Ferreira > Subject: Re: [DISCUSS] Enabling databse upgrades on master branch > > Right...I'll just quickly chime in on this, but Daan described the issue well. > > We simply do not have a way to upgrade the DB (in an automated way) in a > fine-grained enough fashion during development. This is a common problem on > many projects...not just CloudStack. > > Sometimes it's easy enough to perform a manual upgrade. In these cases, I > would encourage developers to send out an e-mail with the [DB-CHANGE] tag and > let others now how to perform such an upgrade. This will decrease the number > of times people have to destroy and re-create their environments. > > > At some point, hopefully we can have Git send out an e-mail to notify people > of schema changes. Then they will at least know not to fetch the latest until > they are ready to deal with the consequences. > > On Tue, Mar 11, 2014 at 4:18 AM, Daan Hoogland > <daan.hoogl...@gmail.com<mailto:daan.hoogl...@gmail.com>> wrote: > Alex, > > That would only work in a dev environment, would it? so running maven > you could choose to upgrade your db. At the moment when your db is at > master, how would you decide you need to upgrade beyond your prior > commit up to the present one? Also, having pulled in new code, not > upgrading the db is not an option with our present way of working as > the dao object would throw all kinds of errors. > > What Mike is seeing is due to the granularity of the upgrade being not > fine grained enough for his needs, not due to the absence of it, i > agree. Any one, developer or not that dares to upgrade more often the > by following the release schedule, thus helping us with valuable test > data, will run into this problem. > > > On Tue, Mar 11, 2014 at 11:06 AM, Alex Huang > <alex.hu...@citrix.com<mailto:alex.hu...@citrix.com>> wrote: >> I'm fine with us moving toward other tools but I do want to point this out. >> What Mike and others are seeing is NOT due to CloudStack's schema upgrade >> procedure is not incremental but the fact that incremental upgrade is >> automatically performed when CloudStack management server is rebooted. >> >> As I've said in other threads, we should just replace the automatic db >> upgrade in CloudStack with a check for the correct version and put in a pom >> step to upgrade the db so a developer can upgrade the db schema when they so >> chooses. >> >> --Alex >> >>> -----Original Message----- >>> From: Miguel Ferreira >>> [mailto:mferre...@schubergphilis.com<mailto:mferre...@schubergphilis.com>] >>> Sent: Tuesday, March 11, 2014 2:58 AM >>> To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> >>> Cc: Alex Huang >>> Subject: RE: [DISCUSS] Enabling databse upgrades on master branch >>> >>> Hi Rajani, >>> >>> Indeed I see the overlap with the previous discussion that is taking place. >>> I >>> actually tried to chip-in that discussion. >>> I do agree with you on introducing tools that can support the database >>> upgrade process, but meanwhile a small change in the way developers >>> maintain the database related code could already bring us very far. >>> >>> As of now I'm working on a separate project with the following roadmap: >>> - detect potential conflicts (and collect data about them) >>> - learn from the data collected which conflicts can we resolve >>> automatically, >>> which we cannot >>> - for the conflicts that we cannot resolve automatically, provide some >>> guidance to the operator on how proceed with the upgrade >>> >>> I'm working on a repository that is not accessible from the outside of >>> Schuberg Philis, but the intention is to move it to >>> github.com<http://github.com> ASAP. >>> >>> I'm very interested in aligning efforts with the rest of the community, and >>> I'm >>> available to help out in whatever is decided. Meanwhile, I will continue to >>> develop the ideas we have, and anyone interested in helping out with that is >>> very welcome. >>> >>> If you have any ideas on how to collaborate, please let me know. >>> >>> Cheers, >>> Miguel >>> >>> -----Original Message----- >>> From: Rajani Karuturi >>> [mailto:rajani.karut...@citrix.com<mailto:rajani.karut...@citrix.com>] >>> Sent: dinsdag 11 maart 2014 7:19 >>> To: dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org> >>> Cc: Alex Huang >>> Subject: Re: [DISCUSS] Enabling databse upgrades on master branch >>> >>> Hi Miguel, >>> >>> This is in-line with discussions related to db changes we are having at [1] >>> and >>> [2] >>> >>> I think it would be better to use existing tools like [liquibase] or >>> [flyway] >>> instead of writing a new one. A good comparison of the both is available at >>> [3]. >>> >>> Also, how do we join the efforts? Is there any design doc? Are you working >>> on a separate branch or as separate project? >>> >>> >>> [1] http://markmail.org/message/r7wv36o356nolq7f >>> [2] http://markmail.org/message/wlua3l4o5shayidf >>> [liquibase] http://www.liquibase.org/ >>> [flyway] http://flywaydb.org/ >>> [3] http://stackoverflow.com/a/8489144/201514 >>> >>> >>> ~Rajani >>> >>> >>> >>> On 10-Mar-2014, at 8:35 pm, Miguel Ferreira >>> <mferre...@schubergphilis.com<mailto:mferre...@schubergphilis.com><mailto:mferre...@schubergphilis.com<mailto:mferre...@schubergphilis.com>>> >>> wrote: >>> >>> Hi all, >>> >>> At Schuberg Philis we are interested in upgrading our running installation >>> of >>> ACS more frequently than the current release cycle. >>> To that end, we are working on tooling to detect potentially conflict >>> introducing changes to the ACS database and upgrade software. >>> By conflict introducing change I mean a change to ACS that requires a clean >>> database to start with. Thus, rendering a database of a running installation >>> useless. >>> Once we can detect the changes that introduce conflicts, we will start to >>> monitor them to better understand how to mitigate and possibly work >>> around the conflicts. >>> >>> One thing we can already foresee as problematic is the way the upgrade >>> software (SQL scripts and Java classes) are being maintained. It is part of >>> the >>> current way-of-working to make all kinds of changes to the upgrade software >>> in the master branch. Say, if a create statement was introduced in a SQL >>> script in commit C1, then the same create statement might be changed in >>> commit C2, or even further down the line. If we want to continuously >>> upgrade our running installation, we would not be able to upgrade past C1 >>> without at least losing some data. One possible way out of this problem is >>> to >>> add an alter statement in C2 instead of changing the create statement >>> introduced in C1. >>> >>> We are in favor of all changes to the upgrade software being made in such a >>> way that they can be applied independently and incrementally. We do realize >>> that this entails more effort from developers, but we also see the benefit >>> of >>> enabling continuous delivery: we basically get a shorter feedback loop on >>> the >>> quality of ACS in a real-world scenario. >>> >>> We are making all tools related to this open source, so anyone that shares >>> the same interest is welcome to join the effort. >>> Lastly, we would like to hear your opinions about the issue I described, as >>> well as, the proposed solution and any other solutions you might come up >>> with. >>> >>> >>> Kind regards, >>> Miguel Ferreira >>> >>> Mission Critical Engineer >>> Schuberg Philis >>> Boeingavenue 271 >>> 1119 PD Schiphol-Rijk >>> schubergphilis.com<http://schubergphilis.com><http://schubergphilis.com> >>> >>> +31 207506617<tel:%2B31%20207506617> >>> +31 610907106<tel:%2B31%20610907106> >>> _____________________ >>> >> > > > -- > Daan > > > > -- > Mike Tutkowski > Senior CloudStack Developer, SolidFire Inc. > e: mike.tutkow...@solidfire.com<mailto:mike.tutkow...@solidfire.com> > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play>(tm) -- Daan
