Feedback inline below. On Fri, Nov 21, 2008 at 6:22 AM, Jonathan Oulds <[EMAIL PROTECTED]> wrote:
> You've covered quite a few points in this e-mail, let me try to summarise > and then deal with each one in turn. First of all let me say thank you for > taking the time to write such an informative mail. > > 1- Tag profusion. The polluting of the SVN repository with tagged commits > from successful CI builds. Your suggestion is not to tag, but to instead > use the SVN version number as the build number in the ivy file. > > To be honest I was worried about our build 'n' tag approach, but the > newness of our system meant that we have never experienced the "unwieldy" > tag folders that you mentioned. So taking your advice I think moving to a > system of linking against the SVN repository version is the best way > forward. > > 2- Linking source (SVN) and artifact (Ivy) versions for reproducible > builds. This you achieve by storing a resolved ivy file and the > corresponding SVN version number (held in a separate file) in another > location possibly a separate SVN repository. > As noted earlier, Shawn's technique of putting the SVN version directly in the ivy.xml looks like an improvement over my suggestion of maintaining a separate properties file with a single property. > > Whilst I like the idea of storing the SVN version number with the ivy file, > I don't see how this could work if the resolved ivy file and SVN source > version file are themselves versioned. You would still require a version > number to extract the correct version of the SVN source version file. > I *don't think *you need to keep a hold of the SVN version of the ivy.xml to serve as a hook for getting to it later, but let me try to follow along with what I think you're saying. Suppose you have two Subversion repositories: * DevRepo: the actual source for various projects * IvyRepo: The Ivy repository with its whole folder hierarchy, but only the published ivy.xml files, not the published artifacts, which you can always rebuild. (We'll set aside how IvySvn might affect this.) The only reason you need the IvyRepo SVN repository is in case your Ivy repository gets deleted or corrupted. If you're taking the "create a new version on every successful CI build" approach, then IvyRepo will behave very much like the content of a tags directory. That is, under normal circumstances the ivy.xml files and their parent directories will be created once and never updated. Anyway, if the only time you're going to need to get a hold of an ivy.xml from source control is if something happens to your Ivy repository, then at that point, you're likely going to re-create the whole repository. But even if you just need a particular ivy.xml file, the thing you're going to use as your hook or input is the project/Ivy revision number combination, e.g. customerportal-1.1.252, not a global SVN revision number. BTW, this idea of putting your Ivy repository in source control, but only for the ivy.xml files, lends itself to the dual resolver<http://ant.apache.org/ivy/history/trunk/tutorial/dual.html>, where you look for your ivy.xml files in one place and your artifacts in another. > > 3- Subsequent issues with dependency management. You suggest hard-coding > depandancies into the ivy file to link to specific versions of each module. > > I must admit I am sceptical about hard-coding dependency versions into ivy > files, firstly I don't trust other developers to enter the correct version > (I of course would always enter the correct version). I also believe that > most integration builds will be the result of bug fixes etc. or minor > implementation changes i.e. non-breaking changes. Any change to a module > that breaks its API should also include a change to its major or minor > version, not just the build number. I don't think I articulated this clearly enough. I wasn't talking about developers *manually *hardcoding explicit dependencies. I was talking about the CI-invoked build process taking care of that *automatically*. The ivy.xml for customerportal *in trunk in source control *might have the following dependency: <dependency org="theteam" name="customerapplication" rev="*1.0.+*" /> I remember, though, with the way things worked on my previous team, if you looked at the ivy.xml in customerportal/tags/1.1.252 (not trunk), you would see: <dependency org="theteam" name="customerapplication" rev="*1.0.92*" /> It is the build process that is automatically hardcoding this dependency. I believe it was an Ivy Ant task doing this--that we didn't have to write a special Ant target to accomplish this--but I don't recall for sure. If you're going to go with the promiscuous versioning approach, it only makes sense that the Ivy repository ivy.xml for customerportal 1.1.252 should express a dependency on the particular customerapplication 1.0.92, rather than customerapplication 1.0.+. > > From reading your post and that of Shawn Castrianni I've come to a few > conclusions which I can hopefully implement and should enhance our > experience with ivy. > > 1- No more tagging of CI builds, instead link to the SVN version via the > <Repository> element of the <Info> tag. > > 2- Let the CI auto increment the build number, there is no need for this to > be the SVN version, so a simple increment should suffice. > > 3- A module should have a dynamic version number specified by a properties > file that gives the major and minor version, this file is stored under SVN. > The build number is generated by CI using the ant buildnumber task. This > gives developers control over the major and minor version number when they > make a breaking change to a modules API, but ensures integration builds get > a distinct version number. > Actually, I was making the case for something very different from this--that, if you really want to rely on the *buildnumber *Ant task, *then your project in source control should be oblivious of its own version*. To be honest, this sounds like a radical proposal, although I'm really just trying to follow things to their logical conclusion. If a developer wants to increment the major version, they would call something like: ant increment-major The implementation of this target would not read a properties file in the project source; it would use *buildnumber *to divine the current version and then do some hocus-pocus to increment the major version and reset the minor version and build number (assuming your versioning goes three levels deep). Now, if anyone knows what that hocus-pocus step ought to be, I'd love to hear. Now, that said, I'm not necessarily saying that keeping track of current major and minor version number in a properties file in source is a bad thing. It does smack of being a little bit homemade. Then again, if there's no built-in facility to increment major or minor version from the *buildnumber *output, then that implementation smacks of being homemade too. I'd be curious to hear how anyone out there has worked out the incrementing of major and minor versions. Of course, there's always the easy way out, which is to just pass in the desired new version as an argument. Like so: ant ivy-publish 2.0 Not very foolproof, you must admit. > 4- Dependencies should either be constrained by "latest.integration" or if > a CI build fails due to an incompatible API then by [major.minor, > major2.minor2] > > As usual feedback is most welcome. > > > Mitch Gitman wrote: > >> *Warning: really long post.* >> >> >> Jonathan, I can give some insight based on my experience. >> >> First off, just for perspective, the top-level decision at play is how to >> deal with integration versions, as covered in Best Practices. The two >> choices are: >> >> - "use a naming convention like a special suffix," i.e. do something >> like >> Maven snapshots >> - "create automatically a new version for each" >> >> The Best Practices page recommends the second, more "promiscuous" >> approach. >> I would beg to differ, for reasons that go beyond what we're discussing on >> this thread, although I acknowledge it's a valid, legitimate approach. And >> well, since this is the tack you've chosen, let's go with it. >> >> Actually, I previously worked on a team that took this second approach and >> published a new Ivy version on every successful CI build. So what we did >> then was tag promiscuously in Subversion. Suppose, for example, that our >> CI >> system built version 1.1.252 of a customerportal project. It would then do >> an svn/copy to create the following: >> * customerportal >> * tags >> * customerportal-1.1.252 >> >> For discussion's sake, let's assume we're talking about Subversion. Now, I >> know there's no one right way to use Subversion, even if you're following >> Subversion's layout conventions. The conventional way--if not the right >> way--to use the tags directory is to have it there to capture >> milestone/release builds. In the online Subversion book, "Chapter 4. >> Branching and Merging," under the "Common Branching Patterns" section, is >> a >> discussion of "Release Branches:" >> http://svnbook.red-bean.com/en/1.1/ch04s04.html#svn-ch-4-sect-4.4.1 >> >> Relevant passage: "The branch is tagged and released. When testing is >> complete, /branches/1.0 is copied to /tags/1.0.0 as a reference snapshot. >> The tag is packaged and released to customers." >> >> Now, I realize that *svn/copy* does a kind of virtual diff copy, so it's >> not >> like the profusion of tags with promiscuous tagging is gobbling up disk >> space. However, I can attest that the tags directory becomes quite >> unwieldy >> to negotiate. Also, unless you create a separate directory for traditional >> release tags, your release tags end up getting commingled with all the "CI >> successful build and Ivy publish" tags. All this strikes me as overkill >> and >> excessive complexity. >> >> So within the context of "Ivy-version promiscuously," there is this >> second-level decision: >> >> - Tag promiscuously in version control. >> - Do something else. >> >> The question is, what's that something else? >> >> One possible alternative to tagging promiscuously, while still >> Ivy-versioning promiscuously, is to rely on the global repository version >> Subversion produces every time there's a commit. The CI build could invoke >> the *svnversion *command, although there's no telling if the result is the >> actual version for that particular project's commit, and not a subsequent >> version for another project's commit that happened right after. (I'm not >> sure how the global repository version could be captured on the commit in >> a >> way that could be picked up by the CI server.) Anyway, the Subversion >> version could be published in its own properties file as an extra artifact >> to the Ivy repository. >> >> Now, you're still not out of the woods yet. >> >> Suppose customerportal 1.1 depends on customerapplication 1.0.+, which in >> turn depends on customerdomain 2.3.+. >> >> Here's what we would have done on my previous team. In the ivy.xml file >> that >> gets committed to customerportal/tags/customerportal-1.1.252, we would >> hardcode a dependency for the particular build of customerapplication, say >> 1.0.92. Then if you go and look in >> customerapplication/tags/customerapplication-1.0.92, there would be a >> dependency on, say, customerdomain 2.3.147. I forget now if we actually >> captured these hardcoded dependencies in the Ivy repository, or if we just >> left them like 1.0.+ and 2.3.+. I believe we did capture this. Really, we >> would have had to. >> >> Now, if you don't tag promiscuously, and instead rely on the global >> Subversion repository version, the question is, how do you reproduce the >> in-house (as opposed to third-party) dependencies of customerportal >> 1.1.252? >> How do you achieve reproducible builds? >> >> Well, customerportal-1.1.252 in the Ivy repository could have its >> properties >> file with its global Subversion version. Call it svnversion.properties. >> Inside it's just one line: >> svnversion=8642 >> >> And customerportal 1.1.252's Ivy file could show a hardcoded dependency on >> customerapplication 1.0.92. Then if you go into >> customerapplication-1.0.92 >> on the Ivy repository, you would see that it has a properties file with >> its >> own global Subversion version. And its Ivy file has a hardcoded dependency >> on customerdomain 2.3.147. And then under the customerdomain-2.3.147, >> you'd >> see something analogous. Recursively. >> >> Now, this in itself is not giving you reproducible builds. What I'm >> describing is navigability, not reproducibility. I mean, what happens if >> the >> Ivy repository gets blown away? On first glance, you could keep the whole >> Ivy repository in source control. I hope we can all agree that, if you're >> doing a new Ivy version on every successful CI build, storing published >> in-house artifacts in source control is a really, really bad practice. I >> mean, that will become untenable in no time. >> >> Here's what I believe the answer is to this problem (though I'd love to >> hear >> other answers): you store two things things from the Ivy repository in >> version control (preferably in its own distinct Subversion repository, >> separate from the source repository): >> >> - The published ivy.xml file with the hardcoded dependencies. >> - The properties file with the global Subversion version. >> >> At this point, you have all the information you need to re-create the >> entire >> Ivy repository, and to reproduce any given build. Granted, the deeper you >> have to go in your in-house dependency hierarchy, the more work you have. >> I'm sure some of this could be automated. >> >> (Yes, I am familiar with IvySvn <http://code.google.com/p/ivysvn/>. >> Whether >> >> your build system is accessing Subversion directly as an Ivy repository is >> kind of a side question.) >> >> You see, there are all sorts of version-control implications to this >> "version everything" approach, and I feel like they were glossed when that >> approach was recommended. Just speaking from my own experience here. >> >>
