On 05/04/2010, Gary Gregory <ggreg...@seagullsoftware.com> wrote: > > -----Original Message----- > > From: sebb [mailto:seb...@gmail.com] > > Sent: Monday, April 05, 2010 11:17 > > To: Commons Developers List > > Subject: Re: [codec][lang] Provide a test jar plus [daemon] > > > > > On 05/04/2010, Gary Gregory <ggreg...@seagullsoftware.com> wrote: > > > I do not think Clirr is enough as exemplified by the change of behavior > in > > Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.) > > > > Agreed - Clirr is necessary, but not sufficient. > > > > > There is no substitute for running unit tests. > > > > Agreed; however unit tests may not pick up changes found by Clirr. > > > > Ideally the tests should exercise all public APIs, but that is not > > always possible - and rarely achieved ;-) > > > > > As we change code in a component, unit tests are created or changed. It > is > > possible to create compatibility problems when code and unit tests change. > You > > have to consciously and carefully leave some tests as-is from a previous > > versions to ensure backward compatibility. Sometimes of course, you are > fixing > > bugs, so unit tests must change too. > > > > > > If you run the previous version's test against the current code base, > you > > are simulating what users do when they drop in a new jar in their > application. > > Since we do not have previous version's tests next to the current tests, > how > > can we test backward compatibility? If we produce a test jar file for each > > release and keep it in SVN and deliver it, that is a snapshot of old > "clients" > > for a given version. The test phase of the build can then run all of the > tests > > in each test jar, including the test jar for the current tests, against > the > > component jar. > > > > > > > I agree, it would be useful to be able to run old tests against the new > code. > > However I'm not sure that the jars should be stored in SVN. Perhaps in > > a local snapshot repo instead? > > > Where do you get the test jars initially? It would painful to have to > download all releases and build your own test jars. If the point is to test > backward compatibility, then we need to keep the backward bits someplace. >
The test jars can be built as part of releases going forward. I just don't think that they belong in SVN. > > > > > If there are a lot of bug fixes in the release, then a lot of old > > tests may fail, and it could get tedious working out which failures > > are OK and which are not. This is not to say that the tests should not > > be run, just that they may involve a lot of manual checking. > > > Right, that would be done and documented once per release, in some XML format > that can be used in a report: "Test T in class C is expected to fail because > x y z." That one report would have to be carefully compiled and checked. > The test process might need a test POM to run the tests; perhaps document the exceptions there? > > Gary > > > > > > > Gary > > > > > > > > > > -----Original Message----- > > > > From: Matt Benson [mailto:gudnabr...@gmail.com] > > > > Sent: Monday, April 05, 2010 10:00 > > > > To: Commons Developers List > > > > > > > Subject: Re: [codec][lang] Provide a test jar plus [daemon] > > > > > > > > Is it not sufficient to simply run clirr reports before a release? > > > > > > > > -Matt > > > > > > > > On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote: > > > > > > > > > Seeing the discussion about [daemon] and not releasing made me > > > > > think of another use for a test jar file. > > > > > > > > > > What I would like to know when evaluating an RC for releasing a > > > > > maintenance of a commons component (from x.y.n to x.y.n+1) is that > > > > > there is 100% binary compatibility. > > > > > > > > > > As part of the build I would run (at least) the 1.0.2 unit tests > > > > > against the 1.0.3 RC. If 100% pass all is well, if not, it is > > > > > either a bug or a known acceptable failure fixing a bug and should > > > > > be documented somehow, probably in a ticket. > > > > > > > > > > This would mean that a release 1.0.3 RC would include foo- > > > > > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo- > > > > > test-1.0.1.jar, hm... > > > > > > > > > > Thoughts? > > > > > > > > > > Gary Gregory > > > > > Senior Software Engineer > > > > > Seagull Software > > > > > email: ggreg...@seagullsoftware.com > > > > > email: ggreg...@apache.org > > > > > www.seagullsoftware.com > > > > > > > > > > > > > > > From: Gary Gregory > > > > > Sent: Tuesday, March 30, 2010 16:58 > > > > > To: Commons Developers List > > > > > Subject: [codec][lang] Provide a test jar > > > > > > > > > > I am starting with codec and lang since it what I am most > > > > > interested in ATM... > > > > > > > > > > I would like to run commons.xxx unit tests as part of my build as a > > > > > sanity check when I try out a new combo of JVM, OS, jars, etc. > > > > > > > > > > Right now, I would have to compile the unit tests as part of my > > > > > build which is not great. > > > > > > > > > > So, what about providing a commons-codec-1.5-test.jar file like we > > > > > provide a sources and javadoc file? > > > > > > > > > > Same for any commons project really... > > > > > > > > > > Thoughts? > > > > > > > > > > Gary Gregory > > > > > Senior Software Engineer > > > > > Seagull Software > > > > > email: ggreg...@seagullsoftware.com > > > > > email: ggreg...@apache.org > > > > > www.seagullsoftware.com > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
