> -----Original Message----- > From: sebb [mailto:seb...@gmail.com] > Sent: Monday, April 05, 2010 11:17 > To: Commons Developers List > Subject: Re: [codec][lang] Provide a test jar plus [daemon] > > On 05/04/2010, Gary Gregory <ggreg...@seagullsoftware.com> wrote: > > I do not think Clirr is enough as exemplified by the change of behavior in > Base64 encoding with CRLFs between version 1.3 and 1.4 (see CODEC-94.) > > Agreed - Clirr is necessary, but not sufficient. > > > There is no substitute for running unit tests. > > Agreed; however unit tests may not pick up changes found by Clirr. > > Ideally the tests should exercise all public APIs, but that is not > always possible - and rarely achieved ;-) > > > As we change code in a component, unit tests are created or changed. It is > possible to create compatibility problems when code and unit tests change. You > have to consciously and carefully leave some tests as-is from a previous > versions to ensure backward compatibility. Sometimes of course, you are fixing > bugs, so unit tests must change too. > > > > If you run the previous version's test against the current code base, you > are simulating what users do when they drop in a new jar in their application. > Since we do not have previous version's tests next to the current tests, how > can we test backward compatibility? If we produce a test jar file for each > release and keep it in SVN and deliver it, that is a snapshot of old "clients" > for a given version. The test phase of the build can then run all of the tests > in each test jar, including the test jar for the current tests, against the > component jar. > > > > I agree, it would be useful to be able to run old tests against the new code. > However I'm not sure that the jars should be stored in SVN. Perhaps in > a local snapshot repo instead?
Where do you get the test jars initially? It would painful to have to download all releases and build your own test jars. If the point is to test backward compatibility, then we need to keep the backward bits someplace. > > If there are a lot of bug fixes in the release, then a lot of old > tests may fail, and it could get tedious working out which failures > are OK and which are not. This is not to say that the tests should not > be run, just that they may involve a lot of manual checking. Right, that would be done and documented once per release, in some XML format that can be used in a report: "Test T in class C is expected to fail because x y z." That one report would have to be carefully compiled and checked. Gary > > > Gary > > > > > > > -----Original Message----- > > > From: Matt Benson [mailto:gudnabr...@gmail.com] > > > Sent: Monday, April 05, 2010 10:00 > > > To: Commons Developers List > > > > > Subject: Re: [codec][lang] Provide a test jar plus [daemon] > > > > > > Is it not sufficient to simply run clirr reports before a release? > > > > > > -Matt > > > > > > On Apr 5, 2010, at 11:16 AM, Gary Gregory wrote: > > > > > > > Seeing the discussion about [daemon] and not releasing made me > > > > think of another use for a test jar file. > > > > > > > > What I would like to know when evaluating an RC for releasing a > > > > maintenance of a commons component (from x.y.n to x.y.n+1) is that > > > > there is 100% binary compatibility. > > > > > > > > As part of the build I would run (at least) the 1.0.2 unit tests > > > > against the 1.0.3 RC. If 100% pass all is well, if not, it is > > > > either a bug or a known acceptable failure fixing a bug and should > > > > be documented somehow, probably in a ticket. > > > > > > > > This would mean that a release 1.0.3 RC would include foo- > > > > test-1.0.2.jar. And maybe also foo-test-1.0.0.jar and foo- > > > > test-1.0.1.jar, hm... > > > > > > > > Thoughts? > > > > > > > > Gary Gregory > > > > Senior Software Engineer > > > > Seagull Software > > > > email: ggreg...@seagullsoftware.com > > > > email: ggreg...@apache.org > > > > www.seagullsoftware.com > > > > > > > > > > > > From: Gary Gregory > > > > Sent: Tuesday, March 30, 2010 16:58 > > > > To: Commons Developers List > > > > Subject: [codec][lang] Provide a test jar > > > > > > > > I am starting with codec and lang since it what I am most > > > > interested in ATM... > > > > > > > > I would like to run commons.xxx unit tests as part of my build as a > > > > sanity check when I try out a new combo of JVM, OS, jars, etc. > > > > > > > > Right now, I would have to compile the unit tests as part of my > > > > build which is not great. > > > > > > > > So, what about providing a commons-codec-1.5-test.jar file like we > > > > provide a sources and javadoc file? > > > > > > > > Same for any commons project really... > > > > > > > > Thoughts? > > > > > > > > Gary Gregory > > > > Senior Software Engineer > > > > Seagull Software > > > > email: ggreg...@seagullsoftware.com > > > > email: ggreg...@apache.org > > > > www.seagullsoftware.com > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
