This is a prompt for Cassandra developers to discuss the alternatives
and let Test Engineering know what you desire.
As discussed a few times in person, on irc, etc., there are a couple
different ways we can run tests in Jenkins, particularly
cassandra-dtest. The Cassandra developers are the committers to unit
tests, so Test Engineering runs whatever is in the branch. If you'd like
to make changes to unit tests to make things blue, just commit those!
Currently, we run dtests as 1), but we could do 2):
1) Run all dtests that don't catastrophically hang a server, pass or
fail, and report the results.
2) Run only known passing dtests, skipping anything that fails - make it
all blue on the main branch builds.
The biggest benefit is that dev branch builds should be easily
recognizable as able to merge, if the dtest run is passing and blue.
There is no comparison with the main branch build needing interpretation.
Test Eng has recently added the ability run *only* the skipped tests and
has a a prototype job, trunk_dtest-skipped-with-require, to dig through.
This could be set up for all main branch builds, moving anything that
doesn't pass 100% to the -skipped job. This is perhaps the drawback with
2) above: we're simply not going to run all the dtests on your dev
branch. I don't think it makes sense to set up a -skipped dtest job on
your dev branches. In addition, there's another job result set to go
look at to properly evaluate the true state of a Cassandra branch or
release. There may be other side effects - feel free to chime in.
I'm on a "disconnected" holiday until Monday Aug 24, so I won't have a
chance to check in until then - the Test Eng team can field questions or
clarifications, if needed.
--
Warm regards,
Michael
