[
https://issues.apache.org/jira/browse/SOLR-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384103#comment-16384103
]
Erick Erickson commented on SOLR-12028:
---------------------------------------
Well, the process is evolving. I saw no use in keeping a separate open JIRA
that simply said "this test doesn't succeed very often". I thought that
pointing people at the uber-JIRA would provide more context. Whether that's the
right call or not is an open question.
SOLR-7736: Right, I linked it under "related to" rather than "duplicates", we
can add a duplicates link in if you'd like. I put in both JIRAs to allow a
complete record, if it's too confusing we can remove, say, the 7736 reference.
As for AwaitsFix, the theory (which I may not have adhered to completely, feel
free to correct) is that when something is actually known to be broken, i.e.
code we can point to and say "this code is broken" doesn't need to be run
until, well, the code is fixed. This would include dependent libs, code where
the root cause is known but not fixed and the like. AwaitsFix aren't run
periodically so disappear from sight.
BadApple, OTOH, is "this test doesn't run reliably, we have no clue why". These
are also run periodically for the various reports to be able to see. That way
they don't get lost.
It looked to me like AwaitsFix and BadApple were applied somewhat
interchangeably so this is an attempt to bring regularity to the annotations.
I reconstructed SOLR-12016 under SOLR-12037 BTW.
bq: so now this test is running for all developers
As it should IMO unless and until we do one of the following
1> decide it's just a crappy test and remove it
2> identify the root problem is and move it to AwaitsFix and link it to a JIRA
with the RCA.
3> fix it and remove the annotation altogether.
Note that for <3>, I'm starting by commenting out the annotation and providing
the date it was commented out on the theory that if the test starts failing
again it would be good information to have. OTOH if we look at this in 6 months
and it hasn't failed in that time, the commented-out annotation can be removed.
My two main goals are:
1> surface any tests that fail regularly with no RCA for reports etc.
2> get clean runs with BadApple=false to catch regressions
I don't particularly care what the mechanics are. You'll see, every Sunday, a
report to the dev list for the current BadApple and AwaitsFix annotations, file
and method. This should complement your reports (which are way cool BTW).
So that's all the theory, I expect a few things to shift around before it all
settles out...
> BadApple and AwaitsFix annotations usage
> ----------------------------------------
>
> Key: SOLR-12028
> URL: https://issues.apache.org/jira/browse/SOLR-12028
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Tests
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
> Attachments: SOLR-12016-buildsystem.patch,
> SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch
>
>
> There's a long discussion of this topic at SOLR-12016. Here's a summary:
> - BadApple annotations are used for tests that intermittently fail, say < 30%
> of the time. Tests that fail more often shold be moved to AwaitsFix. This is,
> of course, a judgement call
> - AwaitsFix annotations are used for tests that, for some reason, the problem
> can't be fixed immediately. Likely reasons are third-party dependencies,
> extreme difficulty tracking down, dependency on another JIRA etc.
> Jenkins jobs will typically run with BadApple disabled to cut down on noise.
> Periodically Jenkins jobs will be run with BadApples enabled so BadApple
> tests won't be lost and reports can be generated. Tests that run with
> BadApples disabled that fail require _immediate_ attention.
> The default for developers is that BadApple is enabled.
> If you are working on one of these tests and cannot get the test to fail
> locally, it is perfectly acceptable to comment the annotation out. You should
> let the dev list know that this is deliberate.
> This JIRA is a placeholder for BadApple tests to point to between the times
> they're identified as BadApple and they're either fixed or changed to
> AwaitsFix or assigned their own JIRA.
> I've assigned this to myself to track so I don't lose track of it. No one
> person will fix all of these issues, this will be an ongoing technical debt
> cleanup effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
