[
https://issues.apache.org/jira/browse/SOLR-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16628943#comment-16628943
]
Erick Erickson commented on SOLR-12801:
---------------------------------------
{quote}You can only meaningful run tests with BadApples off, which means test
coverage is minimal and shrinking and new problems are being added,
{quote}
Not exactly true. BadApple=true is the default, you have to deliberately
disable those tests. Of course if you do in your environment, then the above is
totally true.
What is also totally true is that the noise is such that I can break something
and never see it because I haven't waded through each and every failure in
BadApple'd tests to see if I introduced a legitimate failure in one of them.
I would be _thrilled_ if all the BadApple nonsense were no longer useful. As it
is, what it's mainly recording is whether we're annotating more than we
un-annotate as well as whether tests come and go over longer time frames. There
are tests that fail for 4 weeks in a row, then succeed for 5 weeks, then fail
for 6 weeks etc. That's the purpose of leaving the comments in when tests were
annotated/unannotated.
The BadApple thrashing is not doing much of anything towards actually _fixing_
the issues. About all it's doing is making the extent of the problem more
visible if anyone bothers to read the weekly e-mails.
I guess the net-net here is that I don't like the BadApple process much and
would be glad to stop dealing with it altogether if we define a better process.
Sounds like a hot topic at Activate ;)
> Fix the tests, remove BadApples and AwaitsFix annotations, improve env for
> test development.
> --------------------------------------------------------------------------------------------
>
> Key: SOLR-12801
> URL: https://issues.apache.org/jira/browse/SOLR-12801
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Mark Miller
> Assignee: Mark Miller
> Priority: Critical
>
> A single issue to counteract the single issue adding tons of annotations, the
> continued addition of new flakey tests, and the continued addition of
> flakiness to existing tests.
> Lots more to come.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
