You don’t need to modify the runner to re-run tests on the same VM. You can
have a short mix task that looks something like this:
def rerun() do
for _ <- 0..3 do
Mix.Task.run(:test)
Code.unrequire_files(Code.required_files())
end
end
However, I also can’t think of any race condition that couldn’t be reliably
reproduced by running the tests on a different VM instance with the same
seed but would be reliably reproduced by running the tests in the same VM,
and setting the seed is already available behavior.
Austin Ziegler <[email protected]> schrieb am Mi. 21. Okt. 2020 um 18:32:
> I don’t really have a strong opinion on this feature (it feels like it
> would be easy to do on a per-suite or per-test basis by modifying the test
> code, if tedious), but I think that a lot of the counter-cases are built on
> a fundamental misunderstanding of lee’s use case.
>
> Basically, if something isn’t going to help users of the tool in the long
>> run, it shouldn’t be made easier to do. Yes, it is time intensive _now_ to
>> find and fix things difficult situations like race conditions, but by
>> ignoring them you’re both going to make it harder to find and fix later on,
>> as well as decreasing the quality of your tests until they are properly
>> fixed.
>
>
> The use case presented here is *specifically* to permit the repeated
> execution of a test, test suite, or all suites N number of times to see if
> a failure due to a race condition that does not express itself in a single
> run (which every separate invocation of `mix test` is) can be produced by a
> repeated run. This could be done with a command-line parameter or some
> other mechanism (although other mechanisms would need to wrap either a
> `test` or `describe` block explicitly, and would have no way of repeating
> all suites in the event of some sort of race condition between multiple
> async tests across the suite.
>
> That is, lee’s looking for assistance from ExUnit (or more specifically
> the `mix test` runner of ExUnit) to figure out what is causing a race
> condition that is appearing occasionally in their tests or perhaps more
> frequently in production.
>
> It’s not a bad request, because the shell script repeater approach
> provides a clean VM every time and the in-line code repeater approach
> doesn’t permit cross-suite checking in the event that it really *is* a
> weird interaction issue. Repeating all tests N times in the same VM can
> *only* be done by modifying the runner (and provides the benefit of
> working for test-and-suite repeats when a suite file or test line are
> provided); repeating suite or tests can be done by wrapping the `describe`
> or `test` blocks in loops.
>
> -a
>
> On Wed, Oct 21, 2020 at 10:54 AM Devon Estes <[email protected]>
> wrote:
>
>> Making tools that folks can easily misuse does indeed damage the greater
>> community, in that it can harm the perception of that tool (or In this
>> case, language), which in turn hinders its adoption.
>>
>> There are already existing examples of this sort of choice in ExUnit
>> where restrictions are put on APIs to help guide users towards decisions
>> that are more sustainable in the long run, like not allowing more than one
>> level of nesting with the ‘describe/2’ macro.
>>
>> Basically, if something isn’t going to help users of the tool in the long
>> run, it shouldn’t be made easier to do. Yes, it is time intensive _now_ to
>> find and fix things difficult situations like race conditions, but by
>> ignoring them you’re both going to make it harder to find and fix later on,
>> as well as decreasing the quality of your tests until they are properly
>> fixed.
>>
>> On Wed, Oct 21, 2020 at 4:11 PM lee eggebroten <[email protected]>
>> wrote:
>>
>>> It's Austin's case that's I'm trying to address:
>>> - It's often time-intensive uncovering where a race condition is
>>> occurring. Not to "prove" that it IS fixed 'cause that's just dumb.,
>>>
>>> The counter cases feel very ... elitist and controlling
>>> - "I think you might abuse the tool so I'm not going to let you have it".
>>> - Will someone else's "misuse" damage you in some way?
>>>
>>> -lee
>>>
>>>
>>>
>>> On Tuesday, October 20, 2020 at 3:08:34 PM UTC-6 [email protected] wrote:
>>>
>>>> I agree with Shane.
>>>>
>>>> If you want to do this in your test suite (without a clean VM), I'd
>>>> recommend creating an Elixir library that abstracts the `1..rerun_count |>
>>>> Enum.each(current_test_function)` functionality that you require.
>>>> I think that would be a better way to allow people that *really* want
>>>> it to use this technique,
>>>> while not implicitly encouraging people to resort to this logic in the
>>>> way it would do when it were added as a built-in ExUnit feature.
>>>>
>>>> ~Marten
>>>> On 20-10-2020 22:45, Austin Ziegler wrote:
>>>>
>>>> I agree that this is fairly easy to do in shell script, but that
>>>> results in a clean VM every time. This is intended to find that rare case
>>>> where an error only shows up with repeated executions on the VM, not as a
>>>> “run this repeatedly until it works”. It’s more of a “run this until it
>>>> breaks, up to N times”.
>>>>
>>>> I’m not sure that this is substantially better than modifying the test
>>>> suite in this case, but it’s not a bad idea.
>>>>
>>>> On Tue, Oct 20, 2020 at 12:25 PM [email protected] <
>>>> [email protected]> wrote:
>>>>
>>>>> I feel this is better addressed through simple shell automation, which
>>>>> has probably a dozen different ways to solve for this. Blessing the idea
>>>>> of
>>>>> "just try again" as an actual ExUnit feature implicitly encourages
>>>>> community/developer tolerance of this kind of circumstance when all
>>>>> practical efforts should be made to eliminate it from the code first,
>>>>> wherever the problem actually originates. If those efforts fail, then fall
>>>>> back on surrounding it with retry logic on a case-by-case basis. I would
>>>>> personally discourage use of such a feature if it existed.
>>>>>
>>>>> Shane Sveller
>>>>>
>>>>> On Monday, October 19, 2020 at 9:30:34 PM UTC-5 [email protected]
>>>>> wrote:
>>>>>
>>>>>> *Problem*:
>>>>>> for heisnbugs and race conditions it is useful to rerun a set of
>>>>>> tests a number of times.
>>>>>> Currently you must alter the suite(s) to wrap the the test(s) in
>>>>>> `Enum`s which is obvioiusly tedious
>>>>>>
>>>>>> *Naive Implementation Proposal*:
>>>>>> Add a `--rerun-count` command line option that would default to 1 but
>>>>>> wrap the test run with `1..<rerun-count> |> Enum(
>>>>>> current_test_runner_parent_function )`
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elixir-lang-core" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elixir-lang-core/3581cdf4-4d23-41d1-afaa-151995e450f8n%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/3581cdf4-4d23-41d1-afaa-151995e450f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>
>>>>
>>>> --
>>>> Austin Ziegler • [email protected] • [email protected]
>>>> http://www.halostatue.ca/ • http://twitter.com/halostatue
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elixir-lang-core" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQvmO_sO-QuqbxAbaOZc_Y8PVx7t64PFWU40YSk4ttordg%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQvmO_sO-QuqbxAbaOZc_Y8PVx7t64PFWU40YSk4ttordg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elixir-lang-core" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elixir-lang-core/00dff9ce-fa83-4d20-8b98-d2ef1cb66ae5n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/elixir-lang-core/00dff9ce-fa83-4d20-8b98-d2ef1cb66ae5n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>>
>> _________________
>> Devon Estes
>> +49 176 2356 4717
>> www.devonestes.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>>
> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgrLRT7_njz3673jqPM7a_nLQO4UPFesJenfq3S_zL-Mw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgrLRT7_njz3673jqPM7a_nLQO4UPFesJenfq3S_zL-Mw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> Austin Ziegler • [email protected] • [email protected]
> http://www.halostatue.ca/ • http://twitter.com/halostatue
>
> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuj8g8Eb8eReKQS9gTKa8d%2BC8HLC%2B0H9k%3DzGBJxqZDGTA%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuj8g8Eb8eReKQS9gTKa8d%2BC8HLC%2B0H9k%3DzGBJxqZDGTA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
_________________
Devon Estes
+49 176 2356 4717
www.devonestes.com
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgOoYTj8Y9kBapEo-ZcxAt0w3DbbVj0ZXzCs9WPGO9x_w%40mail.gmail.com.
