> Can you please expand on how the files would be used compared to directly
running the stale tests?
Initially I was thinking of being able to do something like:
1. At one point in time: `mix test --get-stale`
2. Later: `mix test --get-stale`
3. Diff the two outputs, and only run the tests that appear in both?
(now that I type this out, I realize this probably doesn't help)
This might be a case of the "XY problem"; perhaps what I want is not "print
the list of tests that `mix test --stale` would run", but something like
"given the current state of the code, and these new code/module changes,
give me the list of tests that should be considered 'stale' and should
rerun".
Here's more of a typical business scenario I envision:
1. I have a project branch `main` in which CI runs all tests (say, 1000
tests) on every PR merge.
1. Until then, CI should only run a subset of tests on each commit to
a new PR.
2. On `main`, I run `mix test --stale` locally, and all 1000 tests pass.
1. Ideally I could cache/snapshot this "state" or point in time, or
CI could generate it directly on `main`.
3. I make a new branch `feature/a`, make code changes, add a couple new
tests.
4. I run `mix test --stale` locally, lets say that only 100 tests need
to run based on the code changed.
5. I submit a PR.
6. *Current setup:* CI tries to run `mix test --stale` but ends up
running all 1000 tests again instead of just the 100 whose code/state was
actually "stale".
7. *Preferred setup:* CI runs `mix test --stale` (or something else) and
only reruns tests that would be considered stale after my commits were made.
8. PR merges into `main`, and just in case all 1000 tests are run again.
#14393 definitely looks interesting and I'll keep an eye on it!
On Saturday, May 10, 2025 at 12:46:38 AM UTC-7 José Valim wrote:
> Can you please expand on how the files would be used compared to directly
> running the stale tests?
>
> Note there is also https://github.com/elixir-lang/elixir/issues/14393,
> but it is currently unclear if it could be used for this purpose. I asked
> Andrea to add more context, I recommend following the discussion there too.
>
>
>
>
> *José Valimhttps://dashbit.co/ <https://dashbit.co/>*
>
>
> On Sat, May 10, 2025 at 00:20 'Noah Betzen' via elixir-lang-core <
> elixir-l...@googlegroups.com> wrote:
>
>> Currently `mix test --stale` will do some checks for code that has been
>> updated since the last test suite run. Relevant code:
>> https://github.com/elixir-lang/elixir/blob/4fa224099730f59b19912e605a888bab68da6e5b/lib/mix/lib/mix/compilers/test.ex#L119-L176
>>
>> For the purposes of larger Elixir projects and CI, it'd be nice to fetch
>> the list of tests that need to be rerun but not rerun them in the same
>> command.
>>
>> This has partially been brought up before:
>> https://elixirforum.com/t/is-there-a-way-to-predetermine-which-tests-and-why-would-be-run-on-mix-test-stale/48325
>>
>> The most basic implementation idea I had for this was `mix test
>> --print-stale` which prints all currently stale tests to stdout (or to a
>> file?).
>>
>> Another possibly nice feature would be the ability to provide a commit
>> hash similar to Credo's git diff
>> https://hexdocs.pm/credo/diff_command.html that shows which tests are
>> "stale" between two points in time. The usefulness of this idea may be
>> limited.
>>
>> tldr I want to be able to configure my CI to not run ALL tests on every
>> single PR. There are other ways to do this, but providing the data that
>> `mix test --stale` uses to determine staleness seems like a great start!
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elixir-lang-co...@googlegroups.com.
>> To view this discussion visit
>> https://groups.google.com/d/msgid/elixir-lang-core/906a55e5-1b0d-4f4b-813e-43d5afd35b46n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elixir-lang-core/906a55e5-1b0d-4f4b-813e-43d5afd35b46n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/elixir-lang-core/e18fc5ab-b9bb-4a3d-b980-10f069082557n%40googlegroups.com.
