My initial (probably incorrect) implementation: https://github.com/elixir-lang/elixir/pull/14499
On Tuesday, May 13, 2025 at 3:36:57 AM UTC-7 Noah Betzen wrote: > > Can you please expand on how the files would be used compared to > directly running the stale tests? > > Initially I was thinking of being able to do something like: > > 1. At one point in time: `mix test --get-stale` > 2. Later: `mix test --get-stale` > 3. Diff the two outputs, and only run the tests that appear in both? > (now that I type this out, I realize this probably doesn't help) > > This might be a case of the "XY problem"; perhaps what I want is not > "print the list of tests that `mix test --stale` would run", but something > like "given the current state of the code, and these new code/module > changes, give me the list of tests that should be considered 'stale' and > should rerun". > > Here's more of a typical business scenario I envision: > > 1. I have a project branch `main` in which CI runs all tests (say, > 1000 tests) on every PR merge. > 1. Until then, CI should only run a subset of tests on each commit > to a new PR. > 2. On `main`, I run `mix test --stale` locally, and all 1000 tests > pass. > 1. Ideally I could cache/snapshot this "state" or point in time, or > CI could generate it directly on `main`. > 3. I make a new branch `feature/a`, make code changes, add a couple > new tests. > 4. I run `mix test --stale` locally, lets say that only 100 tests need > to run based on the code changed. > 5. I submit a PR. > 6. *Current setup:* CI tries to run `mix test --stale` but ends up > running all 1000 tests again instead of just the 100 whose code/state was > actually "stale". > 7. *Preferred setup:* CI runs `mix test --stale` (or something else) > and only reruns tests that would be considered stale after my commits were > made. > 8. PR merges into `main`, and just in case all 1000 tests are run > again. > > > #14393 definitely looks interesting and I'll keep an eye on it! > On Saturday, May 10, 2025 at 12:46:38 AM UTC-7 José Valim wrote: > >> Can you please expand on how the files would be used compared to directly >> running the stale tests? >> >> Note there is also https://github.com/elixir-lang/elixir/issues/14393, >> but it is currently unclear if it could be used for this purpose. I asked >> Andrea to add more context, I recommend following the discussion there too. >> >> >> >> >> *José Valimhttps://dashbit.co/ <https://dashbit.co/>* >> >> >> On Sat, May 10, 2025 at 00:20 'Noah Betzen' via elixir-lang-core < >> elixir-l...@googlegroups.com> wrote: >> >>> Currently `mix test --stale` will do some checks for code that has been >>> updated since the last test suite run. Relevant code: >>> https://github.com/elixir-lang/elixir/blob/4fa224099730f59b19912e605a888bab68da6e5b/lib/mix/lib/mix/compilers/test.ex#L119-L176 >>> >>> For the purposes of larger Elixir projects and CI, it'd be nice to fetch >>> the list of tests that need to be rerun but not rerun them in the same >>> command. >>> >>> This has partially been brought up before: >>> https://elixirforum.com/t/is-there-a-way-to-predetermine-which-tests-and-why-would-be-run-on-mix-test-stale/48325 >>> >>> The most basic implementation idea I had for this was `mix test >>> --print-stale` which prints all currently stale tests to stdout (or to a >>> file?). >>> >>> Another possibly nice feature would be the ability to provide a commit >>> hash similar to Credo's git diff >>> https://hexdocs.pm/credo/diff_command.html that shows which tests are >>> "stale" between two points in time. The usefulness of this idea may be >>> limited. >>> >>> tldr I want to be able to configure my CI to not run ALL tests on every >>> single PR. There are other ways to do this, but providing the data that >>> `mix test --stale` uses to determine staleness seems like a great start! >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elixir-lang-co...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/elixir-lang-core/906a55e5-1b0d-4f4b-813e-43d5afd35b46n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elixir-lang-core/906a55e5-1b0d-4f4b-813e-43d5afd35b46n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/51ac9f6e-ea12-42e3-bd2c-11099e4485aen%40googlegroups.com.
