We have discussed this item in community forum and community meetups. Capturing
some of the takeaways here. Right now the set of tests we have are a bit mixed
together that causes things to slow down and mix of more recent modules with
legacy ones. In many cases, it is also tempting to just comparing TVM results
against other frameworks is easy to do in tests but it’s slow and very fragile;
many older tests do this.
The main approach we would like to take is to categorize the test cases. We
will start with TensorIR. Hopefully the lessons also expand to other modules
(at least those that are not legacy).
### Guidelines
There will be three categories of tests(using tir as example)
- tests/python/tir: This is the location for unit-tests
- The tests in this folder always use TVMScript/IRBuilder before and after
to test related module(pass)
- There is no explicit build involved
- Tests here should be fast
- tests/python/integration/tir: This is the location for tests that involves
build, but is also still fast
- The tests in this folder can call build
- We still encourage not calling schedule and other things, instead, if we
are testing say a CUDA build,
consider start from a already scheduled TVMScript
- Tests in this folder should be reasonably fast(e.g. finish in 1min or
less)
- tests/python/slow/tir: This is the location for tests that involves build,
runs end to end are slow
- Tests in this folder can take minutes to run
- The idea is that if a bug happens and is not caught by integration or
unit-test, likely it means we should update those tests
The idea is to gradually move tests into those three categories, and we can
move the slow tests to nightly, with the goal that if a regression is caught by
`tests/python/slow/tir`, it should really be covered by ` tests/python/tir` and
`tests/python/integration/tir` instead.
The categorization do not need to be done in a single shot. Since we can start
by having (legacy) tests in their original folder, and migrate old tests and
put new tests into those folders.
--
Reply to this email directly or view it on GitHub:
https://github.com/apache/tvm/issues/15354
You are receiving this because you are subscribed to this thread.
Message ID: <apache/tvm/issues/15...@github.com>
