Hi Dmitriy!
We have over 50 000 test in our tests base. And this number will be
noticeably increased soon by MVCC tests coverage activity. This means
that it is very difficult to rework and rewrite these test manually to
make it run faster. But we can choose another way. Do we have an ability
to perform a statistical analysis over a considerable number of last
tests runs? If we do, let's consider two points:
1. After careful consideration in terms of statistics it may turnout
that significant number of these test are "evergreen" tests. It means
that these tests check cases which are very difficult to break. If so,
why should we run these tests each time? They are great candidates for
night runs.
2. After dropping "evergreen" tests there are may be a number of tests
with correlated results. There could be a lot of test groups with a some
number of tests in each group, where either all tests are red or all
tests are green. In this case in "fast" runs we can launch only one test
from each group instead of entire group. Other tests in group can be
launched at night build.
Having a list of "good" tests (good tests = all tests - evergreen tests
- groups (except chosen represenative from each group)), we can mark
these tests with annotation @Category (or @Tag in junit 5). For fast
tests runs we can run only annotated tests, for night runs - all tests
as usual.
When new test is added developers could decide to add or not to add this
annotation.
Annotated tests list should be reviewed monthly or weekly. Or, if
possible, automate this procedure.
--
Kind Regards
Roman Kondakov
On 15.11.2018 13:34, Dmitriy Pavlov wrote:
Hi Igniters,
Some of us started to use the Bot to get an approval of PRs. It helps to
protect master from new failures, but this requires to run RunAll tests set
for each commit and this makes markable pressure to TC infra.
I would like to ask you to share your ideas on how to make runAll faster,
maybe you can share any of your measurements and any other info about
(possible) bottlenecks.
Sincerely,
Dmitriy Pavlov
