Hi, Petr!
Actually these tests do not behave exactly as one test, but they behave
mostly as one test. Often this is expressed in the following: we have a
feature to test (i.e. transaction isolation) and different sets of
parameters to be tested with this feature: cache mode, backups number,
cluster size, persistence enabled/disabled, etc. And we have a list of
tests with different combinations of these parameters:
testTxIsolation(Replicated, 0 backups, 1 severs 0 clients, persistence
disabled)
testTxIsolation(Replicated, 0 backups, 2 severs 1 clients, persistence
disabled)
testTxIsolation(Replicated, 0 backups, 4 severs 2 clients, persistence
disabled)
testTxIsolation(Replicated, 0 backups, 1 severs 0 clients, persistence
enabled)
testTxIsolation(Replicated, 0 backups, 2 severs 1 clients, persistence
enabled)
testTxIsolation(Replicated, 0 backups, 4 severs 2 clients, persistence
enabled)
testTxIsolation(Partitioned, 0 backups, 1 severs 0 clients,
persistence disabled)
testTxIsolation(Partitioned, 1 backups, 2 severs 1 clients,
persistence disabled)
testTxIsolation(Partitioned, 2 backups, 4 severs 2 clients,
persistence disabled)
testTxIsolation(Partitioned, 0 backups, 1 severs 0 clients,
persistence enabled)
testTxIsolation(Partitioned, 1 backups, 2 severs 1 clients,
persistence enabled)
testTxIsolation(Partitioned, 2 backups, 4 severs 2 clients,
persistence enabled)
Each test in this list represents a special case which should be tested.
If the key functionality of Tx isolation is broken by developer - all
tests will be failed. But there are a very rare cases when only a subset
of this tests is failed when others are green. In my opinion for "fast"
runs we should trigger only one or two tests from this group - it should
be enough to detect the most of bugs. But for night runs, of course, all
cases should be checked.
--
Kind Regards
Roman Kondakov
On 26.11.2018 11:11, Petr Ivanov wrote:
Hi, Roman.
On 25 Nov 2018, at 21:26, Roman Kondakov <[email protected]> wrote:
Hi Dmitriy!
We have over 50 000 test in our tests base. And this number will be noticeably
increased soon by MVCC tests coverage activity. This means that it is very
difficult to rework and rewrite these test manually to make it run faster. But
we can choose another way. Do we have an ability to perform a statistical
analysis over a considerable number of last tests runs? If we do, let's
consider two points:
1. After careful consideration in terms of statistics it may turnout that significant
number of these test are "evergreen" tests. It means that these tests check
cases which are very difficult to break. If so, why should we run these tests each time?
They are great candidates for night runs.
2. After dropping "evergreen" tests there are may be a number of tests with correlated
results. There could be a lot of test groups with a some number of tests in each group, where
either all tests are red or all tests are green. In this case in "fast" runs we can
launch only one test from each group instead of entire group. Other tests in group can be launched
at night build.
What’s the point of having all this tests if they behave as one and will never
fail exclusively?
Maybe such tests require optimisation and shrinking into one?
Having a list of "good" tests (good tests = all tests - evergreen tests -
groups (except chosen represenative from each group)), we can mark these tests with
annotation @Category (or @Tag in junit 5). For fast tests runs we can run only annotated
tests, for night runs - all tests as usual.
When new test is added developers could decide to add or not to add this
annotation.
Annotated tests list should be reviewed monthly or weekly. Or, if possible,
automate this procedure.
--
Kind Regards
Roman Kondakov
On 15.11.2018 13:34, Dmitriy Pavlov wrote:
Hi Igniters,
Some of us started to use the Bot to get an approval of PRs. It helps to
protect master from new failures, but this requires to run RunAll tests set
for each commit and this makes markable pressure to TC infra.
I would like to ask you to share your ideas on how to make runAll faster,
maybe you can share any of your measurements and any other info about
(possible) bottlenecks.
Sincerely,
Dmitriy Pavlov
