I see, thanks!
> On 26 Nov 2018, at 11:46, Roman Kondakov <[email protected]> wrote:
>
> Hi, Petr!
>
> Actually these tests do not behave exactly as one test, but they behave
> mostly as one test. Often this is expressed in the following: we have a
> feature to test (i.e. transaction isolation) and different sets of parameters
> to be tested with this feature: cache mode, backups number, cluster size,
> persistence enabled/disabled, etc. And we have a list of tests with different
> combinations of these parameters:
>
> testTxIsolation(Replicated, 0 backups, 1 severs 0 clients, persistence
> disabled)
> testTxIsolation(Replicated, 0 backups, 2 severs 1 clients, persistence
> disabled)
> testTxIsolation(Replicated, 0 backups, 4 severs 2 clients, persistence
> disabled)
> testTxIsolation(Replicated, 0 backups, 1 severs 0 clients, persistence
> enabled)
> testTxIsolation(Replicated, 0 backups, 2 severs 1 clients, persistence
> enabled)
> testTxIsolation(Replicated, 0 backups, 4 severs 2 clients, persistence
> enabled)
> testTxIsolation(Partitioned, 0 backups, 1 severs 0 clients, persistence
> disabled)
> testTxIsolation(Partitioned, 1 backups, 2 severs 1 clients, persistence
> disabled)
> testTxIsolation(Partitioned, 2 backups, 4 severs 2 clients, persistence
> disabled)
> testTxIsolation(Partitioned, 0 backups, 1 severs 0 clients, persistence
> enabled)
> testTxIsolation(Partitioned, 1 backups, 2 severs 1 clients, persistence
> enabled)
> testTxIsolation(Partitioned, 2 backups, 4 severs 2 clients, persistence
> enabled)
>
> Each test in this list represents a special case which should be tested. If
> the key functionality of Tx isolation is broken by developer - all tests
> will be failed. But there are a very rare cases when only a subset of this
> tests is failed when others are green. In my opinion for "fast" runs we
> should trigger only one or two tests from this group - it should be enough to
> detect the most of bugs. But for night runs, of course, all cases should be
> checked.
>
> --
> Kind Regards
> Roman Kondakov
>
> On 26.11.2018 11:11, Petr Ivanov wrote:
>> Hi, Roman.
>>
>>> On 25 Nov 2018, at 21:26, Roman Kondakov <[email protected]> wrote:
>>>
>>> Hi Dmitriy!
>>>
>>> We have over 50 000 test in our tests base. And this number will be
>>> noticeably increased soon by MVCC tests coverage activity. This means that
>>> it is very difficult to rework and rewrite these test manually to make it
>>> run faster. But we can choose another way. Do we have an ability to perform
>>> a statistical analysis over a considerable number of last tests runs? If we
>>> do, let's consider two points:
>>>
>>> 1. After careful consideration in terms of statistics it may turnout that
>>> significant number of these test are "evergreen" tests. It means that these
>>> tests check cases which are very difficult to break. If so, why should we
>>> run these tests each time? They are great candidates for night runs.
>>>
>>> 2. After dropping "evergreen" tests there are may be a number of tests with
>>> correlated results. There could be a lot of test groups with a some number
>>> of tests in each group, where either all tests are red or all tests are
>>> green. In this case in "fast" runs we can launch only one test from each
>>> group instead of entire group. Other tests in group can be launched at
>>> night build.
>> What’s the point of having all this tests if they behave as one and will
>> never fail exclusively?
>> Maybe such tests require optimisation and shrinking into one?
>>
>>
>>>
>>> Having a list of "good" tests (good tests = all tests - evergreen tests -
>>> groups (except chosen represenative from each group)), we can mark these
>>> tests with annotation @Category (or @Tag in junit 5). For fast tests runs
>>> we can run only annotated tests, for night runs - all tests as usual.
>>>
>>> When new test is added developers could decide to add or not to add this
>>> annotation.
>>>
>>> Annotated tests list should be reviewed monthly or weekly. Or, if possible,
>>> automate this procedure.
>>>
>>>
>>> --
>>> Kind Regards
>>> Roman Kondakov
>>>
>>> On 15.11.2018 13:34, Dmitriy Pavlov wrote:
>>>> Hi Igniters,
>>>>
>>>>
>>>>
>>>> Some of us started to use the Bot to get an approval of PRs. It helps to
>>>> protect master from new failures, but this requires to run RunAll tests set
>>>> for each commit and this makes markable pressure to TC infra.
>>>>
>>>>
>>>>
>>>> I would like to ask you to share your ideas on how to make runAll faster,
>>>> maybe you can share any of your measurements and any other info about
>>>> (possible) bottlenecks.
>>>>
>>>>
>>>>
>>>> Sincerely,
>>>>
>>>> Dmitriy Pavlov
>>>>
