[
https://issues.apache.org/jira/browse/IGNITE-28952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Davydov updated IGNITE-28952:
------------------------------------
Description:
The MDC test fixture hardcodes two things the MDC feature itself does not
require: that every cache carries an MdcTopologyValidator, and that the cluster
spans exactly two data centers.
1. Cache level topology validator
The MDC test applications always configure the cache with MdcTopologyValidator
(MdcCacheAwareApplication), so the scenarios only run against a validator
configured per
cache.
2. Number of data centers
MdcTopologyValidator has two modes, and the number of data centers selects
which one
applies: an even set is validated against a main datacenter, an odd set
against a majority
of the set. The fixture only ever exercised the even one - DCS = (DC1, DC2)
is a module
level constant threaded through the whole of MdcCluster, verify_split_brain()
assumes
exactly two half-rings, and cross_dc_network() impairs a single DC pair.
Changes:
- MdcCacheAwareApplication: new 'topologyValidator' parameter, default true.
When false,
the cache is created without the cache level validator and the
mainDc/datacenters
parameters are not required.
- MdcCluster: new 'mdc_cache_topology_validator' global (default true)
applied to every
cache the fixture creates.
- MdcCluster takes a 'dcs' sequence (DCS_2 by default, DCS_3 provided).
mdc_topology_params()
derives the validator mode from the DC count and emits exactly one of
mainDc/datacenters
- MdcTopologyValidator.checkConfiguration() rejects a main datacenter
alongside an odd DC
set - plus the dcsNum MdcAffinityBackupFilter needs. min_backups() derives
the smallest
backup count giving every DC one copy of every partition, which is 2 for
three DCs.
- verify_split_brain() generalizes into verify_segments() over an arbitrary
segmentation;
verify_half_ring_healthy() becomes verify_segment_healthy().
cross_dc_network() impairs
the whole cross-DC mesh and takes per-pair overrides for asymmetric links.
- NetworkGroupManager: new
enable_network_partitions()/disable_network_partitions() cut
several links at once. Every chain a node takes part in is installed by the
same SSH
round-trip, because rolling a multi-way split out link by link would
briefly present the
cluster with a different, valid segmentation it would legitimately react
to. The pairwise
methods delegate to them.
- New scenarios in mdc/majority_partition_test.py (9 nodes each): a minority
DC isolated
from the two others, parametrized over which DC is isolated; a three way
split in which
every segment goes read-only; and data centers lost one by one, writable
while a majority
survives and read-only below it. Every partition owns one copy per DC, so
every segment
still serves every read - that is what separates the read assertions from
the write ones.
- Framework unit checks for the DC-count dependent parts: validator mode
selection, backup
arithmetic, the impairment mesh and its per-pair overrides, and the
per-node command
compilation of single and multi-link partitions.
was:
The MDC test applications always configure the cache with MdcTopologyValidator
(MdcCacheAwareApplication), so the scenarios only run against a validator
configured per cache.
The scenarios themselves are validator agnostic: the load applications treat
any
CacheException/IgniteException as a rejected write, without inspecting its
type.
Only the cache configuration is in the way.
Changes:
* MdcCacheAwareApplication: new 'topologyValidator' parameter, default true.
When
false, the cache is created without the cache level validator and the
mainDc/datacenters parameters are not required.
* MdcCluster: new 'mdc_cache_topology_validator' global (default true)
applied to
every cache the fixture creates.
* README: document the global.
> [ducktests] Generalize the MDC test fixture: optional cache level validator,
> three data center scenarios
> --------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-28952
> URL: https://issues.apache.org/jira/browse/IGNITE-28952
> Project: Ignite
> Issue Type: Task
> Reporter: Maksim Davydov
> Assignee: Maksim Davydov
> Priority: Minor
> Labels: IEP-140, ducktests, ise
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The MDC test fixture hardcodes two things the MDC feature itself does not
> require: that every cache carries an MdcTopologyValidator, and that the
> cluster spans exactly two data centers.
> 1. Cache level topology validator
> The MDC test applications always configure the cache with
> MdcTopologyValidator
> (MdcCacheAwareApplication), so the scenarios only run against a validator
> configured per
> cache.
> 2. Number of data centers
> MdcTopologyValidator has two modes, and the number of data centers selects
> which one
> applies: an even set is validated against a main datacenter, an odd set
> against a majority
> of the set. The fixture only ever exercised the even one - DCS = (DC1, DC2)
> is a module
> level constant threaded through the whole of MdcCluster,
> verify_split_brain() assumes
> exactly two half-rings, and cross_dc_network() impairs a single DC pair.
> Changes:
> - MdcCacheAwareApplication: new 'topologyValidator' parameter, default
> true. When false,
> the cache is created without the cache level validator and the
> mainDc/datacenters
> parameters are not required.
> - MdcCluster: new 'mdc_cache_topology_validator' global (default true)
> applied to every
> cache the fixture creates.
> - MdcCluster takes a 'dcs' sequence (DCS_2 by default, DCS_3 provided).
> mdc_topology_params()
> derives the validator mode from the DC count and emits exactly one of
> mainDc/datacenters
> - MdcTopologyValidator.checkConfiguration() rejects a main datacenter
> alongside an odd DC
> set - plus the dcsNum MdcAffinityBackupFilter needs. min_backups()
> derives the smallest
> backup count giving every DC one copy of every partition, which is 2 for
> three DCs.
> - verify_split_brain() generalizes into verify_segments() over an arbitrary
> segmentation;
> verify_half_ring_healthy() becomes verify_segment_healthy().
> cross_dc_network() impairs
> the whole cross-DC mesh and takes per-pair overrides for asymmetric links.
> - NetworkGroupManager: new
> enable_network_partitions()/disable_network_partitions() cut
> several links at once. Every chain a node takes part in is installed by
> the same SSH
> round-trip, because rolling a multi-way split out link by link would
> briefly present the
> cluster with a different, valid segmentation it would legitimately react
> to. The pairwise
> methods delegate to them.
> - New scenarios in mdc/majority_partition_test.py (9 nodes each): a
> minority DC isolated
> from the two others, parametrized over which DC is isolated; a three way
> split in which
> every segment goes read-only; and data centers lost one by one, writable
> while a majority
> survives and read-only below it. Every partition owns one copy per DC, so
> every segment
> still serves every read - that is what separates the read assertions from
> the write ones.
> - Framework unit checks for the DC-count dependent parts: validator mode
> selection, backup
> arithmetic, the impairment mesh and its per-pair overrides, and the
> per-node command
> compilation of single and multi-link partitions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
