On 19.11.2020 23:02, Peter Xu wrote:
On Thu, Nov 19, 2020 at 03:59:40PM +0300, Andrey Gruzdev wrote:
Since reading UFFD events and saving paged data are performed
from the same thread, write fault latencies are sensitive to
migration stream stalls. Limiting total page saving rate is a
method to reduce amount of noticiable fault resolution latencies.
Migration bandwidth limiting is achieved via noticing cases of
out-of-threshold write fault latencies and temporarily disabling
(strictly speaking, severely throttling) saving non-faulting pages.
Just curious: have you measured aver/max latency of wr-protected page requests,
or better, even its distribution?
I believe it should also be relevant to where the snapshot is stored, say, the
backend disk of your tests. Is that a file on some fs?
I would expect the latency should be still good if e.g. the throughput of the
backend file system is decent even without a patch like this, but I might have
missed something..
In all cases, it would be very nice if this patch can have the histogram or
aver or max latency measured and compared before/after this patch applied.
Thanks,
So far I have no objective latency measurements, that really
interesting. For testing I commonly used SATA HDD, not too fresh one,
1.5TB Seagate 7200.11 series, specially not to have large hardware cache
or flash buffer. And yes, backend is a file on ext4.
I tried mostly with 'migrate exec:streamdump_utility', a very simple
tool which writes stream to a file. It even doesn't use AIO - so reads
from STDIN and file writes don't overlap. Made so intentionally to
reduce throughput and give some random high-latency writes.
I think snapshotting performance may be severely degraded by I/O
happening in parallel on the same storage media, that's the problem we
need to consider.
And yes, to take latency histogram before/after the patch is nice idea!
Also I need to make stream dumping tool with AIO, to test with full
storage throughput.
--
Andrey Gruzdev, Principal Engineer
Virtuozzo GmbH +7-903-247-6397
virtuzzo.com
