On 2022-08-09 17:00, Morten Brørup wrote:
From: Mattias Rönnblom [mailto:hof...@lysator.liu.se]
Sent: Tuesday, 9 August 2022 14.05
On 2022-08-09 11:46, Morten Brørup wrote:
From: Mattias Rönnblom [mailto:hof...@lysator.liu.se]
Sent: Sunday, 7 August 2022 22.25
On 2022-07-19 17:26, Morten Brørup wrote:
This RFC proposes a set of functions optimized for non-temporal
memory copy.
At this stage, I am asking for feedback on the concept.
Applications sometimes data to another memory location, which is
only
used
much later.
In this case, it is inefficient to pollute the data cache with the
copied
data.
An example use case (originating from a real life application):
Copying filtered packets, or the first part of them, into a capture
buffer
for offline analysis.
The purpose of these functions is to achieve a performance gain by
not
polluting the cache when copying data.
Although the throughput may be improved by further optimization, I
do
not
consider througput optimization relevant initially.
The x86 non-temporal load instructions have 16 byte alignment
requirements [1], while ARM non-temporal load instructions are
available with
4 byte alignment requirements [2].
Both platforms offer non-temporal store instructions with 4 byte
alignment
requirements.
I don't think memcpy() functions should have alignment requirements.
That's not very practical, and violates the principle of least
surprise.
I didn't make the CPUs with these alignment requirements.
However, I will offer optimized performance in a generic NT memcpy()
function in the cases where the individual alignment requirements of
various CPUs happen to be met.
Use normal memcpy() for the unaligned parts, and for the whole thing
for
small sizes (at least on x86).
I'm not going to plunge into some advanced vector programming, so I'm
working on an implementation where misalignment is handled by using a
bounce buffer (allocated on the stack, which is probably cache hot
anyway).
I don't know for the NT load + NT store case, but for regular load + NT
store, this is trivial. The implementation I've used is 36
straight-forward lines of code.
Is that implementation available for inspiration anywhere?
#define NT_THRESHOLD (2 * CACHE_LINE_SIZE)
void nt_memcpy(void *__restrict dst, const void * __restrict src, size_t n)
{
if (n < NT_THRESHOLD) {
memcpy(dst, src, n);
return;
}
size_t n_unaligned = CACHE_LINE_SIZE - (uintptr_t)dst % CACHE_LINE_SIZE;
if (n_unaligned > n)
n_unaligned = n;
memcpy(dst, src, n_unaligned);
dst += n_unaligned;
src += n_unaligned;
n -= n_unaligned;
size_t num_lines = n / CACHE_LINE_SIZE;
size_t i;
for (i = 0; i < num_lines; i++) {
size_t j;
for (j = 0; j < CACHE_LINE_SIZE / sizeof(__m128i); j++) {
__m128i blk = _mm_loadu_si128((const __m128i *)src);
/* non-temporal store */
_mm_stream_si128((__m128i *)dst, blk);
src += sizeof(__m128i);
dst += sizeof(__m128i);
}
n -= CACHE_LINE_SIZE;
}
if (num_lines > 0)
_mm_sfence();
memcpy(dst, src, n);
}
(This was written as a part of a benchmark exercise, and hasn't been
properly tested.)
Use this for inspiration, or I can DPDK-ify this and make it a proper
patch/RFC. I would try to add support for NT load as well, and make both
NT load and store depend on flags parameter.
The above threshold setting is completely arbitrary. What you should
keep in mind when thinking about the threshold, is that it might well be
worth to suffer a little lower performance of NT store + sfence
(compared to regular store), since you will benefit from not trashing
the cache.
For example, back-to-back copying of 1500 bytes buffers with this
copying routine is much slower than regular memcpy() (measured in the
core cycles spent in the copying), but nevertheless in a real-world
application it may still improve the overall performance, since the
packet copies doesn't evict useful data from the various caches. I know
for sure that certain applications do benefit.
