> From: Morten Brørup > Sent: Tuesday, 18 August 2026 11.13 > > > From: Konstantin Ananyev [mailto:[email protected]] > > Sent: Tuesday, 18 August 2026 10.15 > > > > > > > > > > > > > > > > > + /* Common way for small copy size of 64- > > byte > > > > > > blocks. > > > > > > > > > > > > Unlikely, so > > > > > > > > > > > > > > constant size only */ > > > > > > > > > > > > > > > + if (__rte_constant(n) && (n & 63) == 0 && > > n <= > > > > > > > > > > > > > > RTE_MEMCPY_BLOCK_64_MAX) { > > > > > > > > > > > > > > > + void *ret = dst; > > > > > > > > > > > > > > > + > > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe just let compiler decide, it will > > generate > > > > vector > > > > > > > > > > > > instructions in > > > > > > > > > > > > > > most cases. > > > > > > > > > > > > > > > > > > > > > > > > > > > > if (__rte_constant(n)) > > > > > > > > > > > > > > return mempcpy(dst, src, n); > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe in most, but not in all: > > > > > > > > > > > > > https://godbolt.org/z/KvdKqT5rY > > > > > > > > > > > > > > > > > > > > > > > > With '-mavx' or '-mavx512f' it looks like it does > > for > > > > your > > > > > > > > sample > > > > > > > > > > code. > > > > > > > > > > > > > > > > > > > > > > It also does with -msse4.2 when SZ is reduced to > 256 > > > > bytes. > > > > > > > > > > > Clang switches to inline when SZ is reduced to 128 > > bytes. > > > > > > > > > > > > > > > > > > > > > > It seems the compiler has a threshold for when to > > inline > > > > and > > > > > > when > > > > > > > > to > > > > > > > > > > call the C > > > > > > > > > > > library's memcpy subroutine. > > > > > > > > > > > The threshold depends on both copy size and vector > > > > register > > > > > > size. > > > > > > > > > > > And it is compiler dependent. > > > > > > > > > > > > > > > > > > > > I think there are compiler options to specify desired > > > > threshold > > > > > > > > values. > > > > > > > > > > Let say for gcc there is ' -mmemcpy- > > strategy=strategy'. > > > > > > > > > > For that example in that particular case > > > > > > > > > > -mmemcpy-strategy=vector_loop:512:align,loop:-1:align > > > > > > > > > > generates sse loads/stores. > > > > > > > > > > Might be we can exploit it somehow? > > > > > > > > > > > > > > > > > > That could give us higher granularity/control over > memcpy > > for > > > > > > > > individual > > > > > > > > > memcpy instances; might be useful for hot code paths > > where we > > > > > > have > > > > > > > > more > > > > > > > > > knowledge about the copy operation than the compiler > can > > > > infer. > > > > > > > > > However, pragmas are discouraged in DPDK, and this > looks > > like > > > > a > > > > > > very > > > > > > > > similar > > > > > > > > > path. > > > > > > > > > > > > > > > > Well, right now rte_memcpy.h is 700+ lines and keeps > > growing. > > > > > > > > Considering that probably pragmas are not that bad. > > > > > > > > Of course, pragmas have their own issues and it is hard > to > > > > ensure > > > > > > that > > > > > > > > they will produce same code between different > > > > compilers/versions, > > > > > > etc. > > > > > > > > > > > > > > > > > > I am not really happy that our home-brewed memcpy > code- > > > > block > > > > > > keeps > > > > > > > > > > growing, > > > > > > > > > > while we keep talking that it would be good to > > eliminate it > > > > > > > > completely. > > > > > > > > > > > > > > > > > > I agree in principle. > > > > > > > > > However, this rte_memcpy() optimization is for the > > > > pile/mempool > > > > > > > > optimizations > > > > > > > > > I'm working on, so there is a specific use case > > motivating > > > > the > > > > > > added > > > > > > > > code. > > > > > > > > > > > > > > > > I understand that you probably have some specific use- > case > > in > > > > mind. > > > > > > > > BTW for this optimization you mentioned above: what is > the > > gain > > > > > > with > > > > > > > > these changes? > > > > > > > > > > > > > > IMO, the primary benefit is the much simpler (and smaller) > > > > assembly > > > > > > output due > > > > > > > to avoiding the address alignment check (and the resulting > > > > duplicated > > > > > > code). > > > > > > > > > > > > I think that should be measurable too: whole binary and/or > hot > > > > path > > > > > > function size reduction, etc. > > > > > > > > > > The size of the generated code for the copy operation is > reduced > > to > > > > slightly less > > > > > than half (only one instance of the copy operation instead of > > two, > > > > and the > > > > > address alignment comparison is omitted). > > > > > > > > > > > > > > > > > > I haven't measured the performance gain. > > > > > > > Based on the perf gain in a previous mempool optimization > > patch > > > > [1], > > > > > > it seems > > > > > > > avoiding the address alignment check shaves ~2 cycles off > the > > > > copy > > > > > > operation (for > > > > > > > cache-to-cache copy). > > > > > > > I expect that the same gain (from avoiding the address > > alignment > > > > > > check) applies > > > > > > > here. > > > > > > > > > > > > Ok, then I suggest we do some measurements first, before > going > > > > forward > > > > > > with it. > > > > > > > > > > I tried memcpy_perf_autotest, but the branch predictor kicks in > > and > > > > eliminates > > > > > the cost of the address alignment comparison. So the results > are > > very > > > > similar. > > > > > > > > Indeed, they are the same. > > > > > > > > > BTW, that's a general issue with our perf tests: Repeated > testing > > > > doesn't show > > > > > the cost of branches, because they are always eliminated by the > > > > branch > > > > > predictor. > > > > > > > > Might be... > > > > But we do need some measurable evidence that such change improve > > > > things, > > > > otherwise - why to bother? > > > > If that perf test is not goof enough - let's try to extend it or > > use > > > > different one. > > > > BTW, in your mempool pile RFC, I see you also use rte_memcpy over > > fixed > > > > sized buffers. > > > > Do you see any improvement there (wit/without rte_memcpy > > optimixation)? > > > > > > Looking at the generated assembly (objdump -S > > > build/drivers/librte_mempool_stack.so)... > > > > > > This is the optimized copy loop in pile_dequeue: > > > > > > 1650: 89 ca mov %ecx,%edx > > > 1652: c5 fe 6f 40 40 vmovdqu 0x40(%rax),%ymm0 > > > 1657: ff c1 inc %ecx > > > 1659: c1 e2 05 shl $0x5,%edx > > > 165c: 49 8d 14 d4 lea (%r12,%rdx,8),%rdx > > > 1660: c5 fe 7f 02 vmovdqu %ymm0,(%rdx) > > > 1664: c5 fe 6f 48 60 vmovdqu 0x60(%rax),%ymm1 > > > 1669: c5 fe 7f 4a 20 vmovdqu %ymm1,0x20(%rdx) > > > 166e: c5 fe 6f 90 80 00 00 vmovdqu 0x80(%rax),%ymm2 > > > 1675: 00 > > > 1676: c5 fe 7f 52 40 vmovdqu %ymm2,0x40(%rdx) > > > 167b: c5 fe 6f 98 a0 00 00 vmovdqu 0xa0(%rax),%ymm3 > > > 1682: 00 > > > 1683: c5 fe 7f 5a 60 vmovdqu %ymm3,0x60(%rdx) > > > 1688: c5 fe 6f a0 c0 00 00 vmovdqu 0xc0(%rax),%ymm4 > > > 168f: 00 > > > 1690: c5 fe 7f a2 80 00 00 vmovdqu %ymm4,0x80(%rdx) > > > 1697: 00 > > > 1698: c5 fe 6f a8 e0 00 00 vmovdqu 0xe0(%rax),%ymm5 > > > 169f: 00 > > > 16a0: c5 fe 7f aa a0 00 00 vmovdqu %ymm5,0xa0(%rdx) > > > 16a7: 00 > > > 16a8: c5 fe 6f b0 00 01 00 vmovdqu 0x100(%rax),%ymm6 > > > 16af: 00 > > > 16b0: c5 fe 7f b2 c0 00 00 vmovdqu %ymm6,0xc0(%rdx) > > > 16b7: 00 > > > 16b8: c5 fe 6f b8 20 01 00 vmovdqu 0x120(%rax),%ymm7 > > > 16bf: 00 > > > 16c0: c5 fe 7f ba e0 00 00 vmovdqu %ymm7,0xe0(%rdx) > > > 16c7: 00 > > > 16c8: 48 8b 40 08 mov 0x8(%rax),%rax > > > 16cc: 39 f1 cmp %esi,%ecx > > > 16ce: 75 80 jne 1650 > <pile_dequeue+0xe0> > > > 16d0: > > > > > > This is without the rte_memcpy optimization: > > > > > > 1650: 89 ca mov %ecx,%edx > > > 1652: c5 fe 6f 40 40 vmovdqu 0x40(%rax),%ymm0 > > > 1657: 49 89 c1 mov %rax,%r9 > > > 165a: ff c1 inc %ecx > > > 165c: c1 e2 05 shl $0x5,%edx > > > 165f: 49 8d 14 d4 lea (%r12,%rdx,8),%rdx > > > 1663: c5 fe 7f 02 vmovdqu %ymm0,(%rdx) > > > 1667: c5 fe 6f 48 60 vmovdqu 0x60(%rax),%ymm1 > > > 166c: 49 09 d1 or %rdx,%r9 > > > 166f: 41 83 e1 1f and $0x1f,%r9d > > > 1673: c5 fe 7f 4a 20 vmovdqu %ymm1,0x20(%rdx) > > > 1678: c5 fe 6f 90 80 00 00 vmovdqu 0x80(%rax),%ymm2 > > > 167f: 00 > > > 1680: c5 fe 7f 52 40 vmovdqu %ymm2,0x40(%rdx) > > > 1685: c5 fe 6f 98 a0 00 00 vmovdqu 0xa0(%rax),%ymm3 > > > 168c: 00 > > > 168d: c5 fe 7f 5a 60 vmovdqu %ymm3,0x60(%rdx) > > > 1692: c5 fe 6f a0 c0 00 00 vmovdqu 0xc0(%rax),%ymm4 > > > 1699: 00 > > > 169a: c5 fe 7f a2 80 00 00 vmovdqu %ymm4,0x80(%rdx) > > > 16a1: 00 > > > 16a2: c5 fe 6f a8 e0 00 00 vmovdqu 0xe0(%rax),%ymm5 > > > 16a9: 00 > > > 16aa: c5 fe 7f aa a0 00 00 vmovdqu %ymm5,0xa0(%rdx) > > > 16b1: 00 > > > 16b2: c5 fe 6f b0 00 01 00 vmovdqu 0x100(%rax),%ymm6 > > > 16b9: 00 > > > 16ba: c5 fe 7f b2 c0 00 00 vmovdqu %ymm6,0xc0(%rdx) > > > 16c1: 00 > > > 16c2: c5 fe 6f b8 20 01 00 vmovdqu 0x120(%rax),%ymm7 > > > 16c9: 00 > > > 16ca: c5 fe 7f ba e0 00 00 vmovdqu %ymm7,0xe0(%rdx) > > > 16d1: 00 > > > 16d2: 48 8b 40 08 mov 0x8(%rax),%rax > > > 16d6: 0f 85 c4 00 00 00 jne 17a0 > <pile_dequeue+0x230> > > > 16dc: 39 ce cmp %ecx,%esi > > > 16de: 0f 85 6c ff ff ff jne 1650 > <pile_dequeue+0xe0> > > > 16e4: > > > [...] > > > 17a0: 39 f1 cmp %esi,%ecx > > > 17a2: 0f 85 a8 fe ff ff jne 1650 > <pile_dequeue+0xe0> > > > 17a8: e9 37 ff ff ff jmp 16e4 > <pile_dequeue+0x174> > > > > > > Here, the alignment comparison I have optimized away is performed > > using > > > register %r9. > > > > > > With the optimization, the generated assembly is less cluttered, > and > > thus easier > > > to review. > > > (It seems the compiler is clever enough to not duplicate the two > > instances of the > > > code, but reuse the aligned instance for the unaligned case too. So > > the reduction > > > in code size is not as great as I previously claimed.) > > > > > > I agree the performance benefit in CPU cycles is probably > > insignificant. > > > But the generated assembly is cleaner. > > > And if pressured for CPU registers, the optimization also frees up > > one CPU > > > register for other purposes. > > > > > > Code size reduction: > > > The optimized loop is 0x80 bytes of instructions. > > > The non-optimized is 0x8e bytes, 14 bytes more, in the loop, plus > 13 > > bytes > > > outside the loop. > > > > Honestly, with such insignificant gains, I'd either leave it alone, > > or look more closely at compiler pragmas option we discussed before. > > Compiled code is slightly smaller, performance is slightly better. > But source code having slightly more lines of code is more important? > > If the source code added was difficult to read, complexity was > increased, or had side effects or spillover to other modules, I might > buy that argument. > But the addition is very simple and completely isolated. > > If the objection is about source code readability, I could add more > inline comments to the added code, elaborating that the added code path > is the same for all CPU vector sizes, regardless of address alignment. > But I suspect such comments would confuse more than they would help. > And it would add even more lines to the file size. > > Adding compiler pragmas should probably be done in source code calling > rte_memcpy(), like __rte_assume() and the coming > __rte_assume_cache_aligned() [1]. > > [1]: > https://patchwork.dpdk.org/project/dpdk/patch/20260812090723.1771628-1- > [email protected]/ > > If you try experimenting with compiler pragmas, I'm open for reviewing > an RFC! > For now, let's take this small step. >
I seems nobody want more code in the rte_memcpy() implementation, so I have marked this patch as Rejected, and instead posted a separate patch with the cleanups [2]. [2]: https://patchwork.dpdk.org/project/dpdk/patch/[email protected]/ I had an improved version in the works, not requiring constant size; but I assume it will be rejected too, so I'm not pursuing it. For anyone interested, it had this criteria: if (__rte_constant((n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) && (n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) { The full diff: @@ -175,2 +175,3 @@ rte_mov256(uint8_t *__rte_restrict dst, #define RTE_MEMCPY_ALIGNMENT_MASK 0x3F +#define RTE_MEMCPY_BLOCK_64_MAX 512 @@ -321,2 +322,3 @@ COPY_BLOCK_128_BACK63: #define RTE_MEMCPY_ALIGNMENT_MASK 0x1F +#define RTE_MEMCPY_BLOCK_64_MAX 256 @@ -432,2 +434,3 @@ COPY_BLOCK_128_BACK31: #define RTE_MEMCPY_ALIGNMENT_MASK 0x0F +#define RTE_MEMCPY_BLOCK_64_MAX 512 @@ -675,2 +678,36 @@ rte_memcpy(void *__rte_restrict dst, con { + /* + * Common way for small copy size of 64-byte blocks, + * independent of alignment with vector register size. + * Note: + * If known at compile time that the criteria are fulfilled, + * the compiler will use this, and omit everything else; + * otherwise, the compiler will omit this. + */ + static_assert(RTE_MEMCPY_BLOCK_64_MAX <= 512, + "64-byte block copy max size too big for implementation below"); + if (__rte_constant((n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) && + (n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) { + void *ret = dst; + + if (!(RTE_MEMCPY_BLOCK_64_MAX < 512) && n == 512) { + rte_mov256((uint8_t *)dst + 0 * 256, (const uint8_t *)src + 0 * 256); + rte_mov256((uint8_t *)dst + 1 * 256, (const uint8_t *)src + 1 * 256); + } else { + if (n & 256) { + rte_mov256((uint8_t *)dst, (const uint8_t *)src); + src = (const uint8_t *)src + 256; + dst = (uint8_t *)dst + 256; + } + if (n & 128) { + rte_mov128((uint8_t *)dst, (const uint8_t *)src); + src = (const uint8_t *)src + 128; + dst = (uint8_t *)dst + 128; + } + if (n & 64) + rte_mov64((uint8_t *)dst, (const uint8_t *)src); + } + return ret; + } + /* Fast way when copy size doesn't exceed 64 bytes. */ @@ -715,2 +752,3 @@ rte_memcpy(void *__rte_restrict dst, con #undef RTE_MEMCPY_ALIGNMENT_MASK +#undef RTE_MEMCPY_BLOCK_64_MAX
