Re: af_alg benchmarks and performance

nge is endorsed by Redhat, and enabled by default > just for sha*sum in the next major version release, it will provide lots > of supporting evidence that it works well, and might be enabled globally > by default. > FYI I'm doing this as an hobbyist, as I have an home server runni

Re: af_alg benchmarks and performance

cases, you can disable it by default and package maintainers or firmware developers can enable it as needed. [1] http://macchiatobin.net/product/macchiatobin-double-shot/ [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/crypto/virtio -- Matteo Croce per aspera ad upstream

Re: af_alg benchmarks and performance

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/crypto/sha1_ni_asm.S [5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/crypto [6] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/crypto Regards, -- Matteo Croce per aspera ad upstream

Re: af_alg benchmarks and performance

sed by the kernel, yes, not all machines have these instructions, but as ssse3 was introduced in Core 2 Duo in 2006 I hope that the majority of the users machines already have them. I agree that the syscall has some cost and it's not suitable for handling very small buffers. Doing thousand of iterations on a small buffer was not on my check list, it would have spotted it, but we can workaround it by caching the af_alg socket. Cheers, -- Matteo Croce per aspera ad upstream

Re: optimizing af_alg

On Tue, May 8, 2018 at 2:59 AM, Bruno Haible wrote: > Matteo Croce wrote: >> Obviously it works only if you always use the same algo and with a >> single thread > > Why only in a single thread? Can't you do > > socket(AF_ALG, SOCK_SEQPACKET, 0) = 3 >

Re: af_alg: Add ability to use Linux kernel crypto API on data in memory

On Mon, May 7, 2018 at 11:55 AM, Matteo Croce wrote: > On Mon, May 7, 2018 at 4:07 AM, Paul Eggert wrote: >> Bruno Haible wrote: >>> >>> Oops, I goofed with "git diff". Here's the correct patch to test. >> >> >> I tried tho

Re: af_alg: Add ability to use Linux kernel crypto API on data in memory

e infamous patch to mitigate the Intel CPU bug, which adds a big overhead to syscalls, but it will hopefully disappear on future CPUs: $ dmesg |grep isolation [0.00] Kernel/User page tables isolation: enabled -- Matteo Croce per aspera ad upstream

Re: af_alg: Add ability to use Linux kernel crypto API on data in memory

of 2 hunks FAILED -- saving rejects to file lib/sha512.c.rej BTW, the instructions are you referring to, are for AES. For SHA1 and other hashes, an ASM implementation of the algorythm with SSSE3 or AVX2 is compiled into the kernel. FYI: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/l

Re: af_alg: Add ability to use Linux kernel crypto API on data in memory

7;ve checked out latest gnulib, and after double checking that commit 761523ddea70f0456b556c09868910686751fff5 was there I ran this: matteo@turbo:~/src/gnulib/testdir$ strace -e trace=%network with/gltests/bench-md5 1000 100 real 1.138617 user 1.139 sys0.000 +++ exited with 0 +++ matteo@turbo:~/src/gnulib/testdir$ strace -e trace=%network with/gltests/bench-sha1 1000 100 real 1.259929 user 1.260 sys0.000 +++ exited with 0 +++ It seems that kernel API are not used in this test, or I'm running them the wrong way? -- Matteo Croce per aspera ad upstream

[PATCH v3 0/4] Use AF_ALG in checksum utilities

ule description * revert unneeded 'Hey Emacs!' blocks * use correct GNU indentation * prefer size_t over int to denote memory segments * avoid possible overflow by checking arguments size * return -EIO if sendfile() returns a short read/write count * fix a file descriptor leak when bind

[PATCH v3 2/4] sha256sum: use kernel crypto API

Use AF_ALG for sha224 and sha256 too Signed-off-by: Matteo Croce --- lib/sha256.c | 32 ++-- modules/crypto/sha256 | 6 +- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/lib/sha256.c b/lib/sha256.c index 85405b20f..578f43e4d 100644

[PATCH v3 1/4] sha1sum: use AF_ALG when available

g.bin 752ef2367f479e79e4f0cded9c270c2890506ab0 2g.bin real0m3.164s user0m0.000s sys 0m3.162s Signed-off-by: Matteo Croce --- lib/af_alg.c| 115 lib/af_alg.h| 49 ++ lib/sha1.c | 17 +++

[PATCH v3 4/4] md5sum: use kernel crypto API

Use AF_ALG for md5 too Signed-off-by: Matteo Croce --- lib/md5.c | 18 +- modules/crypto/md5 | 6 +- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/lib/md5.c b/lib/md5.c index 68d00a6c7..307abbbe7 100644 --- a/lib/md5.c +++ b/lib/md5.c @@ -32,6

[PATCH v3 3/4] sha512sum: use kernel crypto API

Use AF_ALG for sha384 and sha512 too Signed-off-by: Matteo Croce --- lib/sha512.c | 32 ++-- modules/crypto/sha512 | 6 +- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/lib/sha512.c b/lib/sha512.c index 8a6dd4e83..72e5fdd24 100644

Re: [PATCH v2 1/4] sha1sum: use AF_ALG when available

On Wed, Apr 25, 2018 at 2:25 PM, Dmitry V. Levin wrote: > As struct sockaddr_alg is not provided by glibc, > you'd have to check for HAVE_LINUX_IF_ALG_H after all. > Does this require a change in the coreutils project configure.ac too? Regards, -- Matteo Croce per aspera ad upstream

Re: [PATCH v2 1/4] sha1sum: use AF_ALG when available

On Wed, Apr 25, 2018 at 9:07 PM, Paul Eggert wrote: > Thanks for working on this. Some comments: > Thanks for the review! > On 04/25/2018 04:26 AM, Matteo Croce wrote: >> >> + This file is part of the GNU C Library. > > > Is it really part of glibc? If n

Re: [PATCH v2 1/4] sha1sum: use AF_ALG when available

On Wed, Apr 25, 2018 at 7:41 PM, Bernhard Voelker wrote: > On 04/25/2018 01:26 PM, Matteo Croce wrote: >> +#define BLOCKSIZE 32768 > > Did you try different buffer sizes? > > In coreutils, we have a little script to determine the optimal BLOCKSIZE > - which is 128*10

Re: [PATCH v2 1/4] sha1sum: use AF_ALG when available

On Wed, Apr 25, 2018 at 7:34 PM, Assaf Gordon wrote: > Hello Matteo, > > On Wed, Apr 25, 2018 at 01:26:08PM +0200, Matteo Croce wrote: >> Linux supports accessing kernel crypto API via AF_ALG since >> version 2.6.38. Coreutils uses libcrypto when available and fall

Re: [PATCH v2 1/4] sha1sum: use AF_ALG when available

On Wed, Apr 25, 2018 at 2:25 PM, Dmitry V. Levin wrote: > On Wed, Apr 25, 2018 at 01:26:08PM +0200, Matteo Croce wrote: > [...] >> +#include >> + >> +#include >> + >> +#ifdef AF_ALG >> + >> +#include >> +#include >> +#incl

[PATCH v2 4/4] md5sum: use kernel crypto API

Use AF_ALG for md5 too Signed-off-by: Matteo Croce --- lib/md5.c | 20 +++- modules/crypto/md5 | 4 +++- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/lib/md5.c b/lib/md5.c index 68d00a6c7..8e60a549f 100644 --- a/lib/md5.c +++ b/lib/md5.c @@ -32,6

[PATCH v2 3/4] sha512sum: use kernel crypto API

Use AF_ALG for sha384 and sha512 too Signed-off-by: Matteo Croce --- lib/sha512.c | 34 -- modules/crypto/sha512 | 4 +++- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/lib/sha512.c b/lib/sha512.c index 8a6dd4e83..1d3471f20 100644

[PATCH v2 2/4] sha256sum: use kernel crypto API

Use AF_ALG for sha224 and sha256 too Signed-off-by: Matteo Croce --- lib/sha256.c | 34 -- modules/crypto/sha256 | 4 +++- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/lib/sha256.c b/lib/sha256.c index 85405b20f..7924040cd 100644

[PATCH v2 1/4] sha1sum: use AF_ALG when available

g.bin 752ef2367f479e79e4f0cded9c270c2890506ab0 2g.bin real0m3.164s user0m0.000s sys 0m3.162s Signed-off-by: Matteo Croce --- lib/af_alg.c| 124 lib/af_alg.h| 51 + lib/sha1.c

[PATCH v2 0/4] Use AF_ALG in checksum utilities

t to denote memory segments * avoid possible overflow by checking arguments size * return -EIO if sendfile() returns a short read/write count * fix a file descriptor leak when bind() returns error Matteo Croce (4): sha1sum: use AF_ALG when available sha256sum: use kernel crypto API sha51

Re: [PATCH 0/4] Use AF_ALG in checksum utilities

lib_SOURCES += af_alg.c endif or do you have a better solution? -- Matteo Croce per aspera ad upstream

Re: [PATCH 0/4] Use AF_ALG in checksum utilities

time ./sha1sum 2g.bin If you're using an HW crypto engine, locking can happen depending on how the hardware works, but such chips can be 100x faster than CPUs so it's not an issue. -- Matteo Croce per aspera ad upstream

Re: [PATCH 1/4] sha1sum: use AF_ALG when available

On Mon, Apr 23, 2018 at 5:07 PM, Tim Rühsen wrote: > > On 04/23/2018 01:17 PM, Matteo Croce wrote: >> +#include >> + >> +#include "af_alg.h" >> + >> +/* from linux/include/linux/fs.h: (INT_MAX & PAGE_MASK) */ >> +#define MAX_RW_C

Re: [PATCH 0/4] Use AF_ALG in checksum utilities

Emacs!' section is only needed in files that contains non-ASCII > characters. > Sorry, I just copied it from sha1.h :) > * Put '} else {' on 3 separate lines. > * Use 'size_t', not 'int', for variables that denote the length of a memory > segment. > Makes sense, will do. > Bruno > Regards, -- Matteo Croce per aspera ad upstream

Re: [PATCH 0/4] Use AF_ALG in checksum utilities

On Mon, Apr 23, 2018 at 3:42 PM, Tim Rühsen wrote: > On 04/23/2018 02:55 PM, Matteo Croce wrote: >> I understand that this won't compile on systems other than Linux, I >> will make a v2 which uses proper macros to avoid build failures on non >> Linux systems, if th

Re: [PATCH 0/4] Use AF_ALG in checksum utilities

On Mon, Apr 23, 2018 at 1:17 PM, Matteo Croce wrote: > Let md5sum and all sha*sum utilities use Linux kernel cryptographic API via > the > AF_ALG address family. > > Speed gain depends on the CPU type: > > Xeon E3-1265L V2: > > $ truncate -s 2GB 2g.bin

[PATCH 1/4] sha1sum: use AF_ALG when available

g.bin 752ef2367f479e79e4f0cded9c270c2890506ab0 2g.bin real0m3.164s user0m0.000s sys 0m3.162s Signed-off-by: Matteo Croce --- lib/af_alg.c| 102 lib/af_alg.h| 46 lib/sha1.c

[PATCH 0/4] Use AF_ALG in checksum utilities

=1 count=$len |src/$alg) [ "$hash1" = "$hash2" ] || exec echo "$alg: hash differs with data length $len!" done done Matteo Croce (4): sha1sum: use AF_ALG when available sha256sum: use kernel crypto API sha512sum: use kernel crypto API

[PATCH 3/4] sha512sum: use kernel crypto API

Use AF_ALG for sha384 and sha512 too Signed-off-by: Matteo Croce --- lib/sha512.c | 26 -- modules/crypto/sha512 | 5 - 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/lib/sha512.c b/lib/sha512.c index 8a6dd4e83..1ba3342bb 100644 --- a/lib

[PATCH 2/4] sha256sum: use kernel crypto API

Use AF_ALG for sha224 and sha256 too Signed-off-by: Matteo Croce --- lib/sha256.c | 26 -- modules/crypto/sha256 | 5 - 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/lib/sha256.c b/lib/sha256.c index 85405b20f..566c24a43 100644 --- a/lib

[PATCH 4/4] md5sum: use kernel crypto API

Use AF_ALG for md5 too Signed-off-by: Matteo Croce --- lib/md5.c | 14 +- modules/crypto/md5 | 5 - 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/lib/md5.c b/lib/md5.c index 68d00a6c7..8a70ddba1 100644 --- a/lib/md5.c +++ b/lib/md5.c @@ -36,6 +36,8