>Synopsis: AF_UNIX SOCK_SEQPACKET sendmsg w/ MSG_EOR truncates recvmsg
>Category: system
>Environment:
System : OpenBSD 7.3
Details : OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT
2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
sendmsg with MSG_EOR on a SOCK_SEQPACKET socket appears to cause the
receiving end to get truncated output (and MSG_TRUNC set). I'm only
sending a 153-byte record, so unlikely a problem with system limits.
This behavior does not exhibit on FreeBSD nor Linux, and I didn't see
where MSG_EOR could result in a truncated message in any man pages;
especially when I'm using AF_UNIX SOCK_SEQPACKET.
Using flags=0 with sendmsg appears to work fine, so perhaps I'm
misunderstanding MSG_EOR entirely. Or it's both a misunderstanding
on my part AND also a bug in OpenBSD.
>How-To-Repeat:
Here is a standalone C program reproducing the problem. I extracted
it from code I've used on FreeBSD and Linux systems since 2021.
It can be compiled with: cc -o seqtrunc -Wall seqtrunc.c
Comments are inline:
/*
* Using sendmsg(..., MSG_EOR) on a SOCK_SEQPACKET socket
* appears to cause the recvmsg() caller to receive truncated
* data (and set MSG_TRUNC) on OpenBSD 7.3.
*
* I'm not sure if:
* a) this is a bug in OpenBSD
* b) I'm misunderstanding MSG_EOR with SOCK_SEQPACKET
* c) both of the above
*
* I extracted this test case from a codebase which I've been
* using on FreeBSD 12.x and numerous Linux installs since 2021.
* I normally send data between processes, but this truncation
* happens within the same process, too.
*/
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <assert.h>
#define NSEND 153
#define NRECV (4096 * 33)
#define SEND_FD_CAPA 10
#define SEND_FD_SPACE (SEND_FD_CAPA * sizeof(int))
union my_cmsg {
struct cmsghdr hdr;
char pad[sizeof(struct cmsghdr) + 16 + SEND_FD_SPACE];
};
static void do_sendmsg(int fd)
{
/*
* Is setting MSG_EOR wrong here? OpenBSD recvmsg ends
* up truncating data and setting MSG_TRUNC below.
* OTOH, FreeBSD and Linux work fine with (sflags = 0),
* here, too...
*/
int sflags = MSG_EOR;
/*
* I normally have several FDs to send, but the
* discrepancy happens even with nfds == 0
*/
int i, nfds = 1;
struct msghdr msg = { 0 };
union my_cmsg cmsg = { 0 };
char to_send[NSEND];
struct iovec iov;
int *fdp;
ssize_t sent;
memset(to_send, 'a', sizeof(to_send));
to_send[NSEND - 1] = 'b'; // for kdump || strace output
iov.iov_base = to_send;
iov.iov_len = sizeof(to_send);
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = &cmsg.hdr;
msg.msg_controllen = CMSG_SPACE(nfds * sizeof(int));
cmsg.hdr.cmsg_level = SOL_SOCKET;
cmsg.hdr.cmsg_type = SCM_RIGHTS;
cmsg.hdr.cmsg_len = CMSG_LEN(nfds * sizeof(int));
fdp = (int *)CMSG_DATA(&cmsg.hdr);
for (i = 0; i < nfds; i++)
*fdp++ = i;
sent = sendmsg(fd, &msg, sflags);
fprintf(stderr, "%d sent: %zd of %zu\n",
(int)getpid(), sent, iov.iov_len);
}
static void do_recvmsg(int fd)
{
union my_cmsg cmsg = { 0 };
struct msghdr msg = { 0 };
struct iovec iov;
ssize_t r;
char recvbuf[NRECV];
iov.iov_base = recvbuf;
iov.iov_len = sizeof(recvbuf);
msg.msg_iov = &iov;
msg.msg_iovlen = 1;
msg.msg_control = &cmsg.hdr;
msg.msg_controllen = CMSG_SPACE(SEND_FD_SPACE);
r = recvmsg(fd, &msg, 0);
/*
* FreeBSD and Linux receive everything sent, here, but
* OpenBSD only gets 144 bytes and sets MSG_TRUNC here:
*/
fprintf(stderr, "%d recv: %zd of %zu "
"EOR=%d TRUNC=%d CTRUNC=%d OOB=%d\n",
getpid(), r, iov.iov_len,
msg.msg_flags & MSG_EOR,
msg.msg_flags & MSG_TRUNC,
msg.msg_flags & MSG_CTRUNC,
msg.msg_flags & MSG_OOB);
if (r > 0 && cmsg.hdr.cmsg_level == SOL_SOCKET &&
cmsg.hdr.cmsg_type == SCM_RIGHTS) {
size_t len = cmsg.hdr.cmsg_len;
int *fdp = (int *)CMSG_DATA(&cmsg.hdr);
size_t i;
for (i = 0; CMSG_LEN((i + 1) * sizeof(int)) <= len; i++)
fprintf(stderr, "%d recv fd=%d\n",
(int)getpid(), *fdp++);
}
}
int main(void)
{
int fds[2];
int rc = socketpair(AF_UNIX, SOCK_SEQPACKET, 0, fds);
assert(rc == 0);
do_sendmsg(fds[0]);
do_recvmsg(fds[1]);
return 0;
}
>Fix:
Setting `sflags = 0' in the above code seems to avoid the
problem, but I'm not sure if it's correct to do so because
I want to rely on the kernel to handle record boundaries.
dmesg:
OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17162948608 (16367MB)
avail mem = 16623382528 (15853MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbffffc80 (17 entries)
bios0: vendor SeaBIOS version "1.14.0-2" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.5+, 3300.16 MHz, 06-06-03
cpu0:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: QEMU Virtual CPU version 2.5+, 3300.22 MHz, 06-06-03
cpu1:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: QEMU Virtual CPU version 2.5+, 3300.22 MHz, 06-06-03
cpu2:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu2: smt 0, core 0, package 2
cpu3 at mainbus0: apid 3 (application processor)
cpu3: QEMU Virtual CPU version 2.5+, 3300.22 MHz, 06-06-03
cpu3:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu3: smt 0, core 0, package 3
cpu4 at mainbus0: apid 4 (application processor)
cpu4: QEMU Virtual CPU version 2.5+, 3300.44 MHz, 06-06-03
cpu4:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu4: smt 0, core 0, package 4
cpu5 at mainbus0: apid 5 (application processor)
cpu5: QEMU Virtual CPU version 2.5+, 3300.21 MHz, 06-06-03
cpu5:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu5: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu5: smt 0, core 0, package 5
cpu6 at mainbus0: apid 6 (application processor)
cpu6: QEMU Virtual CPU version 2.5+, 3300.38 MHz, 06-06-03
cpu6:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu6: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu6: smt 0, core 0, package 6
cpu7 at mainbus0: apid 7 (application processor)
cpu7: QEMU Virtual CPU version 2.5+, 3300.36 MHz, 06-06-03
cpu7:
FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu7: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu7: smt 0, core 0, package 7
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0
wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:12:34:56
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 8192MB, 512 bytes/sector, 16777216 sectors
virtio1: msix per-VQ
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (8243d8bd4db9b04b.a) swap on sd0b dump on sd0b
usbdevs:
usbdevs: no USB controllers found
