Are you using altq?
On 2011-02-07, Jeff Ross <jr...@openvistas.net> wrote: > On 02/07/11 12:33, Christiano F. Haesbaert wrote: >> On 7 February 2011 16:53, Jeff Ross<jr...@openvistas.net> wrote: >>> Greetings, >>> >>> I found this explanation for the following OpenVPN error >>> "write UDPv4: No buffer space available (code=55)" >>> >>> "No buffer space available is reported by the kernel network layer when >>> thekernel's transmit buffer is full - i.e. the network connection is >> running >>> at full speed, and there's no space for more. This means that something is >>> filling it, such as other OpenVPN requests, or simultaneous traffic. When >>> this happens, the UDP packet is dropped, and the traffic is shaped to the >>> size of the pipe it is travelling on." >>> >>> http://readlist.com/lists/lists.sourceforge.net/openvpn-users/0/4136.html >>> >>> I also got a very similar no buffer space available message while running >>> traceroute earlier today. >>> >>> However, unlike the OP in the thread above, this is kicking my one VPN >> user >>> off every couple of minutes. >>> >>> I have plenty of ram (4GB--full dmesg below) in these identical and CARPed >>> firewalls--is there a sysctl I can bump to give the kernel more transmit >>> space? >>> >>> Thanks, >>> >>> Jeff Ross >>> >>> OpenBSD 4.9-beta (GENERIC.MP) #764: Thu Jan 27 13:19:27 MST 2011 >>> t...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP >>> cpu0: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz >>> cpu0: >>> >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS >> H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS >>> ,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR >>> real mem = 3757522944 (3583MB) >>> avail mem = 3685912576 (3515MB) >>> mainbus0 at root >>> bios0 at mainbus0: AT/286+ BIOS, date 03/29/05, BIOS32 rev. 0 @ 0xf0010, >>> SMBIOS rev. 2.3 @ 0xfa380 (61 entries) >>> bios0: vendor American Megatrends Inc. version "080010" date 03/29/2005 >>> bios0: SiMech R200 >>> acpi0 at bios0: rev 2 >>> acpi0: sleep states S0 S1 S4 S5 >>> acpi0: tables DSDT FACP APIC OEMB >>> acpi0: wakeup devices PXHA(S4) PXHB(S4) EPA0(S4) EPA1(S4) EPB0(S4) EPB1(S4) >>> EPC0(S4) P0P1(S4) MC97(S4) USB1(S1) U >>> SB2(S1) EUSB(S4) PS2K(S4) PS2M(S4) P0PC(S4) SLPB(S4) >>> acpitimer0 at acpi0: 3579545 Hz, 24 bits >>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat >>> cpu0 at mainbus0: apid 0 (boot processor) >>> cpu0: apic clock running at 200MHz >>> cpu1 at mainbus0: apid 6 (application processor) >>> cpu1: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz >>> cpu1: >>> >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS >> H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS >>> ,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR >>> cpu2 at mainbus0: apid 1 (application processor) >>> cpu2: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz >>> cpu2: >>> >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS >> H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS >>> ,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR >>> cpu3 at mainbus0: apid 7 (application processor) >>> cpu3: Intel(R) Xeon(TM) CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz >>> cpu3: >>> >> FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS >> H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS >>> ,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16,xTPR >>> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins >>> ioapic1 at mainbus0: apid 11 pa 0xfec10000, version 20, 24 pins >>> ioapic1: misconfigured as apic 9, remapped to apid 11 >>> ioapic2 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins >>> ioapic2: misconfigured as apic 10, remapped to apid 9 >>> ioapic3 at mainbus0: apid 10 pa 0xfec80400, version 20, 24 pins >>> ioapic3: misconfigured as apic 11, remapped to apid 10 >>> acpiprt0 at acpi0: bus 0 (PCI0) >>> acpiprt1 at acpi0: bus 1 (EPA0) >>> acpiprt2 at acpi0: bus 3 (PXHA) >>> acpiprt3 at acpi0: bus 2 (PXHB) >>> acpiprt4 at acpi0: bus 4 (EPA1) >>> acpiprt5 at acpi0: bus 6 (P0P1) >>> acpiprt6 at acpi0: bus 5 (P0PC) >>> acpicpu0 at acpi0 >>> acpicpu1 at acpi0 >>> acpicpu2 at acpi0 >>> acpicpu3 at acpi0 >>> acpibtn0 at acpi0: PWRB >>> acpibtn1 at acpi0: SLPB >>> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x800 0xc8800/0x1000 0xc9800/0x1000 >>> ipmi at mainbus0 not configured >>> pci0 at mainbus0 bus 0: configuration mode 1 (bios) >>> pchb0 at pci0 dev 0 function 0 "Intel E7320 Host" rev 0x0c >>> ppb0 at pci0 dev 2 function 0 "Intel E7520 PCIE" rev 0x0c >>> pci1 at ppb0 bus 1 >>> ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09 >>> pci2 at ppb1 bus 3 >>> mpi0 at pci2 dev 5 function 0 "Symbios Logic 53c1030" rev 0xc1: apic 9 int >> 2 >>> (irq 6) >>> scsibus0 at mpi0: 16 targets, initiator 7 >>> ppb2 at pci1 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09 >>> pci3 at ppb2 bus 2 >>> em0 at pci3 dev 1 function 0 "Intel PRO/1000MT (82546GB)" rev 0x03: apic 10 >>> int 0 (irq 10), address 00:1b:21:15:6 >>> a:82 >>> em1 at pci3 dev 1 function 1 "Intel PRO/1000MT (82546GB)" rev 0x03: apic 10 >>> int 1 (irq 11), address 00:1b:21:15:6 >>> a:83 >>> ppb3 at pci0 dev 3 function 0 "Intel E7520 PCIE" rev 0x0c >>> pci4 at ppb3 bus 4 >>> ppb4 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02 >>> pci5 at ppb4 bus 5 >>> em2 at pci5 dev 1 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: apic 11 >>> int 2 (irq 9), address 00:30:48:87:2e >>> :b0 >>> em3 at pci5 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: apic 11 >>> int 3 (irq 9), address 00:30:48:87:2e >>> :b1 >>> uhci0 at pci0 dev 29 function 0 "Intel 6300ESB USB" rev 0x02: apic 8 int 16 >>> (irq 10) >>> uhci1 at pci0 dev 29 function 1 "Intel 6300ESB USB" rev 0x02: apic 8 int 19 >>> (irq 7) >>> "Intel 6300ESB WDT" rev 0x02 at pci0 dev 29 function 4 not configured >>> "Intel 6300ESB APIC" rev 0x02 at pci0 dev 29 function 5 not configured >>> ehci0 at pci0 dev 29 function 7 "Intel 6300ESB USB" rev 0x02: apic 8 int 23 >>> (irq 5) >>> usb0 at ehci0: USB revision 2.0 >>> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 >>> ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x0a >>> pci6 at ppb5 bus 6 >>> vga1 at pci6 dev 2 function 0 "ATI Rage XL" rev 0x27 >>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) >>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation) >>> ichpcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02 >>> pciide0 at pci0 dev 31 function 1 "Intel 6300ESB IDE" rev 0x02: DMA, >> channel >>> 0 configured to compatibility, chann >>> el 1 configured to compatibility >>> pciide0: channel 0 disabled (no drives) >>> pciide0: channel 1 disabled (no drives) >>> ichiic0 at pci0 dev 31 function 3 "Intel 6300ESB SMBus" rev 0x02: apic 8 >> int >>> 17 (irq 0) >>> iic0 at ichiic0 >>> lm1 at iic0 addr 0x2c: W83627HF >>> lm2 at iic0 addr 0x2f: W83792D rev D >>> spdmem0 at iic0 addr 0x51: 1GB DDR SDRAM registered ECC PC2700CL2.5 >>> spdmem1 at iic0 addr 0x53: 1GB DDR SDRAM registered ECC PC2700CL2.5 >>> spdmem2 at iic0 addr 0x55: 1GB DDR SDRAM registered ECC PC2700CL2.5 >>> spdmem3 at iic0 addr 0x56: 1GB DDR SDRAM registered ECC PC2700CL2.5 >>> usb1 at uhci0: USB revision 1.0 >>> uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 >>> usb2 at uhci1: USB revision 1.0 >>> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 >>> isa0 at ichpcib0 >>> isadma0 at isa0 >>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo >>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo >>> pckbc0 at isa0 port 0x60/5 >>> pckbd0 at pckbc0 (kbd slot) >>> pckbc0: using irq 1 for kbd slot >>> wskbd0 at pckbd0: console keyboard, using wsdisplay0 >>> pms0 at pckbc0 (aux slot) >>> pckbc0: using irq 12 for aux slot >>> wsmouse0 at pms0 mux 0 >>> pcppi0 at isa0 port 0x61 >>> spkr0 at pcppi0 >>> wbsio0 at isa0 port 0x2e/2: W83627HF rev 0x41 >>> lm3 at wbsio0 port 0x290/8: W83627HF >>> lm1 detached >>> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 >>> mtrr: Pentium Pro MTRR support >>> umass0 at uhub0 port 3 configuration 1 interface 0 "Imation Nano Pro" rev >>> 2.00/1.00 addr 2 >>> umass0: using SCSI over Bulk-Only >>> scsibus1 at umass0: 2 targets, initiator 0 >>> sd0 at scsibus1 targ 1 lun 0:<Imation, Nano Pro, PMAP> SCSI0 0/direct >>> removable >>> sd0: 3822MB, 512 bytes/sec, 7827456 sec total >>> umass1 at uhub0 port 4 configuration 1 interface 0 "Imation Nano Pro" rev >>> 2.00/1.00 addr 3 >>> umass1: using SCSI over Bulk-Only >>> scsibus2 at umass1: 2 targets, initiator 0 >>> sd1 at scsibus2 targ 1 lun 0:<Imation, Nano Pro, PMAP> SCSI0 0/direct >>> removable >>> sd1: 3822MB, 512 bytes/sec, 7827456 sec total >>> vscsi0 at root >>> scsibus3 at vscsi0: 256 targets >>> softraid0 at root >>> root on sd0a swap on sd0b dump on sd0b >>> >>> >> >> code 55 is ENOBUFS. >> This is weird since the default UDP send buffer is 9kb: >> u_int udp_sendspace = 9216; /* really max datagram size */ >> >> Coincidently I was looking this afternoon for what seems to be a bug, >> it seems poll(2) and friends return a writable state for the socket >> disrespecting SO_SNDLOWAT in UDP. >> Although I don't think it is related to your problem, and I don't >> believe OpenVPN can fill up 9k before something being sent to the >> ethernet device. >> >> You're probably hitting this in udp_usrreq.c: >> >> /* >> * Calculate data length and get a mbuf >> * for UDP and IP headers. >> */ >> M_PREPEND(m, sizeof(struct udpiphdr), M_DONTWAIT); >> if (m == NULL) { >> error = ENOBUFS; >> goto bail; >> } >> >> >> How is the load on the system ? Can you give us some netstat output ? > > Thanks for the reply! > > The load on the network is light. Here are two separate runs, a couple > of minutes apart. > > jross@samsara1:/home/jross $ netstat -w 5 > carp0 in carp0 out total in total out > packets errs packets errs colls packets errs packets errs colls > 1328473 2 14842 66 0 6186480 3 3377032 66 0 > 304 0 5 0 0 1689 0 1001 0 0 > 332 0 5 0 0 1829 0 1108 0 0 > 295 0 5 0 0 1590 0 857 0 0 > 296 0 5 0 0 1615 0 888 0 0 > 300 0 5 0 0 1634 0 865 0 0 > 278 0 5 0 0 1513 0 801 0 0 > 247 0 5 0 0 1258 0 684 0 0 > 257 0 5 0 0 1410 0 743 0 0 > 280 0 5 0 0 1565 0 831 0 0 > 258 0 4 0 0 1416 0 743 0 0 > 260 0 5 0 0 1419 0 751 0 0 > 246 0 5 0 0 1412 0 728 0 0 > 264 0 5 0 0 1443 0 759 0 0 > 250 0 5 0 0 1417 0 737 0 0 > 247 0 5 0 0 1365 0 714 0 0 > 276 0 5 0 0 1488 0 780 0 0 > 262 0 5 0 0 1464 0 758 0 0 > 248 0 5 0 0 1381 0 719 0 0 > 251 0 5 0 0 1403 0 725 0 0 > 271 0 5 0 0 1442 0 762 0 0 > > carp0 in carp0 out total in total out > packets errs packets errs colls packets errs packets errs colls > 1344207 2 15182 66 0 6156025 3 3265613 66 0 > 0 0 5 0 0 20 0 40 0 0 > 0 0 5 0 0 30 0 44 0 0 > 3 0 5 0 0 30 0 43 0 0 > 60 0 5 0 0 283 0 173 0 0 > 228 0 5 0 0 1233 0 636 0 0 > 250 0 5 0 0 1410 0 732 0 0 > 238 0 5 0 0 1306 0 682 0 0 > 268 0 5 0 0 1445 0 752 0 0 > 240 0 5 0 0 1348 0 705 0 0 > 221 0 5 0 0 1193 0 628 0 0 > 210 0 5 0 0 1201 0 618 0 0 > 256 0 5 0 0 1418 0 745 0 0 > 235 0 5 0 0 1348 0 698 0 0 > 249 0 5 0 0 1350 0 706 0 0 > 250 0 5 0 0 1351 0 707 0 0 > 229 0 5 0 0 1334 0 688 0 0 > 265 0 5 0 0 1411 0 738 0 0 > 235 0 4 0 0 1354 0 698 0 0 > 242 0 5 0 0 1343 0 699 0 0 > 262 0 5 0 0 1442 0 759 0 0 > > Jeff
