Hi all, I got to admit I've seen isakmpd dying on 5.9* (amd64 on VMware). But after having to deal with half a dozen peers all over Europe using different proprietary solutions a cronjob like "rcctl ls faulty | grep isakmpd && rcctl restart…" worked well enough for me.
I won't be able to test with the setup at work but I got a little VPS running 6.1 I could use (and update to -STABLE if necessary). We probably won't get to over 100 tunnels but I've seen the problem with ~8 tunnels. The question would be if this problem would even show up in a homogeneous OpenBSD network… Regards, Florian *) the central hub isn't my problem anymore, and it will take some time to convince my replacement there to update to 6.1… Am 29. Mai 2017 09:26:18 MESZ schrieb Alexis VACHETTE <avache...@sisteer.com>: >Hi Michał, > >I'm having same issue without 100 ipsec tunnels and dedicated hardware. > >Unfortunately it's a production environment so I can't really >troubleshooting this issue to track down the culprit. > >Anyway maybe it's not related to your issue. > >Regards, >Alexis. >On 28/05/2017 14:31, Michał Koc wrote: >> Hi all, >> >> I'm running 6.0/amd64 inside KVM/Quemu with over 100 ipsec tunnels. >> >> Everything was running just fine when the number of tunnels was >lower. >> But as we have been setting up more and more tunnels we suddenly run >> on problems. >> The isakmpd deaemon keeps dying quietly. Probably I'm running out of >> something, but I need some help to find out what it is and how to >> monitor it and tweak. >> >> Thank You in advance. >> >> Best Regards >> M.K. >> >> root@vgate0:/root# netstat -m >> 215 mbufs in use: >> 163 mbufs allocated to data >> 46 mbufs allocated to packet headers >> 6 mbufs allocated to socket names and addresses >> 160/920/6144 mbuf 2048 byte clusters in use (current/peak/max) >> 0/8/6144 mbuf 4096 byte clusters in use (current/peak/max) >> 0/8/6144 mbuf 8192 byte clusters in use (current/peak/max) >> 0/14/6146 mbuf 9216 byte clusters in use (current/peak/max) >> 0/10/6150 mbuf 12288 byte clusters in use (current/peak/max) >> 0/8/6144 mbuf 16384 byte clusters in use (current/peak/max) >> 0/8/6144 mbuf 65536 byte clusters in use (current/peak/max) >> 2760 Kbytes allocated to network (13% in use) >> 0 requests for memory denied >> 0 requests for memory delayed >> 0 calls to protocol drain routines >> >> Sample tail of the log: >> When I run "isakmpd -K -d -DA=10": >> 142043.246192 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx > >> SPI 0x42f03e5d >> 142043.246209 Timr 10 timer_add_event: event >> sa_soft_expire(0x1fb9d0bdf400) added before >> sa_soft_expire(0x1fb9c8f05400), expiration in 25056s >> 142043.246223 Timr 10 timer_add_event: event >> sa_hard_expire(0x1fb9d0bdf400) added before >> sa_soft_expire(0x1fb9dd458200), expiration in 28800s >> 142043.246326 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx > >> SPI 0x3ffa5955 >> 142043.268229 Default responder_recv_HASH_SA_NONCE: KEY_EXCH payload >> without a group desc. attribute >> 142043.268250 Default dropped message from xxx.xxx.xxx.xxx port 500 >> due to notification type NO_PROPOSAL_CHOSEN >> 142043.268281 Timr 10 timer_add_event: event >> exchange_free_aux(0x1fb9a5336400) added before >> sa_soft_expire(0x1fba0d6a2a00), expiration in 120s >> 142043.268289 Exch 10 exchange_establish_p2: 0x1fb9a5336400 <unnamed> > >> <no policy> policy initiator phase 2 doi 1 exchange 5 step 0 >> 142043.268295 Exch 10 exchange_establish_p2: icookie 8c58f4e7f8269ed3 > >> rcookie 0fe2d7657125a339 >> 142043.268301 Exch 10 exchange_establish_p2: msgid de2c5cc3 sa_list >> 142043.269079 Timr 10 timer_add_event: event >> message_send_expire(0x1fb994136900) added before >> connection_checker(0x1fb9b2646280), expiration in 7s >> 142043.269614 Exch 10 exchange_finalize: 0x1fb9a5336400 <unnamed> <no > >> policy> policy initiator phase 2 doi 1 exchange 5 step 1 >> 142043.269630 Exch 10 exchange_finalize: icookie 8c58f4e7f8269ed3 >> rcookie 0fe2d7657125a339 >> 142043.269637 Exch 10 exchange_finalize: msgid de2c5cc3 sa_list >> 142043.269653 Timr 10 timer_remove_event: removing event >> exchange_free_aux(0x1fb9a5336400) >> 142043.289465 Timr 10 timer_remove_event: removing event >> message_send_expire(0x1fb994136900) >> 142043.289513 Exch 10 exchange_finalize: 0x1fb972b59400 >> from-xxx.xxx.xxx.xxx/24-to-xxx.xxx.xxx.xxx/24 <no policy> policy >> responder phase 2 doi 1 exchange 32 step 2 >> 142043.289521 Exch 10 exchange_finalize: icookie 8c58f4e7f8269ed3 >> rcookie 0fe2d7657125a339 >> 142043.289528 Exch 10 exchange_finalize: msgid de079ef6 sa_list >> 0x1fb9dd458800 0x1fb985d09e00 >> 142043.289578 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx > >> SPI 0xe5d04953 >> 142043.289594 Timr 10 timer_add_event: event >> sa_soft_expire(0x1fb9dd458800) added before >> sa_soft_expire(0x1fba1d81de00), expiration in 3279s >> 142043.289608 Timr 10 timer_add_event: event >> sa_hard_expire(0x1fb9dd458800) added before >> sa_soft_expire(0x1fba2c980800), expiration in 3600s >> 142043.289710 Sdep 10 pf_key_v2_set_spi: satype 2 dst xxx.xxx.xxx.xxx > >> SPI 0x4d895568 >> root@vgate0:/root# >> >> OpenBSD 6.0-stable (GENERIC.MP) #0: Sat Feb 4 21:55:17 CET 2017 >> root@amd64.vcomp:/usr/src/sys/arch/amd64/compile/GENERIC.MP >> real mem = 1056956416 (1007MB) >> avail mem = 1020506112 (973MB) >> mpath0 at root >> scsibus0 at mpath0: 256 targets >> mainbus0 at root >> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1dc0 (11 entries) >> bios0: vendor Bochs version "Bochs" date 01/01/2011 >> bios0: Bochs Bochs >> acpi0 at bios0: rev 0 >> acpi0: sleep states S3 S4 S5 >> acpi0: tables DSDT FACP SSDT APIC HPET >> acpi0: wakeup devices >> acpitimer0 at acpi0: 3579545 Hz, 24 bits >> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat >> cpu0 at mainbus0: apid 0 (boot processor) >> cpu0: QEMU Virtual CPU version 2.1.2, 3492.32 MHz >> cpu0: >> >FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF,ABM >> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB > >> 64b/line 16-way L2 cache >> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries >direct-mapped >> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries >direct-mapped >> cpu0: smt 0, core 0, package 0 >> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges >> cpu0: apic clock running at 1000MHz >> cpu1 at mainbus0: apid 1 (application processor) >> cpu1: QEMU Virtual CPU version 2.1.2, 3491.95 MHz >> cpu1: >> >FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF,ABM >> cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB > >> 64b/line 16-way L2 cache >> cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries >direct-mapped >> cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries >direct-mapped >> cpu1: smt 0, core 0, package 1 >> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins >> acpihpet0 at acpi0: 100000000 Hz >> acpiprt0 at acpi0: bus 0 (PCI0) >> acpicpu0 at acpi0: C1(@1 halt!) >> acpicpu1 at acpi0: C1(@1 halt!) >> "ACPI0006" at acpi0 not configured >> "PNP0303" at acpi0 not configured >> "PNP0F13" at acpi0 not configured >> "PNP0700" at acpi0 not configured >> "ACPI0007" at acpi0 not configured >> "ACPI0007" at acpi0 not configured >> pvbus0 at mainbus0: KVM >> pci0 at mainbus0 bus 0 >> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 >> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 >> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, >> channel 0 wired to compatibility, channel 1 wired to compatibility >> pciide0: channel 0 disabled (no drives) >> pciide0: channel 1 disabled (no drives) >> uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 >> int 11 >> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic > >> 0 int 9 >> iic0 at piixpm0 >> vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00 >> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) >> wsdisplay0: screen 1-5 added (80x25, vt100 emulation) >> virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00 >> vio0 at virtio0: address 00:50:56:00:0e:94 >> virtio0: msix shared >> virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00 >> vio1 at virtio1: address 52:54:00:05:43:8a >> virtio1: msix shared >> virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00 >> vioblk0 at virtio2 >> scsibus1 at vioblk0: 2 targets >> sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct > >> fixed >> sd0: 4096MB, 512 bytes/sector, 8388608 sectors >> virtio2: msix shared >> virtio3 at pci0 dev 6 function 0 "Qumranet Virtio Memory" rev 0x00 >> viomb0 at virtio3 >> virtio3: apic 0 int 10 >> isa0 at pcib0 >> isadma0 at isa0 >> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 >> fd0 at fdc0 drive 1: density unknown >> pckbc0 at isa0 port 0x60/5 irq 1 irq 12 >> pckbd0 at pckbc0 (kbd slot) >> wskbd0 at pckbd0: console keyboard, using wsdisplay0 >> pms0 at pckbc0 (aux slot) >> wsmouse0 at pms0 mux 0 >> pcppi0 at isa0 port 0x61 >> spkr0 at pcppi0 >> usb0 at uhci0: USB revision 1.0 >> uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 >> nvram: invalid checksum >> vscsi0 at root >> scsibus2 at vscsi0: 256 targets >> softraid0 at root >> scsibus3 at softraid0: 256 targets >> root on sd0a (51b2d1225ee6d760.a) swap on sd0b dump on sd0b >> >> -- >> M.K. >>
