We have a pair of VPN/firewall systems running 5.0 GENERIC.MP#59 i386
(Lanner FW-7535B). These machines are also running OpenBGP, OpenOSPF,
CARP, pfsync, isakmpd and sasyncd.
Everything seems to be working fine except for VPN traffic. We have 33
active ipsec tunnels set up through isakmpd which are establishing
fine (verified through ipsecctl -sa) and they are even passing traffic
if it is originated on the internal interface IP of the firewall
(xx.yy.zz.1 or xx.yy.zz.3). Unfortunately, traffic from other hosts
located on the internal network (e.g. xx.yy.zz.8) is getting ICMP Host
Unreachable responses.
# tcpdump -ni em0
...
13:49:59.752891 xx.yy.zz.8 > vpn.net.1.1: icmp: echo request
13:49:59.752950 xx.yy.zz.3 > xx.yy.zz.8: icmp: host vpn.net.1.1
unreachable
...
net.inet.ip.forwarding=1 is set in sysctl.conf and non-VPN traffic is
traversing the firewalls with no problems.
I can see all the routes created by the ipsec flows using:
# netstat -rnf encap
Routing tables
Encap:
Source Port Destination Port Proto SA(Address/Proto/Type/Direction)
vpn.net/16 0 xx.yy.zz.0/26 0 0 vpn.peer.ip.addr/esp/use/in
xx.yy.zz.0/26 0 vpn.net/16 0 0 vpn.peer.ip.addr/esp/require/out
.
.
.
Because of a mis-communication, these machines currently have a full
set of internet routing tables through BGP (netstat -rn takes a
while). I'm not sure if this may have something to do with it. The
networks accessible through the VPN are not routable over the internet.
In pf.conf I have:
...
set block-policy return
set skip on { lo0 $pfsync_if enc0 }
block return log all
antispoof quick for { lo internal } inet
match out on egress scrub (no-df random-id)
match in on egress scrub (reassemble tcp)
pass on egress proto carp
pass on internal proto carp
pass in on egress proto esp from $vpn_peer to $external_carp_ip
pass in on egress proto udp from $vpn_peer to $external_carp_ip port
{isakmp, ipsec-nat-t}
pass inet proto icmp all icmp-type echoreq
pass out on egress inet proto udp from any to any port 33433 >< 33626
pass in proto igmp all allow-opts
pass out
pass in on internal inet
pass in proto tcp from $dmz_net to (egress) port bgp keep state
pass on egress inet proto ospf
...
I would really appreciate any assistance in getting this traffic
flowing since I'm running out of ideas on where to look. I have seen
some people suggesting that static routes need to be added to get
traffic flowing through the tunnels but others have said this isn't so
and the encap routing table on the machine seems to indicate that this
should just work.
Thanks,
- Aner
====== dmesg output ========
# dmesg
OpenBSD 5.0 (GENERIC.MP) #59: Wed Aug 17 10:19:44 MDT 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class)
1.67 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
real mem = 2137186304 (2038MB)
avail mem = 2092146688 (1995MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 10/06/10, BIOS32 rev. 0 @
0xf0010, SMBIOS rev. 2.6 @ 0xfc120 (24 entries)
bios0: vendor American Megatrends Inc. version "080015" date 10/06/2010
acpi0 at bios0: rev 0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4)
USB2(S4) USB3(S4) EUSB(S4) P0P4(S4) P0P5(S4) P0P6(S4) P0P7(S4)
P0P8(S4) P0P9(S4) HDAC(S4) USB4(S4) USB5(S4) USBE(S4) GBEC(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 166MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class)
1.67 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class)
1.67 GHz
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class)
1.67 GHz
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 1, remapped to apid 4
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 7 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus 2 (P0P5)
acpiprt4 at acpi0: bus 3 (P0P6)
acpiprt5 at acpi0: bus 4 (P0P7)
acpiprt6 at acpi0: bus 5 (P0P8)
acpiprt7 at acpi0: bus 6 (P0P9)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpibtn0 at acpi0: PWRB
bios0: ROM list: 0xc0000/0xda00!
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
vga1 at pci0 dev 2 function 0 "Intel Pineview Video" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp0 at vga1
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0 at vga1: apic 4 int 16
drm0 at inteldrm0
"Intel Pineview Video" rev 0x02 at pci0 dev 2 function 1 not configured
ppb0 at pci0 dev 28 function 0 "Intel 82801H PCIE" rev 0x03: apic 4
int 22
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00:
msi, address 00:90:0b:1c:a9:06
ppb1 at pci0 dev 28 function 1 "Intel 82801H PCIE" rev 0x03: apic 4
int 23
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi,
address 00:90:0b:1c:a9:07
ppb2 at pci0 dev 28 function 2 "Intel 82801H PCIE" rev 0x03: apic 4
int 20
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi,
address 00:90:0b:1c:a9:08
ppb3 at pci0 dev 28 function 3 "Intel 82801H PCIE" rev 0x03: apic 4
int 21
pci4 at ppb3 bus 4
em3 at pci4 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi,
address 00:90:0b:1c:a9:09
ppb4 at pci0 dev 28 function 4 "Intel 82801H PCIE" rev 0x03: apic 4
int 22
pci5 at ppb4 bus 5
em4 at pci5 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi,
address 00:90:0b:1c:a9:0a
ppb5 at pci0 dev 28 function 5 "Intel 82801H PCIE" rev 0x03: apic 4
int 23
pci6 at ppb5 bus 6
em5 at pci6 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi,
address 00:90:0b:1c:a9:0b
uhci0 at pci0 dev 29 function 0 "Intel 82801H USB" rev 0x03: apic 4
int 23
uhci1 at pci0 dev 29 function 1 "Intel 82801H USB" rev 0x03: apic 4
int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801H USB" rev 0x03: apic 4
int 18
ehci0 at pci0 dev 29 function 7 "Intel 82801H USB" rev 0x03: apic 4
int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xf3
pci7 at ppb6 bus 7
ichpcib0 at pci0 dev 31 function 0 "Intel 82801HBM LPC" rev 0x03: PM
disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801HBM IDE" rev 0x03: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
ahci0 at pci0 dev 31 function 2 "Intel 82801HBM AHCI" rev 0x03: msi,
AHCI 1.1
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, KINGSTON SS100S2, D100> SCSI3
0/direct fixed t10.ATA_KINGSTON_SS100S216G_16GB40013375_
sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801H SMBus" rev 0x03: apic
4 int 17
iic0 at ichiic0
spdmem0 at iic0 addr 0x51: 2GB DDR2 SDRAM non-parity PC2-5300CL5 SO-DIMM
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: W83627THF rev 0x85
lm1 at wbsio0 port 0xa00/8: W83627THF
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (2271eac45c0a2f56.a) swap on sd0b dump on sd0b
carp2: state transition: BACKUP -> MASTER
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
