On 14/10/14 17:03, Fede wrote:
Hello,
I am experiencing some problems with OpenBSD 5.5, specifically with
CARP and VLAN.
My setup is: 2 Dell R415 servers, MASTER (system-1)/BACKUP (system-2)
with 8 vlan interfaces (2 WAN + 6 LAN) + 49 carp interfaces (40 WAN +
9 LAN) + pfsync interface + pf configured with several rules.
Switching is provided by 4 Dell 5524 (two for the LAN interfaces, two
for the WAN interfaces).
Why do you have so many CARP interfaces?
Generally it's good practice to have one CARP interface per broadcast
domain / VLAN etc, and have all your alias IP addresses defined in that
one CARP interface.
NB; when adding;
inet alias <ipaddress> <mask> Always set the mask for each alias to
255.255.255.255
This is apparently correct according to the devs. cite; something I was
told a long time ago even though you'll get a spurious error in the logs
at fail-over time..
sw-dmz-1-------------sw-dmz-2
| |
system-1----pfsync---system-2
| |
sw-lan-1-------------sw-lan-2
On the switches, ports are configured a follows:
interface gigabitethernet1/0/1
spanning-tree portfast
switchport mode trunk
This is what happens: if system-1 is rebooted, system-2 becomes the
new MASTER. When system-1 comes back, some carp interfaces (randomly,
even from different physical interfaces) on system-2 stay in MASTER
state, so I have a kind of split brain.
All of the above didn't happens with OpenBSD 5.4, so I can't figure
out where the problem is.
For debugging, I've removed 2 switches, so now I have only one switch
per interface.
I've noticed that if I turn off PF, the configuration works perfectly.
I then tried with a simpler configuration of pf (pass quick in, pass
quick out), and again it doesn't works, so the problem seems to be
with pf.
I've noticed that when the split is ongoing, with tcpdump I can see
advertisement packet from both of servers, as if they are ignoring
each other.
I tried to remove some carp interface, then add them again one at a
time, and I discovered that when I have 19 carp interfaces, the
behaviour described above starts again.
When the split is ongoing, if I run ksh /etc/netstart carpXX on
system-1, system-2 switch the same interface in BACKUP mode.
Does it always start once you get to 19?
I seem to remember having to increase the number of BPF devices which
high numbers of VLANS etc..
for(( i=10; i <= 30; i++ )); do mknod /dev/bpf$i c 23 $i; done
for(( i=10; i <= 30; i++ )); do chmod o-r,g-r /dev/bpf$i; done
This is my test configuration (not patched, but in the production
environment is patched to the last errata release):
# uname -a (both machines)
OpenBSD system-1.my.domain 5.5 GENERIC#271 amd64
# grep carp /etc/sysctl.conf (both machines)
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=7
# ifconfig -g carp (both machines)
carp: carp demote count 0
It's nice to keep them at 1 so if you loose control of one of the
servers, you can increase the other to 0 and preempt it remotely quickly
and the fix it in your own time.. If it's local then doesn't matter so
much I guess..
# cat /etc/pf.conf (both machines)
set skip on { lo0 em0 }
pass quick proto { carp pfsync }
pass in quick
pass out quick
This is an example of how I configured interfaces on the servers (carp5):
# carp5 system-1
inet 192.168.26.17 255.255.255.240 192.168.26.31 vhid 125 carpdev
vlan3 pass password advskew 10
# carp5 system-2
inet 192.168.26.17 255.255.255.240 192.168.26.31 vhid 125 carpdev
vlan3 pass password advskew 200
And this is vlan3:
inet 192.168.26.29 255.255.255.240 NONE vlan 3 vlandev bnx1
We generally define the broadcast (even though its implied by the mask)
and use "carppeer" to make it unicast..
In bnx0 and bnx1 (vlan root interfaces) I've just put "up".
# pfsync0 system-1
up syncdev em0 syncpeer 10.10.26.4 defer
# pfsync0 system-2
up syncdev em0 syncpeer 10.10.26.3 defer
Why are you using defer? I'm guessing you know what this does and that
it slows things down..
Usually only see this on systems with BGP (incase packets are recieved
on the backup), or on active-active systems.
# /etc/hostname.em0 system-1
inet 10.10.26.3 255.255.255.0 NONE
# /etc/hostname.em0 system-2
inet 10.10.26.4 255.255.255.0 NONE
Anyone can help? This issue is driving me crazy
:q!
This all generally looks ok and seems like you know what you're doing.
The usual thing which causes multi master is PF. Also rememer to *not*
sync your carp states over pfsync, this works for us;
pass out quick proto carp keep state (no-sync) set prio 7
pass quick proto carp from { fe80::/10 } to { ff00::/8 } keep state
(no-sync)
pass quick proto carp from { $all_carpv4_ips } keep state (no-sync)
pass quick on { $if_pfsync_dev } proto pfsync keep state (no-sync)
block drop quick proto carp
The fact it works fine until you fail-over could be a state thing..!?!
Though having sooo many CARP interfaces on only 8 VLANs seems wrong.
Good luck, Andy.
Following: dmesg from system-1 (system-2 is pretty the same, except
for interface macaddress).
TIA
OpenBSD 5.5 (GENERIC) #271: Wed Mar 5 09:31:16 MST 2014
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 17136455680 (16342MB)
avail mem = 16671719424 (15899MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xdf79c000 (63 entries)
bios0: vendor Dell Inc. version "2.0.2" date 10/22/2012
bios0: Dell Inc. PowerEdge R415
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET MCFG WD__ SLIC ERST HEST BERT
EINJ IV__ SRAT SLIT SSDT TCPA
acpi0: wakeup devices PCI0(S5) PEX2(S5) PEX3(S5) PEX4(S5) PEX9(S5)
PEXB(S5)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Opteron(tm) Processor 4238 , 3300.54 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,NXE,MMXX,FFXSR,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,XOP,SKINIT,WDT,FMA4,NODEID,TOPEXT,ITSC
cpu0: 64KB 64b/line 2-way I-cache, 16KB 64b/line 4-way D-cache, 2MB
64b/line 16-way L2 cache, 6MB 64b/line 64-way L3 cache
cpu0: ITLB 48 4KB entries fully associative, 24 4MB entries fully
associative
cpu0: DTLB 32 4KB entries fully associative, 32 4MB entries fully
associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 200MHz
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 22 pa 0xfec00000, version 21, 24 pins
ioapic0: misconfigured as apic 0, can't remap to apid 22
ioapic1 at mainbus0: apid 23 pa 0xfec80000, version 21, 32 pins
ioapic1: misconfigured as apic 0, can't remap to apid 23
acpihpet0 at acpi0: 14318180 Hz
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX2)
acpiprt2 at acpi0: bus -1 (PEX3)
acpiprt3 at acpi0: bus 2 (PEX4)
acpiprt4 at acpi0: bus 3 (PEX9)
acpiprt5 at acpi0: bus -1 (PEXB)
acpiprt6 at acpi0: bus 4 (COMP)
acpicpu0 at acpi0: C2, PSS
ipmi at mainbus0 not configured
cpu0: 3300 MHz: speeds: 3300 3000 2400 1800 1400 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "ATI SR5670 Host" rev 0x02
ppb0 at pci0 dev 2 function 0 "ATI SR5690 PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
90:e2:ba:2f:c0:48
em1 at pci1 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
90:e2:ba:2f:c0:49
ppb1 at pci0 dev 4 function 0 "ATI SR5690 PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
mpi0 at pci2 dev 0 function 0 "Symbios Logic SAS1068E" rev 0x08: msi
scsibus0 at mpi0: 112 targets
sd0 at scsibus0 targ 0 lun 0: <Dell, VIRTUAL DISK, 1028> SCSI3
0/direct fixed naa.600508e00000000015f76eff2353b10f
sd0: 571776MB, 512 bytes/sector, 1170997248 sectors
ses0 at scsibus0 targ 8 lun 0: <DP, BACKPLANE, 1.07> SCSI3
13/enclosure services fixed t10.DP_BACKPLANE000000
ppb2 at pci0 dev 9 function 0 "ATI SR5690 PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
bnx0 at pci3 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 23 int 24
bnx1 at pci3 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 23 int 25
ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x00: apic 22 int
22, AHCI 1.1
scsibus1 at ahci0: 32 targets
ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 22 int
16, version 1.0, legacy support
ohci1 at pci0 dev 18 function 1 "ATI SB700 USB" rev 0x00: apic 22 int
16, version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 22 int 17
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "ATI EHCI root hub" rev 2.00/1.00 addr 1
ohci2 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 22 int
18, version 1.0, legacy support
ohci3 at pci0 dev 19 function 1 "ATI SB700 USB" rev 0x00: apic 22 int
18, version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 22 int 19
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "ATI EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "ATI SBx00 SMBus" rev 0x3d: SMI
iic0 at piixpm0
pciide0 at pci0 dev 20 function 1 "ATI SB700 IDE" rev 0x00: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 22 int 16 for native-PCI interrupt
pcib0 at pci0 dev 20 function 3 "ATI SB700 ISA" rev 0x00
ppb3 at pci0 dev 20 function 4 "ATI SB600 PCI" rev 0x00
pci4 at ppb3 bus 4
vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pchb1 at pci0 dev 24 function 0 "AMD AMD64 15/0xh Link Cfg" rev 0x00
pchb2 at pci0 dev 24 function 1 "AMD AMD64 15/0xh Address Map" rev 0x00
pchb3 at pci0 dev 24 function 2 "AMD AMD64 15/0xh DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD AMD64 15/0xh Misc Cfg" rev 0x00
pchb4 at pci0 dev 24 function 4 "AMD AMD64 15/0xh CPU Power" rev 0x00
pchb5 at pci0 dev 24 function 5 "AMD AMD64 15/0xh Host" rev 0x00
pchb6 at pci0 dev 25 function 0 "AMD AMD64 15/0xh Link Cfg" rev 0x00
pchb7 at pci0 dev 25 function 1 "AMD AMD64 15/0xh Address Map" rev 0x00
pchb8 at pci0 dev 25 function 2 "AMD AMD64 15/0xh DRAM Cfg" rev 0x00
km1 at pci0 dev 25 function 3 "AMD AMD64 15/0xh Misc Cfg" rev 0x00
pchb9 at pci0 dev 25 function 4 "AMD AMD64 15/0xh CPU Power" rev 0x00
pchb10 at pci0 dev 25 function 5 "AMD AMD64 15/0xh Host" rev 0x00
usb2 at ohci0: USB revision 1.0
uhub2 at usb2 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb3 at ohci1: USB revision 1.0
uhub3 at usb3 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb4 at ohci2: USB revision 1.0
uhub4 at usb4 "ATI OHCI root hub" rev 1.00/1.00 addr 1
usb5 at ohci3: USB revision 1.0
uhub5 at usb5 "ATI OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub6 at uhub0 port 3 "Standard Microsystems product 0x2514" rev
2.00/0.00 addr 2
uhidev0 at uhub6 port 1 configuration 1 interface 0 "DELL Dell USB
Entry Keyboard" rev 1.10/1.78 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uplcom0 at uhub6 port 2 "Prolific Technology Inc. USB-Serial
Controller D" rev 1.10/4.00 addr 4
ucom0 at uplcom0
uhidev1 at uhub2 port 2 configuration 1 interface 0 "Avocent USB
Composite Device-0" rev 1.10/0.00 addr 2
uhidev1: iclass 3/1
ukbd1 at uhidev1: 8 variable keys, 6 key codes
wskbd2 at ukbd1 mux 1
wskbd2: connecting to wsdisplay0
uhidev2 at uhub2 port 2 configuration 1 interface 1 "Avocent USB
Composite Device-0" rev 1.10/0.00 addr 2
uhidev2: iclass 3/1
ums0 at uhidev2: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (ad29f48720345855.a) swap on sd0b dump on sd0b
bnx0: address 08:9e:01:90:2c:d3
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address 08:9e:01:90:2c:d4
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
