I have two firewalls running OpenBSD 5.1 with a 5.2 kernel amd64. I am running the 5.2 kernel because of another, unrelated bug. I have 4 ethernet interfaces (em0-4). em0 and em1 are in a failover trunk mode on trunk0 while em2 and em3 are members of trunk1 in failover mode. On trunk0, I have 3 VLANs (2,3,4) and on trunk1, I have 2 VLANs(10,11). I am running carp on each of these vlan interfaces. I am also running pfsync. I have an ipsec vpn configured which is using sasync between the two firewalls.
We had fw1 kernel panic and die yesterday. Everything seemed to switch over as expected to fw2. When we restarted fw1, all carp interfaces switched back to master on fw1 and *most* switched to backup on fw2. However, carp2 (carp for vlan2) stayed master on fw2. This was obviously an issue because it was also master on fw1. This caused lots of dropped packets since two machines are claiming the same IP address. I ifconfig carp2 down'd the carp interface and traffic was passing as it should again. However, as soon as I ifconfig carp2 up'd the carp interface, the carp2 interface on fw2 went to master mode again, and carp2 on fw1 stayed master as well. I have all carp interfaces on fw2 configured with an advskew of 128 and I have preempt enabled. I had to reboot fw2 for things to go back to normal with all interfaces on fw2 in backup mode while all on fw1 were in master mode. Below are my hostname.* config files as well as the carp sysctl values. Please let me know if anyone needs more information or if you have any suggestions on how to avoid this in the future. =========== FW1 ============== ** hostname.em0 ** up ** hostname.em1 ** up ** hostname.em2 ** up ** hostname.em3 ** up ** hostname.trunk0 ** up trunkproto failover trunkport em0 trunkport em1 ** hostname.trunk1 ** up trunkproto failover trunkport em2 trunkport em3 ** hostname.vlan10 ** up inet x.x.x.27 255.255.255.248 NONE vlan 10 vlandev trunk1 ** hostname.vlan11 ** up inet x.x.x.131 255.255.255.248 NONE vlan 11 vlandev trunk1 ** hostname.vlan2 ** up inet 172.16.20.2 255.255.255.0 NONE vlan 2 vlandev trunk0 ** hostname.vlan3 ** up inet x.x.x.210 255.255.255.240 NONE vlan 3 vlandev trunk0 ** hostname.vlan4 ** up inet x.x.x.98 255.255.255.224 NONE vlan 4 vlandev trunk0 ** hostname.carp10 ** up inet x.x.x.26 255.255.255.248 x.x.x.31 vhid 10 pass xxxxxxx carpdev vlan10 ** hostname.carp11 ** up inet x.x.x.130 255.255.255.248 x.x.x.135 vhid 11 pass xxxxxx carpdev vlan11 ** hostname.carp2 ** up inet 172.16.20.1 255.255.255.0 172.16.20.255 vhid 2 pass xxxxx carpdev vlan2 ** hostname.carp3 ** up inet x.x.x.209 255.255.255.240 x.x.x.223 vhid 3 pass xxxxx carpdev vlan3 ** hostname.carp4 ** up inet x.x.x.97 255.255.255.224 x.x.x.127 vhid 4 pass xxxxx carpdev vlan4 ** hostname.pfsync0 ** up syncdev vlan2 syncpeer 172.16.20.3 =========== FW2 ============ ** hostname.em0 ** up ** hostname.em1 ** up ** hostname.em2 ** up ** hostname.em3 ** up ** hostname.trunk0 ** up trunkproto failover trunkport em0 trunkport em1 ** hostname.trunk1 ** up trunkproto failover trunkport em2 trunkport em3 ** hostname.vlan10 ** up inet x.x.x.28 255.255.255.248 NONE vlan 10 vlandev trunk1 ** hostname.vlan11 ** up inet x.x.x.132 255.255.255.248 NONE vlan 11 vlandev trunk1 ** hostname.vlan2 ** up inet 172.16.20.3 255.255.255.0 NONE vlan 2 vlandev trunk0 ** hostname.vlan3 ** up inet x.x.x.213 255.255.255.240 NONE vlan 3 vlandev trunk0 ** hostname.vlan4 ** up inet x.x.x.99 255.255.255.224 NONE vlan 4 vlandev trunk0 ** hostname.carp10 ** up inet x.x.x 26 255.255.255.248 x.x.x 31 vhid 10 pass xxxx carpdev vlan10 advskew 128 ** hostname.carp11 ** up inet x.x.x 130 255.255.255.248 x.x.x 135 vhid 11 pass xxxx carpdev vlan11 advskew 128 ** hostname.carp2 ** up inet 172.16.20.1 255.255.255.0 172.16.20.255 vhid 2 pass xxxx carpdev vlan2 advskew 128 ** hostname.carp3 ** up inet x.x.x 209 255.255.255.240 x.x.x.223 vhid 3 carpdev vlan3 pass xxxx advskew 128 ** hostname.carp4 ** up inet x.x.x..97 255.255.255.224 x.x.x.127 vhid 4 pass xxxx carpdev vlan4 advskew 128 ** hostname.pfsync0 ** up syncdev vlan2 syncpeer 172.16.20.2
