W dniu 06.12.2016 o 14:40, Martin Pieuchot pisze:
On 06/12/16(Tue) 13:48, Rafał Błaszczyk wrote:
At first I would like to say hello and greet everyone as this is my first
post here.
I am having strange issues with one of the CARP interfaces.
I have two OpenBSD boxes (fw1, fw2) running as HA firewalls with CARP
interfaces in each VLAN.
Both boxes are running on two Linux KVM (Proxmox 4.2) hosts.
One of CARP interfaces stopped responding on ARP requests on CARP IP - it's
carp1
running on physical dev vio1 which is also running pfsync on top.
It's weird because the rest of carp interfaces behave correctly.
# ifconfig carp1
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:37
index 18 priority 15 llprio 3
carp: MASTER carpdev vio1 vhid 55 advbase 1 advskew 0
groups: carp
status: master
inet 10.24.5.1 netmask 0xffffff00 broadcast 10.24.5.255
I've checked arp table on two boxes and there is no entry for carp1.
That's the problem. We'll have to figure out where does it come from.
Could you share your routing table? Doing "# netstat -rnf inet"
You can find it below from fw1, I masked my public gw with G.G.G.G and
public IP with P.P.P.P,
masked first bytes of other MAC addresses with XXX
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default G.G.G.G UGS 364 1698856 - 8 pppoe0
224/4 127.0.0.1 URS 1 97662 32768 8 lo0
10.8/16 10.24.5.1 UGS 0 0 - 8 vio1
10.9.0/24 10.9.0.2 UGS 0 40756 - 8 tun0
10.9.0.1 10.9.0.1 UHl 0 0 - 1 tun0
10.9.0.2 10.9.0.1 UH 1 1 - 8 tun0
10.24.5/24 10.24.5.2 UC 5 24293 - 4 vio1
10.24.5/24 10.24.5.1 UC 0 0 - 4 carp1
10.24.5.2 52:54:00:84:51:e0 UHLl 0 8265 - 1 vio1
10.24.5.13 XXX:31:27 UHLc 0 24306 - 4 vio1
10.24.5.14 XXX:02:30 UHLc 1 24302 - 4 vio1
10.24.5.53 XXX:df:7c UHLc 0 24302 - 4 vio1
10.24.5.54 XXX:70:c1 UHLc 0 24303 - 4 vio1
10.24.5.201 XXX:9d:c1 UHLc 1 28783 - 4 vio1
10.24.5.255 10.24.5.2 UHb 0 2184 - 1 vio1
10.24.5.255 10.24.5.1 UHb 0 0 - 1 carp1
10.24.10/24 10.24.10.2 UC 8 197 - 4 vio2
10.24.10/24 10.24.10.1 UC 0 0 - 4 carp2
10.24.10.1 00:00:5e:00:01:02 UHLl 0 4093 - 1 carp2
10.24.10.2 52:54:00:a7:c6:bd UHLl 0 14664 - 1 vio2
10.24.10.11 XXX:f4:82 UHLc 0 210 - 4 vio2
10.24.10.15 XXX:36:37 UHLc 0 3979 - 4 vio2
10.24.10.16 XXX:37:37 UHLc 0 6644 - 4 vio2
10.24.10.23 XXX:61:33 UHLc 0 413 - 4 vio2
10.24.10.24 XXX:30:38 UHLc 0 3252 - 4 vio2
10.24.10.37 link#3 UHRLc 0 245 - 4 vio2
10.24.10.38 XXX:61:34 UHLc 0 4475 - 4 vio2
10.24.10.51 XXX:b7:fb UHLc 0 698374 - 4 vio2
10.24.10.255 10.24.10.2 UHb 0 544 - 1 vio2
10.24.10.255 10.24.10.1 UHb 0 0 - 1 carp2
10.24.20/24 10.24.20.2 UC 3 8327 - 4 vio3
10.24.20/24 10.24.20.1 UC 0 0 - 4 carp3
10.24.20.1 00:00:5e:00:01:03 UHLl 0 1374 - 1 carp3
10.24.20.2 52:54:00:e0:03:95 UHLl 0 37679 - 1 vio3
10.24.20.11 XXX:a3:f9 UHLc 0 19191 - 4 vio3
10.24.20.212 XXX:ee:99 UHLc 0 8362 - 4 vio3
10.24.20.214 XXX:b4:d0 UHLc 0 15250 - 4 vio3
10.24.20.255 10.24.20.2 UHb 0 0 - 1 vio3
10.24.20.255 10.24.20.1 UHb 0 0 - 1 carp3
10.24.21/24 10.24.21.2 UC 2 174 - 4 vio4
10.24.21/24 10.24.21.1 UC 0 0 - 4 carp4
10.24.21.1 00:00:5e:00:01:04 UHLl 0 368 - 1 carp4
10.24.21.2 52:54:00:62:92:1e UHLl 0 74 - 1 vio4
10.24.21.12 XXX:88:e2 UHLc 1 4267 - 4 vio4
10.24.21.16 XXX:12:88 UHLc 0 536 - 4 vio4
10.24.21.255 10.24.21.2 UHb 0 0 - 1 vio4
10.24.21.255 10.24.21.1 UHb 0 0 - 1 carp4
10.24.22/24 10.24.22.2 UC 4 4560 - 4 vio5
10.24.22/24 10.24.22.1 UC 0 0 - 4 carp5
10.24.22.1 00:00:5e:00:01:05 UHLl 0 2755 - 1 carp5
10.24.22.2 52:54:00:ad:e0:a2 UHLl 0 2305 - 1 vio5
10.24.22.5 XXX:30:65 UHLc 0 117738 - 4 vio5
10.24.22.13 XXX:c3:97 UHLc 1 4745 - 4 vio5
10.24.22.14 XXX:bf:73 UHLc 0 4744 - 4 vio5
10.24.22.15 XXX:c6:cd UHLc 0 4754 - 4 vio5
10.24.22.255 10.24.22.2 UHb 0 0 - 1 vio5
10.24.22.255 10.24.22.1 UHb 0 0 - 1 carp5
10.24.23/24 10.24.23.2 UC 6 1141860 - 4 vio6
10.24.23/24 10.24.23.1 UC 0 0 - 4 carp6
10.24.23.1 00:00:5e:00:01:06 UHLl 0 13706 - 1 carp6
10.24.23.2 52:54:00:ee:7f:20 UHLl 0 37284 - 1 vio6
10.24.23.5 XXX:36:33 UHLc 0 1119653 - 4 vio6
10.24.23.11 XXX:49:a3 UHLc 0 1469582 - 4 vio6
10.24.23.14 XXX:51:2f UHLc 1 4415 - 4 vio6
10.24.23.16 XXX:b7:05 UHLc 0 9539 - 4 vio6
10.24.23.17 XXX:69:15 UHLc 0 8786 - 4 vio6
10.24.23.18 XXX:5c:d8 UHLc 0 1786 - 4 vio6
10.24.23.255 10.24.23.2 UHb 0 0 - 1 vio6
10.24.23.255 10.24.23.1 UHb 0 0 - 1 carp6
10.24.24/24 10.24.24.2 UC 2 474 - 4 vio7
10.24.24/24 10.24.24.1 UC 0 0 - 4 carp7
10.24.24.1 00:00:5e:00:01:07 UHLl 0 11479 - 1 carp7
10.24.24.2 52:54:00:7c:8a:34 UHLl 0 612 - 1 vio7
10.24.24.11 XXX:f1:b8 UHLc 0 6164 - 4 vio7
10.24.24.12 XXX:e5:b7 UHLc 0 2045 - 4 vio7
10.24.24.255 10.24.24.2 UHb 0 0 - 1 vio7
10.24.24.255 10.24.24.1 UHb 0 0 - 1 carp7
10.24.30/24 10.24.30.2 UC 0 13 - 4 vio8
10.24.30/24 10.24.30.1 UC 0 0 - 4 carp8
10.24.30.1 00:00:5e:00:01:08 UHLl 0 6 - 1 carp8
10.24.30.2 52:54:00:0d:ce:cc UHLl 0 0 - 1 vio8
10.24.30.255 10.24.30.2 UHb 0 0 - 1 vio8
10.24.30.255 10.24.30.1 UHb 0 0 - 1 carp8
10.24.51/24 10.24.51.2 UC 1 3329 - 4 vio9
10.24.51/24 10.24.51.1 UC 0 0 - 4 carp9
10.24.51.1 00:00:5e:00:01:09 UHLl 0 11 - 1 carp9
10.24.51.2 52:54:00:9f:64:aa UHLl 0 19497 - 1 vio9
10.24.51.22 XXX:f9:5d UHLc 1 997353 - 4 vio9
10.24.51.255 10.24.51.2 UHb 0 0 - 1 vio9
10.24.51.255 10.24.51.1 UHb 0 0 - 1 carp9
10.24.52/24 10.24.52.2 UC 1 4632 - 4 vio10
10.24.52/24 10.24.52.1 UC 0 0 - 4 carp10
10.24.52.1 00:00:5e:00:01:0a UHLl 0 25 - 1 carp10
10.24.52.2 52:54:00:6f:75:78 UHLl 0 4133 - 1 vio10
10.24.52.12 XXX:cf:65 UHLc 1 84789 - 4 vio10
10.24.52.255 10.24.52.2 UHb 0 0 - 1 vio10
10.24.52.255 10.24.52.1 UHb 0 0 - 1 carp10
10.24.53/24 10.24.53.2 UC 0 13 - 4 vio11
10.24.53/24 10.24.53.1 UC 0 0 - 4 carp11
10.24.53.1 00:00:5e:00:01:0b UHLl 0 6 - 1 carp11
10.24.53.2 52:54:00:15:50:ce UHLl 0 0 - 1 vio11
10.24.53.255 10.24.53.2 UHb 0 0 - 1 vio11
10.24.53.255 10.24.53.1 UHb 0 0 - 1 carp11
10.24.54/24 10.24.54.2 UC 1 5730 - 4 vio12
10.24.54/24 10.24.54.1 UC 0 0 - 4 carp12
10.24.54.1 00:00:5e:00:01:0c UHLl 0 99 - 1 carp12
10.24.54.2 52:54:00:bd:d2:78 UHLl 0 4423 - 1 vio12
10.24.54.11 XXX:ba:a8 UHLc 1 105197 - 4 vio12
10.24.54.255 10.24.54.2 UHb 0 0 - 1 vio12
10.24.54.255 10.24.54.1 UHb 0 0 - 1 carp12
10.24.55/24 10.24.55.2 UC 0 13 - 4 vio13
10.24.55/24 10.24.55.1 UC 0 0 - 4 carp13
10.24.55.1 00:00:5e:00:01:0d UHLl 0 6 - 1 carp13
10.24.55.2 52:54:00:36:d4:4f UHLl 0 0 - 1 vio13
10.24.55.255 10.24.55.2 UHb 0 0 - 1 vio13
10.24.55.255 10.24.55.1 UHb 0 0 - 1 carp13
P.P.P.P P.P.P.P UHl 0 13355 - 1 pppoe0
G.G.G.G P.P.P.P UH 1 1 - 8 pppoe0
127/8 127.0.0.1 UGRS 0 0 32768 8 lo0
127.0.0.1 127.0.0.1 UHl 2 29 32768 1 lo0
192.168.188/24 192.168.188.27 UC 0 9157 - 4 vio0
192.168.188/24 192.168.188.30 UC 0 0 - 4 carp0
192.168.188.27 52:54:00:5e:75:94 UHLl 0 0 - 1 vio0
192.168.188.30 00:00:5e:00:01:1e UHLl 0 6 - 1 carp0
192.168.188.255 192.168.188.27 UHb 0 0 - 1 vio0
192.168.188.255 192.168.188.30 UHb 0 0 - 1 carp0
How does your /etc/hostname.carp1 look like?
passwords masked (it's the same unique password on both nodes):
fw1:
inet 10.24.5.1 255.255.255.0 10.24.5.255 vhid 55 carpdev vio1 pass <pass>
fw2:
inet 10.24.5.1 255.255.255.0 10.24.5.255 vhid 55 carpdev vio1 pass
<pass> advskew 128
I've changed vhid before to 55 (was 1) to check if it's not a problem
with the switch arp table.
The funny thing is that the arp entry for carp1 now appeared on fw1 (not
on fw2)
10.24.5.1 00:00:5e:00:01:37 carp1 permanent l
The only thing I've changed was ifconfig vio1 down; ifconfig vio1 up;
running # sh /etc/netstart carp1
and updating hypervisor host which fw2 is running on which involved
rebooting a couple machines VMs.
I wonder if I don't have any MAC or IP address collision... I need to
check that also.
Do you see an error when running "# sh /etc/netstart carp1" ?
exit status 0, no errors
BUT... I believe that did the trick _carp1 appeared as arp entry on fw1
because I ran /etc/netstart as you said_.
I did the same on fw2 and it also appeared on fw2 so I believe this is
the case and it's doing something that
it didn't do at boot time.
If you grep for 'arp' in /var/log/messages do you get anything?
On both nodes I got:
/bsd: carp3: ip_output failed: 13 # (only on carp3)
but:
# pfctl -s rules | grep carp
pass quick proto carp all
One more thing that is interesting is I got "carp1 demoted group
carp..." message while transiting states,
Dec 6 15:15:29 fw1 /bsd: carp1: state transition: MASTER -> INIT
Dec 6 15:15:29 fw1 /bsd: carp: carp1 demoted group carp by 1 to 1 (carpdev)
Dec 6 15:15:29 fw1 /bsd: carp0: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp5: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp7: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp13: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp12: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp11: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp10: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp9: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp8: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp6: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp4: state transition: MASTER -> BACKUP
Dec 6 15:15:29 fw1 /bsd: carp2: state transition: MASTER -> BACKUP
Dec 6 15:15:31 fw1 /bsd: carp1: state transition: INIT -> BACKUP
Dec 6 15:15:31 fw1 /bsd: carp: carp1 demoted group carp by -1 to 0
(carpdev)
Dec 6 15:15:32 fw1 /bsd: carp1: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp0: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp5: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp2: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp4: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp6: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp7: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp8: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp9: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp10: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp11: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp12: state transition: BACKUP -> MASTER
Dec 6 15:15:32 fw1 /bsd: carp13: state transition: BACKUP -> MASTER
Dec 6 15:27:21 fw1 /bsd: carp1: state transition: MASTER -> INIT
Dec 6 15:27:21 fw1 /bsd: carp1: state transition: INIT -> BACKUP
Dec 6 15:27:24 fw1 /bsd: carp1: state transition: BACKUP -> MASTER
Dec 6 15:27:30 fw1 /bsd: carp1: state transition: MASTER -> INIT
Dec 6 15:27:30 fw1 /bsd: carp1: state transition: INIT -> BACKUP
Dec 6 15:27:33 fw1 /bsd: carp1: state transition: BACKUP -> MASTER
I don't know why the flapping at 15:27 occurred. Possibly i was doing
ifconfig up/down or netstart but I am not sure.
I need to do it step by step to see some differences.
