Hello, I have EC2 instance with a custom FreeBSD kernel (9.2-RELEASE) with two interfaces – nic0 & nic1. Both have elastic IPs associated - E0 & E1.
I noticed that when I ping E1, the ICMP response packets go out of nic1 instead of nic0 & thus, from the ping source perspective, host is down. To workaround the asymmetric routing, I added a default route via nic1 to FIB 1 & added an IPFW rule to ensure FIB 1 is used for all packets received via nic1. Now, when nic1 tries to respond to an ICMP request, the first thing it does is to ARP for the router’s IP. The router responds with its MAC address but nic1 doesn’t seem to be processing it. The entry for default router’s IP in the ARP table remains marked ‘incomplete’ and never gets updated - ********************* # tcpdump -ni nic1 -p arp 07:29:04.922209 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28 07:29:04.922316 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42 # arp -a ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at (incomplete) on nic1 expired [ethernet] ip-172-31-30-133.us-west-2.compute.internal (172.31.30.133) at 02:6e:32:90:0e:ad on nic1 permanent [ethernet] ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic0 expires in 1199 seconds [ethernet] ip-172-31-22-113.us-west-2.compute.internal (172.31.22.113) at 02:ea:39:ce:5a:49 on nic0 permanent [ethernet] ********************* But… if I delete the arp entry (arp –d 172.31.16.1), ARP entry for the default router IP gets updated properly - ********************* # arp -a ip-172-31-16-1.us-west-2.compute.internal (172.31.16.1) at 02:18:c9:53:d0:2f on nic1 expires in 1195 seconds [ethernet] … ********************* But but… this only lasts until the entry expires. After that, the entry gets marked ‘incomplete’ again. I also notice these ARP requests start being received from the router around the same time I delete the entry (may the source address of these packets are being used to update the ARP entry?) - ********************* 07:32:41.411229 ARP, Request who-has 172.31.16.1 tell 172.31.30.133, length 28 07:32:41.411297 ARP, Reply 172.31.16.1 is-at 02:18:c9:53:d0:2f, length 42 07:33:04.482719 ARP, Request who-has 172.31.30.133 tell 172.31.16.1, length 42 <<<<<< 07:33:04.482734 ARP, Reply 172.31.30.133 is-at 02:6e:32:90:0e:ad, length 28 ********************* Any idea why the ARP replies are not being processed by nic1? Any known workarounds? Thanks, Rohit PS – FIB 1: Internet: Destination Gateway Flags Refs Use Netif Expire default 172.31.16.1 UGS 0 1985 nic1 127.0.0.1 link#2 UH 0 0 lo0 172.31.16.0/20 link#4 U 0 0 nic1 IFPW - 00002 setfib 1 ip from any to any via nic1 65535 allow ip from any to any _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
