I've been having a good time running some VMD guests on 6.2 and assigning them external IPs which are binat'd to them by the VM host. Recently I learned my hosting provider delegates a /64 to it's dedicated boxes and thought this might be an interesting scenario to improve, and possibly simplify, by routing IPv6 directly to my guests.
To start, I ensured that IPv6 was properly functional on the host: > # cat /etc/hostname.em0 > inet6 autoconf > inet6 2607:53xx:6x:7a3b:: 64 eui64 > inet xxx.xxx.211.59 255.255.255.0 > inet alias xxx.xxx.219.108 255.255.255.0 > inet alias xxx.xx.248.240 255.255.255.0 > inet alias xxx.xx.248.241 255.255.255.0 > inet alias xxx.xx.248.242 255.255.255.0 > inet alias xxx.xx.248.243 255.255.255.0 > # cat /etc/mygate > xxx.xxx.211.254 > fe80::205:73ff:fea0:1%em0 > # ifconfig em0 > em0: flags=208843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500 > lladdr 0c:c4:7a:45:37:34 > index 1 priority 0 llprio 3 > groups: egress > media: Ethernet autoselect (1000baseT full-duplex,master,rxpause) > status: active > inet6 fe80::ec4:7aff:fe45:3734%em0 prefixlen 64 scopeid 0x1 > inet6 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 prefixlen 64 > inet xxx.xxx.211.59 netmask 0xffffff00 broadcast xxx.xxx.211.255 > inet xxx.xxx.219.108 netmask 0xffffff00 broadcast xxx.xxx.219.255 > inet xxx.xx.248.240 netmask 0xffffff00 broadcast xxx.xx.248.255 > inet xxx.xx.248.241 netmask 0xffffff00 broadcast xxx.xx.248.255 > inet xxx.xx.248.242 netmask 0xffffff00 broadcast xxx.xx.248.255 > inet xxx.xx.248.243 netmask 0xffffff00 broadcast xxx.xx.248.255 > # ifconfig vether0 > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:00:d3:00:d0:0d > index 5 priority 0 llprio 3 > groups: vether > media: Ethernet autoselect > status: active > inet6 fe80::200:d3ff:fe00:d00d%vether0 prefixlen 64 scopeid 0x5 > inet 10.0.23.1 netmask 0xffffff00 broadcast 10.0.23.255 > # ifconfig bridge0 > bridge0: flags=41<UP,RUNNING> > description: switch1-local > index 6 llprio 3 > groups: bridge > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp > designated: id 00:00:00:00:00:00 priority 0 > vether0 flags=3<LEARNING,DISCOVER> > port 5 ifpriority 0 ifcost 0 > tap0 flags=3<LEARNING,DISCOVER> > port 8 ifpriority 0 ifcost 0 > Addresses (max cache: 100, timeout: 240): > 00:00:d3:00:12:00 tap0 1 flags=0<> > # slaacctl show interface em0 > em0: > index: 1 running: yes privacy: yes > lladdr: 0c:c4:7a:45:37:34 > inet6: fe80::ec4:7aff:fe45:3734%em0 > Router Advertisement from fe80::205:73ff:fea0:1%em0 > received: 2018-03-17 20:01:39; 143s ago > Cur Hop Limit: 64, M: 0, O: 0, Router Lifetime: 1800s > Default Router Preference: Medium > Reachable Time: 0ms, Retrans Timer: 0ms > prefix: 2607:53xx:6x:7aff:ff:ff:ff:ff/56 > On-link: 1, Autonomous address-configuration: 1 > vltime: 2592000, pltime: 604800 > Default router proposals > id: 1, state: CONFIGURED > router: fe80::205:73ff:fea0:1%em0 > router lifetime: 1800 > Preference: Medium > updated: 2018-03-17 20:01:39; 143s ago, timeout: 1642s > # route -nv show -inet6 > Routing tables > > Internet6: > Destination Gateway Flags > Refs Use Mtu Prio Iface Label > default fe80::205:73ff:fea0:1%em0 UGS > 0 4 - 56 em0 "slaacd" > ::/96 ::1 UGRS > 0 0 32768 8 lo0 > ::/104 ::1 UGRS > 0 0 32768 8 lo0 > ::1 ::1 UHhl > 14 28 32768 1 lo0 > ::127.0.0.0/104 ::1 UGRS > 0 0 32768 8 lo0 > ::224.0.0.0/100 ::1 UGRS > 0 0 32768 8 lo0 > ::255.0.0.0/104 ::1 UGRS > 0 0 32768 8 lo0 > ::ffff:0.0.0.0/96 ::1 UGRS > 0 0 32768 8 lo0 > 2002::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2002:7f00::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2002:e000::/20 ::1 UGRS > 0 0 32768 8 lo0 > 2002:ff00::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2607:53xx:6x:7a3b::/64 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 UCn > 0 0 - 4 em0 > 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 0c:c4:7a:45:37:34 UHLl > 0 173 - 1 em0 > fe80::/10 ::1 UGRS > 0 2 32768 8 lo0 > fec0::/10 ::1 UGRS > 0 0 32768 8 lo0 > fe80::%em0/64 fe80::ec4:7aff:fe45:3734%em0 UCn > 3 5 - 4 em0 > fe80::205:73ff:fea0:1%em0 00:05:73:a0:00:01 UHLch > 1 95 - 3 em0 > fe80::2ff:ffff:feff:fffd%em0 00:ff:ff:ff:ff:fd UHLc > 0 1155 - 3 em0 > fe80::2ff:ffff:feff:fffe%em0 00:ff:ff:ff:ff:fe UHLc > 0 945 - 3 em0 > fe80::ec4:7aff:fe45:3734%em0 0c:c4:7a:45:37:34 UHLl > 0 373 - 1 em0 > fe80::1%lo0 fe80::1%lo0 UHl > 0 0 32768 1 lo0 > fe80::%vether0/64 fe80::200:d3ff:fe00:d00d%vether0 UCn > 1 1 - 4 vether0 > fe80::200:d3ff:fe00:1200%vether0 00:00:d3:00:12:00 UHLc > 1 11 - 3 vether0 > fe80::200:d3ff:fe00:d00d%vether0 00:00:d3:00:d0:0d UHLl > 0 38 - 1 vether0 > ff01::/16 ::1 UGRS > 0 2 32768 8 lo0 > ff01::%em0/32 fe80::ec4:7aff:fe45:3734%em0 Um > 0 1 - 4 em0 > ff01::%lo0/32 ::1 Um > 0 1 32768 4 lo0 > ff01::%vether0/32 fe80::200:d3ff:fe00:d00d%vether0 Um > 0 0 - 4 vether0 > ff02::/16 ::1 UGRS > 0 2 32768 8 lo0 > ff02::%em0/32 fe80::ec4:7aff:fe45:3734%em0 Um > 0 1 - 4 em0 > ff02::%lo0/32 ::1 Um > 0 1 32768 4 lo0 > ff02::%vether0/32 fe80::200:d3ff:fe00:d00d%vether0 Um > 0 0 - 4 vether0 > # ping6 -c3 google.com > PING google.com (2607:f8b0:4004:809::200e): 56 data bytes > 64 bytes from 2607:f8b0:4004:809::200e: icmp_seq=0 hlim=56 time=13.756 ms > 64 bytes from 2607:f8b0:4004:809::200e: icmp_seq=1 hlim=56 time=13.748 ms > 64 bytes from 2607:f8b0:4004:809::200e: icmp_seq=2 hlim=56 time=13.744 ms > > --- google.com ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 13.744/13.749/13.756/0.005 ms > # ftp -6 -Mo - https://wtfismyip.com/text > Trying 2607:53xx:6x:7f8a::... > Requesting https://wtfismyip.com/text > 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 > 37 bytes received in 0.00 seconds (145.11 KB/s) Next up, IPv6 forwarding on the host and interfaces for the guests: > # cat /etc/sysctl.conf > net.inet.ip.forwarding=1 > net.inet6.ip6.forwarding=1 > # cat /etc/hostname.vether0 > lladdr 00:00:D3:00:D0:0D > inet6 eui64 > inet 10.0.23.1 255.255.255.0 NONE > # cat /etc/hostname.bridge0 > add vether0 > up > # egrep 'rtadvd|vmd' /etc/rc.conf.local > rtadvd_flags=vether0 > vmd_flags= > # stat /etc/rtadvd.conf > stat: /etc/rtadvd.conf: No such file or directory > # cat /etc/vm.conf > switch "local" { > interface bridge0 > add vether0 > } > vm "chat" { > disable > memory 1G > owner maxp > disk "/home/images/chat.img" > interface tap { > locked lladdr "00:00:D3:00:12:00" > switch "local" > } > } > # pfctl -d > pfctl: pf not enabled I've got PF disabled for the moment, just to keep things simple. Then, within the VM... > # cat /etc/hostname.vio0 > dhcp > inet6 autoconf > inet6 alias 2607:53xx:6x:7a3b:: 64 eui64 > # slaacctl show interface vio0 > vio0: > index: 1 running: yes privacy: yes > lladdr: 00:00:d3:00:12:00 > inet6: fe80::200:d3ff:fe00:1200%vio0 > Router Advertisement from fe80::200:d3ff:fe00:d00d%vio0 > received: 2018-03-17 22:05:48; 83s ago > Cur Hop Limit: 64, M: 0, O: 0, Router Lifetime: 1800s > Default Router Preference: Medium > Reachable Time: 0ms, Retrans Timer: 0ms > Default router proposals > id: 1, state: CONFIGURED > router: fe80::200:d3ff:fe00:d00d%vio0 > router lifetime: 1800 > Preference: Medium > updated: 2018-03-17 22:05:48; 83s ago, timeout: 1702s > # ifconfig vio0 > vio0: > flags=208b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,AUTOCONF6> > mtu 1500 > lladdr 00:00:d3:00:12:00 > index 1 priority 0 llprio 3 > groups: egress > media: Ethernet autoselect > status: active > inet 10.0.23.30 netmask 0xffffff00 broadcast 10.0.23.255 > inet6 fe80::200:d3ff:fe00:1200%vio0 prefixlen 64 scopeid 0x1 > inet6 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 prefixlen 64 > # route -nv show -inet6 > Routing tables > > Internet6: > Destination Gateway Flags > Refs Use Mtu Prio Iface Label > default fe80::200:d3ff:fe00:d00d%vio0 UGS > 0 0 - 56 vio0 "slaacd" > ::/96 ::1 UGRS > 0 0 32768 8 lo0 > ::/104 ::1 UGRS > 0 0 32768 8 lo0 > ::1 ::1 UHhl > 14 28 32768 1 lo0 > ::127.0.0.0/104 ::1 UGRS > 0 0 32768 8 lo0 > ::224.0.0.0/100 ::1 UGRS > 0 0 32768 8 lo0 > ::255.0.0.0/104 ::1 UGRS > 0 0 32768 8 lo0 > ::ffff:0.0.0.0/96 ::1 UGRS > 0 0 32768 8 lo0 > 2002::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2002:7f00::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2002:e000::/20 ::1 UGRS > 0 0 32768 8 lo0 > 2002:ff00::/24 ::1 UGRS > 0 0 32768 8 lo0 > 2607:53xx:6x:7a3b::/64 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 UCn > 0 0 - 4 vio0 > 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 00:00:d3:00:12:00 UHLl > 0 0 - 1 vio0 > fe80::/10 ::1 UGRS > 0 1 32768 8 lo0 > fec0::/10 ::1 UGRS > 0 0 32768 8 lo0 > fe80::%vio0/64 fe80::200:d3ff:fe00:1200%vio0 UCn > 1 1 - 4 vio0 > fe80::200:d3ff:fe00:1200%vio0 00:00:d3:00:12:00 UHLl > 0 3 - 1 vio0 > fe80::200:d3ff:fe00:d00d%vio0 00:00:d3:00:d0:0d UHLch > 1 43 - 3 vio0 > fe80::1%lo0 fe80::1%lo0 UHl > 0 0 32768 1 lo0 > ff01::/16 ::1 UGRS > 0 1 32768 8 lo0 > ff01::%vio0/32 fe80::200:d3ff:fe00:1200%vio0 Um > 0 1 - 4 vio0 > ff01::%lo0/32 ::1 Um > 0 1 32768 4 lo0 > ff02::/16 ::1 UGRS > 0 1 32768 8 lo0 > ff02::%vio0/32 fe80::200:d3ff:fe00:1200%vio0 Um > 0 1 - 4 vio0 > ff02::%lo0/32 ::1 Um > 0 1 32768 4 lo0 > # ifconfig vio0 > vio0: > flags=208b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,AUTOCONF6> > mtu 1500 > lladdr 00:00:d3:00:12:00 > index 1 priority 0 llprio 3 > groups: egress > media: Ethernet autoselect > status: active > inet 10.0.23.30 netmask 0xffffff00 broadcast 10.0.23.255 > inet6 fe80::200:d3ff:fe00:1200%vio0 prefixlen 64 scopeid 0x1 > inet6 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 prefixlen 64 > # ping6 fe80::200:d3ff:fe00:d00d%vio0 > PING fe80::200:d3ff:fe00:d00d%vio0 (fe80::200:d3ff:fe00:d00d%vio0): 56 data > bytes > 64 bytes from fe80::200:d3ff:fe00:d00d%vio0: icmp_seq=0 hlim=64 time=0.344 ms > 64 bytes from fe80::200:d3ff:fe00:d00d%vio0: icmp_seq=1 hlim=64 time=0.208 ms > 64 bytes from fe80::200:d3ff:fe00:d00d%vio0: icmp_seq=2 hlim=64 time=0.207 ms > ^C > --- fe80::200:d3ff:fe00:d00d%vio0 ping statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 0.207/0.253/0.344/0.064 ms > # ping6 2607:f8b0:400a:800::200e > PING 2607:f8b0:400a:800::200e (2607:f8b0:400a:800::200e): 56 data bytes > ^C > --- 2607:f8b0:400a:800::200e ping statistics --- > 3 packets transmitted, 0 packets received, 100.0% packet loss > # ndp -an > Neighbor Linklayer Address Netif Expire S > Flags > 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 00:00:d3:00:12:00 vio0 permanent R l > fe80::200:d3ff:fe00:1200%vio0 00:00:d3:00:12:00 vio0 permanent R l > fe80::200:d3ff:fe00:d00d%vio0 00:00:d3:00:d0:0d vio0 23h59m57s S R Back on the host: > # ping6 2607:53xx:6x:7a3b:200:d3ff:fe00:1200%vether0 > ping6: no address associated with name > # ping6 fe80::200:d3ff:fe00:1200%vether0 > PING fe80::200:d3ff:fe00:1200%vether0 (fe80::200:d3ff:fe00:1200%vether0): 56 > data bytes > 64 bytes from fe80::200:d3ff:fe00:1200%vether0: icmp_seq=0 hlim=64 time=0.265 > ms > 64 bytes from fe80::200:d3ff:fe00:1200%vether0: icmp_seq=1 hlim=64 time=0.237 > ms > 64 bytes from fe80::200:d3ff:fe00:1200%vether0: icmp_seq=2 hlim=64 time=0.218 > ms > 64 bytes from fe80::200:d3ff:fe00:1200%vether0: icmp_seq=3 hlim=64 time=0.220 > ms > ^C > --- fe80::200:d3ff:fe00:1200%vether0 ping statistics --- > 4 packets transmitted, 4 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 0.218/0.235/0.265/0.019 ms > # ndp -a > Neighbor Linklayer Address Netif Expire S > Flags > 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 0c:c4:7a:45:37:34 em0 permanent R l > fe80::205:73ff:fea0:1%em0 00:05:73:a0:00:01 em0 18s R R > fe80::2ff:ffff:feff:fffd%em0 00:ff:ff:ff:ff:fd em0 23h59m31s S R > fe80::2ff:ffff:feff:fffe%em0 00:ff:ff:ff:ff:fe em0 23h58m49s S R > fe80::ec4:7aff:fe45:3734%em0 0c:c4:7a:45:37:34 em0 permanent R l > fe80::200:d3ff:fe00:1200%vether0 00:00:d3:00:12:00 vether0 23h59m34s S > fe80::200:d3ff:fe00:d00d%vether0 00:00:d3:00:d0:0d vether0 permanent R lR Watching on em0 on the host I can see the v6 traffic from the guest leaving and I can see NDP traffic looking for who has 1200, but the reply never flows out: > 1521332053.797396 00:ff:ff:ff:ff:fd 0c:c4:7a:45:37:34 86dd 86: > fe80::2ff:ffff:feff:fffd > 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734: icmp6: > neighbor sol: who has 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 [class 0xe0] > 1521332053.797413 0c:c4:7a:45:37:34 00:ff:ff:ff:ff:fd 86dd 78: > fe80::ec4:7aff:fe45:3734 > fe80::2ff:ffff:feff:fffd: icmp6: neighbor adv: tgt > is 2607:53xx:6x:7a3b:ec4:7aff:fe45:3734 > 1521332054.211392 00:ff:ff:ff:ff:fd 33:33:ff:00:12:00 86dd 86: > fe80::2ff:ffff:feff:fffd > ff02::1:ff00:1200: icmp6: neighbor sol: who has > 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 [class 0xe0] > 1521332055.055953 00:05:73:a0:00:01 33:33:00:00:00:66 86dd 114: > fe80::2ff:ffff:feff:fffd.2029 > ff02::66.2029: udp 52 [class 0xc0] > 1521332055.393267 0c:c4:7a:45:37:34 00:05:73:a0:00:01 86dd 118: > 2607:53xx:6x:7a3b:200:d3ff:fe00:1200 > 2607:f8b0:400a:800::200e: icmp6: echo > request I also don't see the NDP traffic on vio0 on the guest. I'm probably missing something elementary, being a bit new to IPv6 routers. Any insights would be hugely helpful. -- 0x7D964D3361142ACF
