Hi all, So fun story for you all, and a good lesson as to why spending the time to set up IPv6 can save your ass in a pinch.
The players in this story are Me (and the company I consult with for when they have problems like this), Comcast (gig biz fiber), and CenturyLink (1/4th gig biz fiber). Now, some of you who have Comcast and/or CenturyLink/Lumen probably remember issues last year regarding IPSec traffic getting heavily fouled up at peering points somewhere. And, if you were like me, you probably remember that it was, well, lets be honest, impossible to get it looked into or dealt with (in reality). We resolved the issue ourselves between the three offices by switching to WireGuard which magically made the problems go away. Things have been going great until last week, when we noticed one of our WireGuard peers between CL/Lumen in Cheyenne and Comcast Denver was down. Packets from Den -> Cys were going through, but not Cys -> Den. Cys -> Boise on CL was still working perfectly fine and was acting as a backup connection to the Den office. I did my usual testing - changed ports, same behavior, changed IPs on the WireGuard endpoints on each end, same behavior. Even temp changed destination of the tunnel on Cys end to another off network node, and packets were going through, so we knew it had to be something relating going CL/Lumen -> Comcast. Weird thing was, I could dump iperf udp traffic over the same ports from same devices Cys -> Den, and the packets would go through perfectly fine... So.. sounds like there's some sort of throttling or IDS in the way somewhere toying with things. As expected, our first dealing with Comcast was less than spectacular where the tech tried to tell us that the live IPs they had assigned us, because they were a /27, they wouldn't work for VPN traffic (what?). I had to walk away from that call and let my partner finish it. We went to dinner, and as we were returning home and pulling into the driveway, I remembered we had 'wasted' (as some of you would put it) a bunch of time setting up IPv6 on the outward facing devices at each office... including the WireGuard boxes. I quickly reconfigured the Cys WireGuard node to connect to the Den node over IPv6 and, after WireGuard did its magic dynamically reconfiguring endpoints, suddenly the connection was back up and routing at full speed. Hell yeah! So, moral / TLDR of the story? Don't discount taking the time to set up IPv6, even if it's just for your important devices. Also, WireGuard > IPsec. -- Brie Sent from my iPad
