On July 5, 2023 4:35:30 AM GMT+03:00, Anthony Coulter
<b...@anthonycoulter.name> wrote:
>Short version:
>
>I'm trying to set up a "road warrior"-style VPN like the one described
>at https://www.openbsd.org/faq/faq17.html but I'm trying to use IPv6 so
>I can have globally-routable addresses (so I'm not using NAT). So far
>I've gotten the initiator and the responder to set up a security
>association (to be precise, according to "ipsecctl -sa" the
>initiator/client sees one SA and zero flows, and the responder/server
>sees two SA's and two flows). When I try to ping the client's virtual
>IP address from the server, tcpdump running on the egress interfaces of
>both the client and the server see appropriately-sized "esp" packets
>going from the server to the client as expected, but tcpdump doesn't
>see any traffic emerging on the "lo1" loopback interface the FAQ told
>me to create, nor does it see any evidence of ping-responses being
>sent. When I try to ping from the client to the server, I get
>"ping6: sendmsg: Permission denied".
>
>
>Longer version:
>
>The client is running -current and the server is running 7.3.
>
>The server is a VPS hosted in a datacenter somewhere, and my VPS
>provider gives me a whole /64 subnet to work with. My VPS provider
>doesn't appear to look at neighbor discovery traffic; it routes all
>traffic in that /64 to my VirtIO interface, regardless of which IP
>address it is, which seems like a handy feature for what I'm trying to
>do.
>
>Here is what I want to do: the client should open an IPsec tunnel to
>the server. The client should request an IPv6 address from the server's
>enormous /64 subnet. Then, applications running on the client should
>have the option to use that "virtual" IPv6 address as though it was
>available on a local interface on the client machine. The client would
>effectively be multihosted: in addition to its regular physical
>Internet connection it would also have this IPsec tunnel that also
>connects to the global Internet. My plan right now is to use rdomains
>to control which applications use the tunnel and which ones use the
>regular egress interface, but I'm not attached to that specific
>approach.
>
>The application should be obvious: I'm trying to set up a region-faking
>VPN, so that I can have applications appear to connect to the Internet
>from different continents.
>
>Both client and server are using the same pf.conf that is installed by
>default. I tried adding "pass on enc0" to the client's pf.conf and
>"pass on enc0 tagged ROADW" to the server's, but it hasn't made a
>difference because (I assume) my existing pf.conf rules weren't
>blocking much to begin with.
>
>Here are my config files, with IP addresses and server names changed.
>But basically you should see the client at 2001:db8:1::1 connecting to
>the server at 2001:db8:2::1 and requesting a random IP address in the
>server's subnet, e.g. 2001:db8:2::485b:4ac7. I have also put the
>appropriate keys in /etc/iked/pubkeys/fqdn/.
>
>=== Client-side configuration ===
>$ cat /etc/hostname.lo1
>rdomain 1
>
>$ cat /etc/iked.conf
>ikev2 'client_config' active esp \
> rdomain 1 \
> from dynamic to any \
> local 2001:db8:1::1 \
> peer 2001:db8:2::1 \
> srcid client.example.org \
> dstid server.example.org \
> request address any \
> iface lo1
>
>=== Server-side configuration ===
>
>$ cat /etc/sysctl.conf
>net.inet6.ip6.forwarding=1
>
>$ cat /etc/iked.conf
>ikev2 'server_config' passive esp \
> from any to dynamic \
> local 2001:db8:2::1 \
> peer 2001:db8:1::1 \
> srcid server.example.org \
> config address 2001:db8:2::/64 \
> tag "ROADW"
>
>=== End of configurations ===
>
>Does it work? Sort of. Here's what I see on the server:
>
>=== Checking status on the server ===
>
># ipsecctl -sa
>FLOWS:
>flow esp in from 2001:db8:2::485b:4ac7 to ::/0 peer 2001:db8:1::1 srcid
>FQDN/server.example.org dstid FQDN/client.example.org type require
>flow esp out from ::/0 to 2001:db8:2::485b:4ac7 peer 2001:db8:1::1 srcid
>FQDN/server.example.org dstid FQDN/client.example.org type require
>
>SAD:
>esp tunnel from 2001:db8:2::1 to 2001:db8:1::1 spi 0x001a3754 enc aes-128-gcm
>esp tunnel from 2001:db8:1::1 to 2001:cb8:2::1 spi 0x7634a3b6 enc aes-128-gcm
>
>$ route show
>Internet6:
>Destination Gateway Flags Refs Use Mtu Prio Iface
>default fe80::1234%vio0 UGS 8 578 -
>8 vio0
>::/96 localhost UGRS 0 0 32768 8 lo0
>localhost localhost UHhl 10 90 32768 1 lo0
>::ffff:0.0.0.0/96 localhost UGRS 0 0 32768 8 lo0
>2001:db8:2::/64 server UCn 3 3630 -
>4 vio0
>2001:db8:2::485b:4ac7 link#1 UHLc 0 3633 - 3 vio0
>...and a whole lot more
>
>=== Checking status on the client ===
>
># ipsecctl -sa
>FLOWS:
>No flows
>
>SAD:
>esp tunnel from 2001:db8:2::1 to 2600:db8:1::1 spi 0x001a3754 enc aes-128-gcm
>
>$ ifconfig lo1
>lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> rdomain 1 mtu 32768
> index 4 priority 0 llprio 3
> groups: lo
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
> inet6 2001:db8:2::485b:4ac7 prefixlen 128
>
>$ route -T1 show
>Routing tables
>
>Internet6:
>Destination Gateway Flags Refs Use Mtu Prio Iface
>default 2001:db8:2::485b:4ac7 UGS 0 1 32768 6
> lo1
>localhost localhost UHl 0 37 32768 1 lo1
>2001:db8:2::485b:4ac7 2001:db8:2::485b:4ac7 UHhl 1 2 32768 1 lo1
>fe80::1%lo1 fe80::1%lo1 UHl 0 2893 32768 1 lo1
>ff01::%lo1/32 fe80::1%lo1 Um 0 12 32768 4 lo1
>ff02::%lo1/32 fe80::1%lo1 Um 0 12 32768 4 lo1
>
>[This is the complete routing table; I didn't delete any rows.]
>
>====
>
>We see that the server sees two security associations (one in each
>direction) and two flows (also one in each direction), but the client
>sees only one security association (in the server-to-client direction)
>and no flows.
>
>I trimmed a lot of irrelevant stuff out of the server's routing table
>but it looks correct. The server recognizes itself as the gateway for
>the entire 2001:db8:2::/64 subnet (though this has nothing to do with
>the IPsec tunnel, and is just a statement that traffic on the local
>subnet doesn't need to go through a router, and thus does not require
>an external gateway). It also recognizes that traffic to the client's
>virtual IP should be sent over "link#1", which I assume is the tunnel.
>
>Meanwhile, the client's routing table seems wrong. The client
>recognizes that it owns its own virtual IP address, and it recognizes
>itself as its own default gateway. But it doesn't have any references
>to "link#n" for any n. That perhaps makes sense bcause there is no
>outward flow to route packets to.
>
>We get three interesting results from pings:
>
>=== Test #1 ===
>
>If I run "ping6 2001:db8:2::485b:4ac7" from the server and watch the
>egress interfaces on both the client and server and filter on "esp",
>I see appropriately-sized packets with "spi 0x001a3754". (If I pass
>the "-s" option to ping6, the sizes of the packets change.) If I run
>"netstat -s -pesp" on both the client and the server, I see that the
>server thinks it has output some number of ESP packets and received
>none, and the client htinks it has received the same number of ESP
>packets but sent none. So, OK, clearly we have a one-sided ping.
>
>But when I run "tcpdump -ilo1" on the client I get nothing! The
>encrypted ESP packets are coming in, and iked.conf says they're
>supposed to emerge on lo1 (which ifconfig confirms exists and has
>the expected automatically-assigned IP address) but the decrypted
>ICMP packets are nowhere to be seen.
>
>I get the same behavior if I run "nc -u 2001:db8:2::485b:4ac7 5000"
>from the server---the only differences between this and ping6 are
>that I'm not *expecting* any replies, but also I'm sending UDP
>packets instead of ICMPv6 packets so it seems like they'd be more
>likely to show up on the lo1 interface. But they don't. (I also
>tried "tcpdump -ienc0" on the client with both tests but the packets
>never show up anywhere.)
>
>=== Test #2 ===
>
>The client can't ping the server at all!
>
>$ route -T1 exec ping6 2001:db8:2::1
>PING 2001:db8:2::1 (2001:db8:2::1): 56 data bytes
>ping6: sendmsg: Permission denied
>ping: wrote mudge 64 chars, ret=-1
>ping6: sendmsg: Permission denied
>ping: wrote mudge 64 chars, ret=-1
>
>Curiously, I get the same results if I try to have the client ping
>itself at 2001:db8:2::485b:4ac7, but the client *can* ping itself
>at both ::1 and fe80::1%lo1 from routing domain 1.
>
>How in the world am I getting "Permission denied" when I ping
>certain addresses? What does that even mean?
>
>=== Test #3 ===
>
>I know I'm getting ahead of myself, if I want this connection to be
>usable for actual connections, the server is going to have to be able
>to route traffic from *other* servers to the client. I happen to have
>a second server handy, so I do the following:
>
>On the original server:
>$ tcpdump -l -ivio0 icmp6 | grep 4ac7
>
>On the extra server:
>$ ping6 2001:db8:2::485b:4ac7
>
>What I see on tcpdump is that the original server is spamming its local
>link with neighbor solicitation requests asking who has
>2001:db8:2::485b:4ac7. That behavior strikes me as a little weird. When
>the server originates a packet that is destined for the VPN IP address,
>it correctly sends it over the IPsec tunnel (even though the reciever
>of that packet appears to drop it on the floor). But when the server
>receives a packet from someone else that is destined for that same VPN
>IP address, the server forgets that this route exists and instead
>thinks, "Hey, I recognize that subnet---I'll bet this address belongs
>to someone else on the local link." Why the discrepancy? The server has
>only one routing domain, so I would expect the same routing rules to
>apply to both.
>
>
>=== In summary ===
>
>For all this mess of detail, I think I have only two problems:
>
>1. Why does my client have only one security association and zero
> flows? I suspect that this is the cause of all of my other issues
> with the tunnel.
what does "route -T1 ipsecctl -sa" say? Do you have an enc interface in rdomain
1?
>
>2. Why does the server route packets over the tunnel only when they
> originate on the server itself? (The ip6.forwarding sysctl is
> enabled---the problem is that it wants to route these packets
> onto the local link, not that it refuses to route at all.)
>
>
>Does anybody have any thoughts on what I'm doing wrong?
>
>Thanks,
>Anthony Coulter
>
