While I suppose the /64 your VPS provider gives you is "enormous" compared to IPv4, I don't find such a comparison relevant since IPv6 and IPv4 are entirely different protocols. In fact I actually think it is small. Why? RFC 6177 (https://datatracker.ietf.org/doc/html/rfc6177) recommends that /48 or at least /56 subnets be given to end sites, so your _small_ /64 violates that recommendation. Hell, even my lowly residential ISP, Xfinity/Comcast, gives me a /60. Unfortunately a great many ISPs and VPS providers violate this. Not sure if it is due to incompetence where they incorrectly think such allocations are "wasteful" or what. IPv6 not only restores end-to-end communication the way IPv4 initially started, but it is designed so that sites have many _subnets_. This brings me to the next point.
You would like to rely on SLAAC for your VPN peers, but SLAAC will likely not work on anything smaller than /64. Why? Because the first 64 bits of an IPv6 address is designated as the network identifier. You already carved out some IPs from the /64 though which means you have less than /64 to use for SLAAC inside the tunnel. I used to use Vultr; but when they were unwilling to provide something bigger than a /64 in addition to actually routing the entire block, I left them. If you insist on using IPv6 without relying on NAT or NDP proxying, then I recommend you find another provider. What you are trying to do is trivial when IPv6 is done properly. I have a similar setup myself except I use WireGuard, but I'm confident IKEv2/IPSec would be easy to set up as well.
