Le mardi 26 septembre 2023 à 13:20 +0200, Stefan Hanreich a écrit :
> On 9/20/23 23:48, DERUMIER, Alexandre wrote:
> > Finally, It's not so easy without writing ip on proxmox side (in vm
> > config or somewhere else), because to retrieve a reserved ip from
> > external ipam when vm start, we need to lookup maybe from mac
> > address,
> > maybe from hostname of the vm, or maybe some custom attributes, but
> > not
> > all ipams accept same attributes.
> >
> > (at least phpipam && netbox don't support all features, or not
> > easyly.
> > Netbox for example, for macaddress need to register the full vm
> > object
> > && interfaces + mac + mapping to ip, Phpipam is a single ip object
> > with mac as attribute).
>
> Yes, I think so as well. It also would make us dependent on external
> systems, which might not always be up and would create an additional
> hurdle for setting things up. Having our own solution for this seems
> preferable imo. We can still provide integrations with netbox /
> phpipam
> so they can take over from our small IPAM if they implement the
> features
> we need.
>
> I'll take a closer look at netbox, since I was under the impression
> that
> they should support this - although it's been awhile since I played
> around with it. Not sure about phpIPAM, but I wasn't too stoked on
> using
> it anyway after browsing their source code for a bit.
>
> > So I think the best way is still to write the ip into the vm
> > config,
> > this allow to inject already reserved ip in dhcp at vm
> > start/migrate
> > without need to call the ipam (also avoid start problem is ipam
> > server
> > is down).
> >
> > and this allow to use it for firewall ipfilter, I see a usecase for
> > sdn
> > vxlan too or special /32 route injection)
> >
>
> Yes, I think so as well, although we would need to take care of
> proper
> synchronization between Configs and IPAM. If this diverges for
> whatever
> reason we will run into trouble. Of course, this *should* never
> happen
> when properly implemented.
>
> Another option I thought about would be storing a VMID -> IP mapping
> in
> the (pve) IPAM itself. This would have the upside of having a
> centralized storage and single source of truth without having to
> maintain two different locations where we store the IP.
>
> Though it would also be a bit more intransparent to the user if we
> don't
> expose it somewhere in the UI.
>
> This would have the downside that starting VMs is an issue when the
> IPAM
> is down. While using the pve IPAM in a cluster (or a single node) I
> can
> see this being alright, since you need quorum to start a VM. As long
> as
> you have a quorate cluster the pve IPAM *should* be available as
> well.
>
> In the case of using phpIPAM or netbox this is an issue we would need
> to
> think about.
>
Yes, this is my main concern, as it'll be my case in production, as I
managing multiple clusters, on differents location, with subnets
sharing.
for me, it's ok if ipam is down when allocating a new ip or vm.
But for vm start/stop, I think we should have at minimum some cache
somewhere. (I'm think about a disaster recovery or big network problem,
where you want to fast restart all vms without need to call the ipam).
Maybe a way, could be to use the local pve ipam, as a local mirror of
the external ipam ? (and don't store ip in vm config, but only in
pve ipam, the source of truth)
vm with ipam=auto, external ipam is configured:
- if (ip_exist(pve_ipam)) {
return ip
} elsif (ip_exist(external_ipam)) {
return ip
} else {
ip = findnextfreeip(external_ipam)
sync_pve_ipam(ip)
return ip
}
I have see this in vmware or openstack/opennebula with external ipam,
I don't remember exactly.
> > I just need some protections for snapshot, but nothing too
> > difficult,
> > but we really need to avoid to try to manage in ipam multiple
> > version/snapshot of ip entry for a vm.
> > I had tried 2years ago, it was really painful to handle this in
> > differents ipam.
> > So maybe the best way is to forbid to change ip address when a
> > snapshot
> > already exist.
>
> Yes, it might be just the best way to check on restore if the IP is
> the
> same or at least currently available and otherwise just get a new IP
> from IPAM automatically (maybe with a warning).
>
> On the other hand, this should not be an issue when storing the VMID
> ->
> IP mapping centralized somewhere, since we then can just rely on the
> IP
> being stored there. Of course this would exclude the DHCP/IP setting
> from the snapshot which can be good or bad I'd say (depending on the
> use
> case).
>
> > I think we could implement ipam call like:
> >
> >
> > create vm or add a new nic -->
> > -----------------------------
> > qm create ... -net0
> > bridge=vnet,....,ip=(auto|192.168.0.1|dynamic),ip6=(..)
> >
> >
> > auto : search a free ip in ipam. write the ip address in net0:
> > ...,ip=
> > ip field
> >
> > 192.168.0.1: check if ip is free in ipam && register ip in ipam.
> > write
> > the ip in ip field.
> >
> >
> > dynamic: write "ephemeral" in net0: ....,ip=ephemeral (This is a
> > dynamic ip registered at vm start, and release at vm stop)
>
> Sounds good to me.
>
> >
> > vm start
> > ---------
> > - if ip=ephemeral, find && register a free ip in ipam, write it in
> > vm
> > net0: ...,ip=192.168.0.10[E] . (maybe with a special flag [E] to
> > indicate it's ephemeral)
> > - read ip from vm config && inject in dhcp
>
> Maybe we can even get away with setting the IP in the DHCP config as
> soon as we set it in the VM configuration, as long as it is not
> ephemeral, thus avoiding the need for having to do it while starting
> VMs?
>
> > vm_stop
> > -------
> > if ip is ephemeral (netX: ip=192.168.0.10[E]), delete ip from
> > ipam,
> > set ip=ephemeral in vm config
> >
> >
> > vm_destroy or nic remove/unplug
> > -------------------------
> > if netX: ...,ip=192.168.0.10 , remove ip from ipam
> >
> >
> >
> > nic update when vm is running:
> > ------------------------------
> > if ip is defined : netX: ip=192.168.0.10, we don't allow bridge
> > change
> > or ip change, as vm is not notified about theses changes, and still
> > use
> > old ip.
> >
> > We can allow nic hot-unplug && hotplug. (guest os will remove the
> > ip on
> > nic removal, and will call dhcp again on nic hotplug)
>
> Yes, I think so as well. Maybe we could give the option to change the
> IP
> together with a forced reboot and a warning like 'When changing the
> IP
> this VM will get rebooted' as a quality of life feature?
>
> >
> > nic hotplug with ip=auto:
> > -------------------------
> >
> > --> add nic in pending state ----> find ip in ipam && write it in
> > pending ---> do the hotplug in qemu.
> >
> > We need to handle the config revert to remove ip from ipam if the
> > nic
> > hotplug is blocked in pending state(I never see this case until os
> > don't have pci_hotplug module loaded, but it's better to be
> > carefull )
> >
>
> Yes - defensive programming is always good!
>
> > The ipam modules (internal pve, phpipam,netbox) are already for
> > this, I
> > think it shouldn't be too difficult.
> >
> > dnsmasq seem to have a reservation file option, where we can
> > dynamically add ip-mac without need to reload it.
> >
> > I'll try it, re-using your current dnsmasq patches.
>
> Since you want to take a shot at implementing it, is there anything I
> could help you with? I'd have some resources now for taking a shot at
> this as well.
>
I'm a bit busy currently on other stuff and I would like to finish them
first.
So if you have a little bit time to work on this, it could be great :)
I have send some patches in 2021 for ipam integration in qemu/lxc, if
you want to take some inspiration. (without the ip in the vm config, it
should be a lot easier)
> It would also be interesting to improve and add some features to our
> built-in IPAM, maybe even add the VMID -> IP mapping functionality
> I've
> touched upon earlier. It would also be interesting to be able to
> expose
> some of this information to the frontend, so users have an overview
> of
> currently leased IPs in the frontend - what do you think?
>
Yes,admin should be able to see allocated ip. (like a real ipam).
I was thinking about other stuff for later, but maybe it could be great
for an admin to be able to reserve ips and put them in a pool.
Then user could choose ip from this pool.
(Usecase is public ip addresses, where a customer could buy some of
them,
then allocated them like he want)
> Would it also make sense to set IPSet entries for VMs, so they are
> only
> allowed to use the IPs we dedicate to them? This would be a decent
> safeguard for preventing issues down the line.
>
> Additionally it would be interesting to automatically create Aliases
> for
> VNets/VMs in the Firewall configuration - what do you think? If we
> add
> VMs as Aliases, we would have to recompile the iptables on every IP
> change. For this feature it would make sense to be able to set names
> /
> comments on VNets, so we can reference them this way. What do you
> think?
a Big yes ! (as I'm already doing this manually in production ;)
>
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
