Hi JeanPaul, Just for your information, CloudStack starts the VMs from a XML file which is different from "virsh dumpxml".
You can get the VM definition XML in agent.log if you have enabled DEBUG in log4j. If not, you can do it by the following command cp /etc/cloudstack/agent/log4j-cloud.xml /etc/cloudstack/agent/log4j-cloud.xml.orig && \ sed -i "s/INFO/DEBUG/g" /etc/cloudstack/agent/log4j-cloud.xml && \ systemctl restart cloudstack-agent -Wei On Thu, 16 Feb 2023 at 11:42, JeanPaul van der Mijle < [email protected]> wrote: > Hi all, > > Another update, think we can close this anyway now, if there are other > things going south it is most likely somewhere else. > > In the tests of KVM/Qemu before Cloudstack I could give any address it > needs to be exposed to. In Cloudstack you would need to start on domain 0, > bus 0, slot 0 and function 0 and count up the function for each device. > Doing it differently will not expose it to the vm for me. > > I should take a look further to better understand why this difference as > I'd love to understand and also document this very valuable information for > myself. > > Thanks again! > > JeanPaul > ________________________________ > From: JeanPaul van der Mijle <[email protected]> > Sent: Wednesday, February 15, 2023 4:29 PM > To: [email protected] <[email protected]> > Subject: Re: PCI-E Passthrough > > Hi All, > > Adding some additional information, as for custom configuration it now > works. > I did have some trouble first with getting the VM boot, as it apparently > needs specific settings for the pci bus/slot etc. I took over the config as > is from an VM where it worked with before, before I installed cloudstack. > Now I have found that out, I am stuck with the following: > > When starting the VM, I see in syslog on the host/hypervisor that it is > enabling the device for passthrough: > Feb 15 16:20:48 gc-hs1 kernel: [18145.274514] vfio-pci 0000:81:00.2: > enabling device (0000 -> 0002) > > I also can see it in the log of the VM, it uses the parameters: > -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:00:34:00:00:f6,bus=pci.0,addr=0x3 > \ > > When doing virsh edit on the domain, I get: > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/> > </source> > <alias name='hostdev0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' > function='0x1'/> > </hostdev> > > > Only thing I never added was driver name, I am a bit suspicious on that > but probably has it's reasons. It worked on stand alone virsh+qemu setup > but there I never had to give up the driver. > > Is there some additional configuration that is required? > > For now I test with 1 device, but eventually there should be 4 Mellanox > VF's being exposed/passed to the VM. > > Thank you yet again, > > JeanPaul > ________________________________ > From: Wei ZHOU <[email protected]> > Sent: Wednesday, February 15, 2023 10:42 AM > To: [email protected] <[email protected]> > Subject: Re: PCI-E Passthrough > > You are welcome. Good to know it works. > > -Wei > > > On Wednesday, 15 February 2023, JeanPaul van der Mijle < > [email protected]> wrote: > > > Hi All, > > > > Nevermind, figured out, it is very important on the alphabetic order when > > signing the request. > > > > Thanks anyways! > > > > JeanPaul > > ________________________________ > > From: JeanPaul van der Mijle <[email protected]> > > Sent: Wednesday, February 15, 2023 10:05 AM > > To: [email protected] <[email protected]> > > Subject: Re: PCI-E Passthrough > > > > Hi Wei, All, > > > > I am currently stuck with the API call, I manage to sign requests as > > listUsers and plain updateVirtualMachine with id works, but as soon as I > > add the extraconfig parameter, it keeps complaining it is not correct. > This > > is so far with GET request only, I have not yet tried POST, as I haven't > > taken a look if I should do every value with post (command, id, > > extraconfig, response and signature?) or just the extraconfig and how > > signing would go then. > > > > I have tried to sign the request with: "command=updatevirtualmachine& > > id=xxxx&extraconfig=urlencodedconfig" > > both fully lowercase, as well lowercase only the command. > > > > I also tried without adding extraconfig to the signature generation but > > they all end up with: > > > > ["errorcode"]=> > > int(401) > > ["errortext"]=> > > string(58) "unable to verify user credentials and/or request > signature" > > > > Currently using PHP to do the signatures and it looks like as following: > > > > <?php > > $baseurl = "http://localhost:8080/client/api?";; > > > > $response = "response=json"; > > > > $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91"; > > > > $apikey = "apikey=<apikey>"; > > $secretkey = "<secret>"; > > $configxml = <<<EOD > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > > function='0x0' multifunction='on'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x09' slot='0x00' > > function='0x0' multifunction='on'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > > function='0x1'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x09' slot='0x00' > > function='0x1'/> > > </hostdev> > > <memballoon model='virtio'> > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > > function='0x0'/> > > </memballoon> > > <rng model='virtio'> > > <backend model='random'>/dev/urandom</backend> > > <address type='pci' domain='0x0000' bus='0x05' slot='0x00' > > function='0x0'/> > > </rng> > > EOD; > > > > $extraconfig = "extraconfig=" . urlencode($configxml); > > > > $command = "command=updateVirtualMachine&".$vmid; > > > > $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" . > > $extraconfig . "&" . $response,$secretkey, true); > > $base64encoded = base64_encode($hash); > > $signature = "signature=" . urlencode($base64encoded); > > > > $link = $baseurl . $apikey . "&" . $command . "&" . $extraconfig . "&" > > .$response . "&" . $signature; > > > > $cURLConnection = curl_init(); > > > > curl_setopt($cURLConnection, CURLOPT_URL, $link); > > curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true); > > > > $json_output = curl_exec($cURLConnection); > > curl_close($cURLConnection); > > > > $jsonArrayResponse = json_decode($json_output,true); > > > > var_dump($jsonArrayResponse); > > > > > > The URL end up looking like this: > > > > http://localhost:8080/client/api?apikey=<Apikey>&command= > > updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode% > > 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27% > > 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D% > > 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D% > > 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++% > > 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D% > > 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+ > > multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++ > > ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci% > > 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++% > > 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D% > > 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource% > > 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000% > > 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0% > > 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E% > > 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci% > > 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++% > > 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D% > > 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource% > > 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000% > > 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1% > > 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D% > > 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++ > > ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000% > > 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6% > > 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D% > > 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D% > > 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev% > > 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++% > > 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D% > > 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+ > > +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E% > > 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev% > > 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci% > > 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00% > > 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response= > > json&signature=<signature> > > > > Thanks, > > > > JeanPaul > > ________________________________ > > From: JeanPaul van der Mijle <[email protected]> > > Sent: Sunday, February 12, 2023 6:41 PM > > To: [email protected] <[email protected]> > > Subject: Re: PCI-E Passthrough > > > > Hi Wei, > > > > Thanks, going to take a deeper look into this. > > > > I assume I will have to add the list as following, to allow the XML below > > to be accepted by the API when I submit this encoded? > > > > I set this to be allowed for KVM: domain,devices,hostdev,source, > > address,memballoon,rng,backend > > > > It should reflect this XML part: > > > > <domain> > > <device> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > > function='0x0' multifunction='on'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x09' slot='0x00' > > function='0x0' multifunction='on'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > > function='0x1'/> > > </hostdev> > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/> > > </source> > > <address type='pci' domain='0x0000' bus='0x09' slot='0x00' > > function='0x1'/> > > </hostdev> > > <memballoon model='virtio'> > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > > function='0x0'/> > > </memballoon> > > <rng model='virtio'> > > <backend model='random'>/dev/urandom</backend> > > <address type='pci' domain='0x0000' bus='0x05' slot='0x00' > > function='0x0'/> > > </rng> > > </device> > > </domain> > > > > I only need to figure out if domain and device are required or not but I > > will see when I test it out. > > > > Thanks so far! > > > > JeanPaul > > ________________________________ > > From: Wei ZHOU <[email protected]> > > Sent: Sunday, February 12, 2023 5:23 PM > > To: [email protected] <[email protected]> > > Subject: Re: PCI-E Passthrough > > > > I think you need to add additional configuration to vm. > > Refer to > > https://www.shapeblue.com/cloudstack-feature-first-look- > > enable-sending-of-arbitrary-configuration-data-to-vms/ > > > > -Wei > > > > > > > > On Sunday, 12 February 2023, JeanPaul van der Mijle < > > [email protected]> wrote: > > > > > Hi all, > > > > > > Does anyone have a good guide on adding a PCI-E card (A Mellanox > > Connect-X > > > 5 VF interface) to cloudstack? > > > I had this all running before in stock Qemu, and all settings are still > > > present for the card and VF's, I just need to know how the management > > sees > > > that I have additional cards that I have available to assign to VMs. > > > > > > I tried adding: > > > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4| > > > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0| > > > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4| > > > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1 > > > > > > However, after restarting the agent, I noticed that it added escape > > > characters: > > > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4| > > > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0| > > > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4| > > > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0| > > 100GE,c1\:01.1 > > > > > > Not sure if this is fine, but so far it was the only thing I could find > > > back on google, but this is 10 years ago. I suspect this is no longer > > > sufficient. > > > After restarting the agent, I do not see any changes unfortunately. > > > > > > Thanks! > > > > > > JeanPaul > > > > > > > > >
