Hi John, Great to hear from you on Cinder with pcipassthrough. I thought that it would be coming. I like the idea.
thanks, Robert On 10/29/13 6:46 AM, "John Garbutt" <j...@johngarbutt.com> wrote: >I would love to see a symmetry between Cinder local volumes and >Neutron PCI passthrough VIFs. > >Not entirely sure I have that clear in my head right now, but I just >wanted to share the idea: >* describe resource external to nova that is attached to VM in the API >(block device mapping and/or vif references) >* ideally the nova scheduler needs to be aware of the local capacity, >and how that relates to the above information (relates to the cross >service scheduling issues) >* state of the device should be stored by Neutron/Cinder >(attached/detached, capacity, IP, etc), but still exposed to the >"scheduler" >* connection params get given to Nova from Neutron/Cinder >* nova still has the vif driver or volume driver to make the final >connection >* the disk should be formatted/expanded, and network info injected in >the same way as before (cloud-init, config drive, DHCP, etc) > >John > >On 29 October 2013 10:17, Irena Berezovsky <ire...@mellanox.com> wrote: >> Hi Jiang, Robert, >> >> IRC meeting option works for me. >> >> If I understand your question below, you are looking for a way to tie up >> between requested virtual network(s) and requested PCI device(s). The >>way we >> did it in our solution is to map a provider:physical_network to an >> interface that represents the Physical Function. Every virtual network >>is >> bound to the provider:physical_network, so the PCI device should be >> allocated based on this mapping. We can map a PCI alias to the >> provider:physical_network. >> >> >> >> Another topic to discuss is where the mapping between neutron port and >>PCI >> device should be managed. One way to solve it, is to propagate the >>allocated >> PCI device details to neutron on port creation. >> >> In case there is no qbg/qbh support, VF networking configuration >>should be >> applied locally on the Host. >> >> The question is when and how to apply networking configuration on the >>PCI >> device? >> >> We see the following options: >> >> · it can be done on port creation. >> >> · It can be done when nova VIF driver is called for vNIC >>plugging. >> This will require to have all networking configuration available to >>the VIF >> driver or send request to the neutron server to obtain it. >> >> · It can be done by having a dedicated L2 neutron agent on each >> Host that scans for allocated PCI devices and then retrieves networking >> configuration from the server and configures the device. The agent will >>be >> also responsible for managing update requests coming from the neutron >> server. >> >> >> >> For macvtap vNIC type assignment, the networking configuration can be >> applied by a dedicated L2 neutron agent. >> >> >> >> BR, >> >> Irena >> >> >> >> From: Jiang, Yunhong [mailto:yunhong.ji...@intel.com] >> Sent: Tuesday, October 29, 2013 9:04 AM >> >> >> To: Robert Li (baoli); Irena Berezovsky; prashant.upadhy...@aricent.com; >> chris.frie...@windriver.com; He, Yongli; Itzik Brown >> >> >> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle >>Mestery >> (kmestery); Sandhya Dasu (sadasu) >> Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network >> support >> >> >> >> Robert, is it possible to have a IRC meeting? I¹d prefer to IRC meeting >> because it¹s more openstack style and also can keep the minutes clearly. >> >> >> >> To your flow, can you give more detailed example. For example, I can >> consider user specify the instance with nic option specify a network >>id, >> and then how nova device the requirement to the PCI device? I assume the >> network id should define the switches that the device can connect to , >>but >> how is that information translated to the PCI property requirement? Will >> this translation happen before the nova scheduler make host decision? >> >> >> >> Thanks >> >> --jyh >> >> >> >> From: Robert Li (baoli) [mailto:ba...@cisco.com] >> Sent: Monday, October 28, 2013 12:22 PM >> To: Irena Berezovsky; prashant.upadhy...@aricent.com; Jiang, Yunhong; >> chris.frie...@windriver.com; He, Yongli; Itzik Brown >> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle >>Mestery >> (kmestery); Sandhya Dasu (sadasu) >> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network >> support >> >> >> >> Hi Irena, >> >> >> >> Thank you very much for your comments. See inline. >> >> >> >> --Robert >> >> >> >> On 10/27/13 3:48 AM, "Irena Berezovsky" <ire...@mellanox.com> wrote: >> >> >> >> Hi Robert, >> >> Thank you very much for sharing the information regarding your efforts. >>Can >> you please share your idea of the end to end flow? How do you suggest >>to >> bind Nova and Neutron? >> >> >> >> The end to end flow is actually encompassed in the blueprints in a >>nutshell. >> I will reiterate it in below. The binding between Nova and Neutron >>occurs >> with the neutron v2 API that nova invokes in order to provision the >>neutron >> services. The vif driver is responsible for plugging in an instance >>onto the >> networking setup that neutron has created on the host. >> >> >> >> Normally, one will invoke "nova boot" api with the ‹nic options to >>specify >> the nic with which the instance will be connected to the network. It >> currently allows net-id, fixed ip and/or port-id to be specified for the >> option. However, it doesn't allow one to specify special networking >> requirements for the instance. Thanks to the nova pci-passthrough work, >>one >> can specify PCI passthrough device(s) in the nova flavor. But it doesn't >> provide means to tie up these PCI devices in the case of ethernet >>adpators >> with networking services. Therefore the idea is actually simple as >>indicated >> by the blueprint titles, to provide means to tie up SRIOV devices with >> neutron services. A work flow would roughly look like this for 'nova >>boot': >> >> >> >> -- Specifies networking requirements in the ‹nic option. >>Specifically >> for SRIOV, allow the following to be specified in addition to the >>existing >> required information: >> >> . PCI alias >> >> . direct pci-passthrough/macvtap >> >> . port profileid that is compliant with 802.1Qbh >> >> >> >> The above information is optional. In the absence of them, the >> existing behavior remains. >> >> >> >> -- if special networking requirements exist, Nova api creates PCI >> requests in the nova instance type for scheduling purpose >> >> >> >> -- Nova scheduler schedules the instance based on the requested >>flavor >> plus the PCI requests that are created for networking. >> >> >> >> -- Nova compute invokes neutron services with PCI passthrough >> information if any >> >> >> >> -- Neutron performs its normal operations based on the request, >>such >> as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it >> should validate the information such as profileid, and stores them in >>its >> db. It's also possible to associate a port profileid with a neutron >>network >> so that port profileid becomes optional in the ‹nic option. Neutron >>returns >> nova the port information, especially for PCI passthrough related >> information in the port binding object. Currently, the port binding >>object >> contains the following information: >> >> binding:vif_type >> >> binding:host_id >> >> binding:profile >> >> binding:capabilities >> >> >> >> -- nova constructs the domain xml and plug in the instance by >>calling >> the vif driver. The vif driver can build up the interface xml based on >>the >> port binding information. >> >> >> >> >> >> >> >> >> >> The blueprints you registered make sense. On Nova side, there is a need >>to >> bind between requested virtual network and PCI device/interface to be >> allocated as vNIC. >> >> On the Neutron side, there is a need to support networking >>configuration of >> the vNIC. Neutron should be able to identify the PCI device/macvtap >> interface in order to apply configuration. I think it makes sense to >>provide >> neutron integration via dedicated Modular Layer 2 Mechanism Driver to >>allow >> PCI pass-through vNIC support along with other networking technologies. >> >> >> >> I haven't sorted through this yet. A neutron port could be associated >>with a >> PCI device or not, which is a common feature, IMHO. However, a ML2 >>driver >> may be needed specific to a particular SRIOV technology. >> >> >> >> >> >> During the Havana Release, we introduced Mellanox Neutron plugin that >> enables networking via SRIOV pass-through devices or macvtap interfaces. >> >> We want to integrate our solution with PCI pass-through Nova support. I >> will be glad to share more details if you are interested. >> >> >> >> >> >> Good to know that you already have a SRIOV implementation. I found out >>some >> information online about the mlnx plugin, but need more time to get to >>know >> it better. And certainly I'm interested in knowing its details. >> >> >> >> The PCI pass-through networking support is planned to be discussed >>during >> the summit: http://summit.openstack.org/cfp/details/129. I think it¹s >>worth >> to drill down into more detailed proposal and present it during the >>summit, >> especially since it impacts both nova and neutron projects. >> >> >> >> I agree. Maybe we can steal some time in that discussion. >> >> >> >> Would you be interested in collaboration on this effort? Would you be >> interested to exchange more emails or set an IRC/WebEx meeting during >>this >> week before the summit? >> >> >> >> Sure. If folks want to discuss it before the summit, we can schedule a >>webex >> later this week. Or otherwise, we can continue the discussion with >>email. >> >> >> >> >> >> >> >> Regards, >> >> Irena >> >> >> >> From: Robert Li (baoli) [mailto:ba...@cisco.com] >> Sent: Friday, October 25, 2013 11:16 PM >> To: prashant.upadhy...@aricent.com; Irena Berezovsky; >> yunhong.ji...@intel.com; chris.frie...@windriver.com; >>yongli...@intel.com >> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle >>Mestery >> (kmestery); Sandhya Dasu (sadasu) >> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network >> support >> >> >> >> Hi Irena, >> >> >> >> This is Robert Li from Cisco Systems. Recently, I was tasked to >>investigate >> such support for Cisco's systems that support VM-FEX, which is a SRIOV >> technology supporting 802-1Qbh. I was able to bring up nova instances >>with >> SRIOV interfaces, and establish networking in between the instances that >> employes the SRIOV interfaces. Certainly, this was accomplished with >>hacking >> and some manual intervention. Based on this experience and my study >>with the >> two existing nova pci-passthrough blueprints that have been implemented >>and >> committed into Havana >> (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and >> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt), I >> registered a couple of blueprints (one on Nova side, the other on the >> Neutron side): >> >> >> >> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov >> >> https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov >> >> >> >> in order to address SRIOV support in openstack. >> >> >> >> Please take a look at them and see if they make sense, and let me know >>any >> comments and questions. We can also discuss this in the summit, I >>suppose. >> >> >> >> I noticed that there is another thread on this topic, so copy those >>folks >> from that thread as well. >> >> >> >> thanks, >> >> Robert >> >> >> >> On 10/16/13 4:32 PM, "Irena Berezovsky" <ire...@mellanox.com> wrote: >> >> >> >> Hi, >> >> As one of the next steps for PCI pass-through I would like to discuss >>is the >> support for PCI pass-through vNIC. >> >> While nova takes care of PCI pass-through device resources management >>and >> VIF settings, neutron should manage their networking configuration. >> >> I would like to register asummit proposal to discuss the support for PCI >> pass-through networking. >> >> I am not sure what would be the right topic to discuss the PCI >>pass-through >> networking, since it involve both nova and neutron. >> >> There is already a session registered by Yongli on nova topic to >>discuss the >> PCI pass-through next steps. >> >> I think PCI pass-through networking is quite a big topic and it worth to >> have a separate discussion. >> >> Is there any other people who are interested to discuss it and share >>their >> thoughts and experience? >> >> >> >> Regards, >> >> Irena >> >> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
