I would love to see a symmetry between Cinder local volumes and Neutron PCI passthrough VIFs.
Not entirely sure I have that clear in my head right now, but I just wanted to share the idea: * describe resource external to nova that is attached to VM in the API (block device mapping and/or vif references) * ideally the nova scheduler needs to be aware of the local capacity, and how that relates to the above information (relates to the cross service scheduling issues) * state of the device should be stored by Neutron/Cinder (attached/detached, capacity, IP, etc), but still exposed to the "scheduler" * connection params get given to Nova from Neutron/Cinder * nova still has the vif driver or volume driver to make the final connection * the disk should be formatted/expanded, and network info injected in the same way as before (cloud-init, config drive, DHCP, etc) John On 29 October 2013 10:17, Irena Berezovsky <ire...@mellanox.com> wrote: > Hi Jiang, Robert, > > IRC meeting option works for me. > > If I understand your question below, you are looking for a way to tie up > between requested virtual network(s) and requested PCI device(s). The way we > did it in our solution is to map a provider:physical_network to an > interface that represents the Physical Function. Every virtual network is > bound to the provider:physical_network, so the PCI device should be > allocated based on this mapping. We can map a PCI alias to the > provider:physical_network. > > > > Another topic to discuss is where the mapping between neutron port and PCI > device should be managed. One way to solve it, is to propagate the allocated > PCI device details to neutron on port creation. > > In case there is no qbg/qbh support, VF networking configuration should be > applied locally on the Host. > > The question is when and how to apply networking configuration on the PCI > device? > > We see the following options: > > · it can be done on port creation. > > · It can be done when nova VIF driver is called for vNIC plugging. > This will require to have all networking configuration available to the VIF > driver or send request to the neutron server to obtain it. > > · It can be done by having a dedicated L2 neutron agent on each > Host that scans for allocated PCI devices and then retrieves networking > configuration from the server and configures the device. The agent will be > also responsible for managing update requests coming from the neutron > server. > > > > For macvtap vNIC type assignment, the networking configuration can be > applied by a dedicated L2 neutron agent. > > > > BR, > > Irena > > > > From: Jiang, Yunhong [mailto:yunhong.ji...@intel.com] > Sent: Tuesday, October 29, 2013 9:04 AM > > > To: Robert Li (baoli); Irena Berezovsky; prashant.upadhy...@aricent.com; > chris.frie...@windriver.com; He, Yongli; Itzik Brown > > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery > (kmestery); Sandhya Dasu (sadasu) > Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network > support > > > > Robert, is it possible to have a IRC meeting? I’d prefer to IRC meeting > because it’s more openstack style and also can keep the minutes clearly. > > > > To your flow, can you give more detailed example. For example, I can > consider user specify the instance with –nic option specify a network id, > and then how nova device the requirement to the PCI device? I assume the > network id should define the switches that the device can connect to , but > how is that information translated to the PCI property requirement? Will > this translation happen before the nova scheduler make host decision? > > > > Thanks > > --jyh > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > Sent: Monday, October 28, 2013 12:22 PM > To: Irena Berezovsky; prashant.upadhy...@aricent.com; Jiang, Yunhong; > chris.frie...@windriver.com; He, Yongli; Itzik Brown > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery > (kmestery); Sandhya Dasu (sadasu) > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network > support > > > > Hi Irena, > > > > Thank you very much for your comments. See inline. > > > > --Robert > > > > On 10/27/13 3:48 AM, "Irena Berezovsky" <ire...@mellanox.com> wrote: > > > > Hi Robert, > > Thank you very much for sharing the information regarding your efforts. Can > you please share your idea of the end to end flow? How do you suggest to > bind Nova and Neutron? > > > > The end to end flow is actually encompassed in the blueprints in a nutshell. > I will reiterate it in below. The binding between Nova and Neutron occurs > with the neutron v2 API that nova invokes in order to provision the neutron > services. The vif driver is responsible for plugging in an instance onto the > networking setup that neutron has created on the host. > > > > Normally, one will invoke "nova boot" api with the —nic options to specify > the nic with which the instance will be connected to the network. It > currently allows net-id, fixed ip and/or port-id to be specified for the > option. However, it doesn't allow one to specify special networking > requirements for the instance. Thanks to the nova pci-passthrough work, one > can specify PCI passthrough device(s) in the nova flavor. But it doesn't > provide means to tie up these PCI devices in the case of ethernet adpators > with networking services. Therefore the idea is actually simple as indicated > by the blueprint titles, to provide means to tie up SRIOV devices with > neutron services. A work flow would roughly look like this for 'nova boot': > > > > -- Specifies networking requirements in the —nic option. Specifically > for SRIOV, allow the following to be specified in addition to the existing > required information: > > . PCI alias > > . direct pci-passthrough/macvtap > > . port profileid that is compliant with 802.1Qbh > > > > The above information is optional. In the absence of them, the > existing behavior remains. > > > > -- if special networking requirements exist, Nova api creates PCI > requests in the nova instance type for scheduling purpose > > > > -- Nova scheduler schedules the instance based on the requested flavor > plus the PCI requests that are created for networking. > > > > -- Nova compute invokes neutron services with PCI passthrough > information if any > > > > -- Neutron performs its normal operations based on the request, such > as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it > should validate the information such as profileid, and stores them in its > db. It's also possible to associate a port profileid with a neutron network > so that port profileid becomes optional in the —nic option. Neutron returns > nova the port information, especially for PCI passthrough related > information in the port binding object. Currently, the port binding object > contains the following information: > > binding:vif_type > > binding:host_id > > binding:profile > > binding:capabilities > > > > -- nova constructs the domain xml and plug in the instance by calling > the vif driver. The vif driver can build up the interface xml based on the > port binding information. > > > > > > > > > > The blueprints you registered make sense. On Nova side, there is a need to > bind between requested virtual network and PCI device/interface to be > allocated as vNIC. > > On the Neutron side, there is a need to support networking configuration of > the vNIC. Neutron should be able to identify the PCI device/macvtap > interface in order to apply configuration. I think it makes sense to provide > neutron integration via dedicated Modular Layer 2 Mechanism Driver to allow > PCI pass-through vNIC support along with other networking technologies. > > > > I haven't sorted through this yet. A neutron port could be associated with a > PCI device or not, which is a common feature, IMHO. However, a ML2 driver > may be needed specific to a particular SRIOV technology. > > > > > > During the Havana Release, we introduced Mellanox Neutron plugin that > enables networking via SRIOV pass-through devices or macvtap interfaces. > > We want to integrate our solution with PCI pass-through Nova support. I > will be glad to share more details if you are interested. > > > > > > Good to know that you already have a SRIOV implementation. I found out some > information online about the mlnx plugin, but need more time to get to know > it better. And certainly I'm interested in knowing its details. > > > > The PCI pass-through networking support is planned to be discussed during > the summit: http://summit.openstack.org/cfp/details/129. I think it’s worth > to drill down into more detailed proposal and present it during the summit, > especially since it impacts both nova and neutron projects. > > > > I agree. Maybe we can steal some time in that discussion. > > > > Would you be interested in collaboration on this effort? Would you be > interested to exchange more emails or set an IRC/WebEx meeting during this > week before the summit? > > > > Sure. If folks want to discuss it before the summit, we can schedule a webex > later this week. Or otherwise, we can continue the discussion with email. > > > > > > > > Regards, > > Irena > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > Sent: Friday, October 25, 2013 11:16 PM > To: prashant.upadhy...@aricent.com; Irena Berezovsky; > yunhong.ji...@intel.com; chris.frie...@windriver.com; yongli...@intel.com > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery > (kmestery); Sandhya Dasu (sadasu) > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network > support > > > > Hi Irena, > > > > This is Robert Li from Cisco Systems. Recently, I was tasked to investigate > such support for Cisco's systems that support VM-FEX, which is a SRIOV > technology supporting 802-1Qbh. I was able to bring up nova instances with > SRIOV interfaces, and establish networking in between the instances that > employes the SRIOV interfaces. Certainly, this was accomplished with hacking > and some manual intervention. Based on this experience and my study with the > two existing nova pci-passthrough blueprints that have been implemented and > committed into Havana > (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt), I > registered a couple of blueprints (one on Nova side, the other on the > Neutron side): > > > > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov > > https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov > > > > in order to address SRIOV support in openstack. > > > > Please take a look at them and see if they make sense, and let me know any > comments and questions. We can also discuss this in the summit, I suppose. > > > > I noticed that there is another thread on this topic, so copy those folks > from that thread as well. > > > > thanks, > > Robert > > > > On 10/16/13 4:32 PM, "Irena Berezovsky" <ire...@mellanox.com> wrote: > > > > Hi, > > As one of the next steps for PCI pass-through I would like to discuss is the > support for PCI pass-through vNIC. > > While nova takes care of PCI pass-through device resources management and > VIF settings, neutron should manage their networking configuration. > > I would like to register asummit proposal to discuss the support for PCI > pass-through networking. > > I am not sure what would be the right topic to discuss the PCI pass-through > networking, since it involve both nova and neutron. > > There is already a session registered by Yongli on nova topic to discuss the > PCI pass-through next steps. > > I think PCI pass-through networking is quite a big topic and it worth to > have a separate discussion. > > Is there any other people who are interested to discuss it and share their > thoughts and experience? > > > > Regards, > > Irena > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
