> * describe resource external to nova that is attached to VM in the API > (block device mapping and/or vif references) > * ideally the nova scheduler needs to be aware of the local capacity, > and how that relates to the above information (relates to the cross > service scheduling issues)
I think this possibly a bit different. For volume, it's sure managed by Cinder, but for PCI devices, currently It ;s managed by nova. So we possibly need nova to translate the information (possibly before nova scheduler). > * state of the device should be stored by Neutron/Cinder > (attached/detached, capacity, IP, etc), but still exposed to the > "scheduler" I'm not sure if we can keep the state of the device in Neutron. Currently nova manage all PCI devices. Thanks --jyh > * connection params get given to Nova from Neutron/Cinder > * nova still has the vif driver or volume driver to make the final connection > * the disk should be formatted/expanded, and network info injected in > the same way as before (cloud-init, config drive, DHCP, etc) > > John > > On 29 October 2013 10:17, Irena Berezovsky <ire...@mellanox.com> > wrote: > > Hi Jiang, Robert, > > > > IRC meeting option works for me. > > > > If I understand your question below, you are looking for a way to tie up > > between requested virtual network(s) and requested PCI device(s). The > way we > > did it in our solution is to map a provider:physical_network to an > > interface that represents the Physical Function. Every virtual network is > > bound to the provider:physical_network, so the PCI device should be > > allocated based on this mapping. We can map a PCI alias to the > > provider:physical_network. > > > > > > > > Another topic to discuss is where the mapping between neutron port > and PCI > > device should be managed. One way to solve it, is to propagate the > allocated > > PCI device details to neutron on port creation. > > > > In case there is no qbg/qbh support, VF networking configuration > should be > > applied locally on the Host. > > > > The question is when and how to apply networking configuration on the > PCI > > device? > > > > We see the following options: > > > > * it can be done on port creation. > > > > * It can be done when nova VIF driver is called for vNIC > plugging. > > This will require to have all networking configuration available to the > VIF > > driver or send request to the neutron server to obtain it. > > > > * It can be done by having a dedicated L2 neutron agent on > each > > Host that scans for allocated PCI devices and then retrieves networking > > configuration from the server and configures the device. The agent will > be > > also responsible for managing update requests coming from the neutron > > server. > > > > > > > > For macvtap vNIC type assignment, the networking configuration can be > > applied by a dedicated L2 neutron agent. > > > > > > > > BR, > > > > Irena > > > > > > > > From: Jiang, Yunhong [mailto:yunhong.ji...@intel.com] > > Sent: Tuesday, October 29, 2013 9:04 AM > > > > > > To: Robert Li (baoli); Irena Berezovsky; > prashant.upadhy...@aricent.com; > > chris.frie...@windriver.com; He, Yongli; Itzik Brown > > > > > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle > Mestery > > (kmestery); Sandhya Dasu (sadasu) > > Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network > > support > > > > > > > > Robert, is it possible to have a IRC meeting? I'd prefer to IRC meeting > > because it's more openstack style and also can keep the minutes > clearly. > > > > > > > > To your flow, can you give more detailed example. For example, I can > > consider user specify the instance with -nic option specify a network id, > > and then how nova device the requirement to the PCI device? I assume > the > > network id should define the switches that the device can connect to , > but > > how is that information translated to the PCI property requirement? Will > > this translation happen before the nova scheduler make host decision? > > > > > > > > Thanks > > > > --jyh > > > > > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > > Sent: Monday, October 28, 2013 12:22 PM > > To: Irena Berezovsky; prashant.upadhy...@aricent.com; Jiang, Yunhong; > > chris.frie...@windriver.com; He, Yongli; Itzik Brown > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle > Mestery > > (kmestery); Sandhya Dasu (sadasu) > > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network > > support > > > > > > > > Hi Irena, > > > > > > > > Thank you very much for your comments. See inline. > > > > > > > > --Robert > > > > > > > > On 10/27/13 3:48 AM, "Irena Berezovsky" <ire...@mellanox.com> > wrote: > > > > > > > > Hi Robert, > > > > Thank you very much for sharing the information regarding your efforts. > Can > > you please share your idea of the end to end flow? How do you suggest > to > > bind Nova and Neutron? > > > > > > > > The end to end flow is actually encompassed in the blueprints in a > nutshell. > > I will reiterate it in below. The binding between Nova and Neutron > occurs > > with the neutron v2 API that nova invokes in order to provision the > neutron > > services. The vif driver is responsible for plugging in an instance onto the > > networking setup that neutron has created on the host. > > > > > > > > Normally, one will invoke "nova boot" api with the -nic options to specify > > the nic with which the instance will be connected to the network. It > > currently allows net-id, fixed ip and/or port-id to be specified for the > > option. However, it doesn't allow one to specify special networking > > requirements for the instance. Thanks to the nova pci-passthrough work, > one > > can specify PCI passthrough device(s) in the nova flavor. But it doesn't > > provide means to tie up these PCI devices in the case of ethernet > adpators > > with networking services. Therefore the idea is actually simple as > indicated > > by the blueprint titles, to provide means to tie up SRIOV devices with > > neutron services. A work flow would roughly look like this for 'nova > boot': > > > > > > > > -- Specifies networking requirements in the -nic option. > Specifically > > for SRIOV, allow the following to be specified in addition to the existing > > required information: > > > > . PCI alias > > > > . direct pci-passthrough/macvtap > > > > . port profileid that is compliant with 802.1Qbh > > > > > > > > The above information is optional. In the absence of them, the > > existing behavior remains. > > > > > > > > -- if special networking requirements exist, Nova api creates PCI > > requests in the nova instance type for scheduling purpose > > > > > > > > -- Nova scheduler schedules the instance based on the requested > flavor > > plus the PCI requests that are created for networking. > > > > > > > > -- Nova compute invokes neutron services with PCI passthrough > > information if any > > > > > > > > -- Neutron performs its normal operations based on the request, > such > > as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it > > should validate the information such as profileid, and stores them in its > > db. It's also possible to associate a port profileid with a neutron network > > so that port profileid becomes optional in the -nic option. Neutron > returns > > nova the port information, especially for PCI passthrough related > > information in the port binding object. Currently, the port binding object > > contains the following information: > > > > binding:vif_type > > > > binding:host_id > > > > binding:profile > > > > binding:capabilities > > > > > > > > -- nova constructs the domain xml and plug in the instance by > calling > > the vif driver. The vif driver can build up the interface xml based on the > > port binding information. > > > > > > > > > > > > > > > > > > > > The blueprints you registered make sense. On Nova side, there is a need > to > > bind between requested virtual network and PCI device/interface to be > > allocated as vNIC. > > > > On the Neutron side, there is a need to support networking > configuration of > > the vNIC. Neutron should be able to identify the PCI device/macvtap > > interface in order to apply configuration. I think it makes sense to > provide > > neutron integration via dedicated Modular Layer 2 Mechanism Driver to > allow > > PCI pass-through vNIC support along with other networking technologies. > > > > > > > > I haven't sorted through this yet. A neutron port could be associated > with a > > PCI device or not, which is a common feature, IMHO. However, a ML2 > driver > > may be needed specific to a particular SRIOV technology. > > > > > > > > > > > > During the Havana Release, we introduced Mellanox Neutron plugin that > > enables networking via SRIOV pass-through devices or macvtap > interfaces. > > > > We want to integrate our solution with PCI pass-through Nova support. > I > > will be glad to share more details if you are interested. > > > > > > > > > > > > Good to know that you already have a SRIOV implementation. I found out > some > > information online about the mlnx plugin, but need more time to get to > know > > it better. And certainly I'm interested in knowing its details. > > > > > > > > The PCI pass-through networking support is planned to be discussed > during > > the summit: http://summit.openstack.org/cfp/details/129. I think it's > worth > > to drill down into more detailed proposal and present it during the > summit, > > especially since it impacts both nova and neutron projects. > > > > > > > > I agree. Maybe we can steal some time in that discussion. > > > > > > > > Would you be interested in collaboration on this effort? Would you be > > interested to exchange more emails or set an IRC/WebEx meeting during > this > > week before the summit? > > > > > > > > Sure. If folks want to discuss it before the summit, we can schedule a > webex > > later this week. Or otherwise, we can continue the discussion with email. > > > > > > > > > > > > > > > > Regards, > > > > Irena > > > > > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > > Sent: Friday, October 25, 2013 11:16 PM > > To: prashant.upadhy...@aricent.com; Irena Berezovsky; > > yunhong.ji...@intel.com; chris.frie...@windriver.com; > yongli...@intel.com > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle > Mestery > > (kmestery); Sandhya Dasu (sadasu) > > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network > > support > > > > > > > > Hi Irena, > > > > > > > > This is Robert Li from Cisco Systems. Recently, I was tasked to investigate > > such support for Cisco's systems that support VM-FEX, which is a SRIOV > > technology supporting 802-1Qbh. I was able to bring up nova instances > with > > SRIOV interfaces, and establish networking in between the instances that > > employes the SRIOV interfaces. Certainly, this was accomplished with > hacking > > and some manual intervention. Based on this experience and my study > with the > > two existing nova pci-passthrough blueprints that have been > implemented and > > committed into Havana > > (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and > > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt), I > > registered a couple of blueprints (one on Nova side, the other on the > > Neutron side): > > > > > > > > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov > > > > https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov > > > > > > > > in order to address SRIOV support in openstack. > > > > > > > > Please take a look at them and see if they make sense, and let me know > any > > comments and questions. We can also discuss this in the summit, I > suppose. > > > > > > > > I noticed that there is another thread on this topic, so copy those folks > > from that thread as well. > > > > > > > > thanks, > > > > Robert > > > > > > > > On 10/16/13 4:32 PM, "Irena Berezovsky" <ire...@mellanox.com> > wrote: > > > > > > > > Hi, > > > > As one of the next steps for PCI pass-through I would like to discuss is > the > > support for PCI pass-through vNIC. > > > > While nova takes care of PCI pass-through device resources > management and > > VIF settings, neutron should manage their networking configuration. > > > > I would like to register asummit proposal to discuss the support for PCI > > pass-through networking. > > > > I am not sure what would be the right topic to discuss the PCI > pass-through > > networking, since it involve both nova and neutron. > > > > There is already a session registered by Yongli on nova topic to discuss > the > > PCI pass-through next steps. > > > > I think PCI pass-through networking is quite a big topic and it worth to > > have a separate discussion. > > > > Is there any other people who are interested to discuss it and share > their > > thoughts and experience? > > > > > > > > Regards, > > > > Irena > > > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
