> -----Original Message----- > From: Isaku Yamahata [mailto:isaku.yamah...@gmail.com] > Sent: Tuesday, October 29, 2013 8:24 PM > To: OpenStack Development Mailing List (not for usage questions) > Cc: isaku.yamah...@gmail.com; Itzik Brown > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network > support > > Hi Yunhong. > > On Tue, Oct 29, 2013 at 08:22:40PM +0000, > "Jiang, Yunhong" <yunhong.ji...@intel.com> wrote: > > > > * describe resource external to nova that is attached to VM in the API > > > (block device mapping and/or vif references) > > > * ideally the nova scheduler needs to be aware of the local capacity, > > > and how that relates to the above information (relates to the cross > > > service scheduling issues) > > > > I think this possibly a bit different. For volume, it's sure managed by > Cinder, but for PCI devices, currently > > It ;s managed by nova. So we possibly need nova to translate the > information (possibly before nova scheduler). > > > > > * state of the device should be stored by Neutron/Cinder > > > (attached/detached, capacity, IP, etc), but still exposed to the > > > "scheduler" > > > > I'm not sure if we can keep the state of the device in Neutron. Currently > nova manage all PCI devices. > > Yes, with the current implementation, nova manages PCI devices and it > works. > That's great. It will remain so in Icehouse cycle (maybe also J?). > > But how about long term direction? > Neutron should know/manage such network related resources on > compute nodes?
So you mean the PCI device management will be spited between Nova and Neutron? For example, non-NIC device owned by nova and NIC device owned by neutron? There have been so many discussion of the scheduler enhancement, like https://etherpad.openstack.org/p/grizzly-split-out-scheduling , so possibly that's the right direction? Let's wait for the summit discussion. > The implementation in Nova will be moved into Neutron like what Cinder > did? > any opinions/thoughts? > It seems that not so many Neutron developers are interested in PCI > passthrough at the moment, though. > > There are use cases for this, I think. > For example, some compute nodes use OVS plugin, another nodes LB > plugin. > (Right now it may not possible easily, but it will be with ML2 plugin and > mechanism driver). User wants their VMs to run on nodes with OVS plugin > for > some reason(e.g. performance difference). > Such usage would be handled similarly. > > Thanks, > --- > Isaku Yamahata > > > > > > Thanks > > --jyh > > > > > > > * connection params get given to Nova from Neutron/Cinder > > > * nova still has the vif driver or volume driver to make the final > connection > > > * the disk should be formatted/expanded, and network info injected in > > > the same way as before (cloud-init, config drive, DHCP, etc) > > > > > > John > > > > > > On 29 October 2013 10:17, Irena Berezovsky > <ire...@mellanox.com> > > > wrote: > > > > Hi Jiang, Robert, > > > > > > > > IRC meeting option works for me. > > > > > > > > If I understand your question below, you are looking for a way to tie > up > > > > between requested virtual network(s) and requested PCI device(s). > The > > > way we > > > > did it in our solution is to map a provider:physical_network to an > > > > interface that represents the Physical Function. Every virtual > network is > > > > bound to the provider:physical_network, so the PCI device should > be > > > > allocated based on this mapping. We can map a PCI alias to the > > > > provider:physical_network. > > > > > > > > > > > > > > > > Another topic to discuss is where the mapping between neutron > port > > > and PCI > > > > device should be managed. One way to solve it, is to propagate the > > > allocated > > > > PCI device details to neutron on port creation. > > > > > > > > In case there is no qbg/qbh support, VF networking configuration > > > should be > > > > applied locally on the Host. > > > > > > > > The question is when and how to apply networking configuration on > the > > > PCI > > > > device? > > > > > > > > We see the following options: > > > > > > > > * it can be done on port creation. > > > > > > > > * It can be done when nova VIF driver is called for vNIC > > > plugging. > > > > This will require to have all networking configuration available to > the > > > VIF > > > > driver or send request to the neutron server to obtain it. > > > > > > > > * It can be done by having a dedicated L2 neutron agent > on > > > each > > > > Host that scans for allocated PCI devices and then retrieves > networking > > > > configuration from the server and configures the device. The agent > will > > > be > > > > also responsible for managing update requests coming from the > neutron > > > > server. > > > > > > > > > > > > > > > > For macvtap vNIC type assignment, the networking configuration can > be > > > > applied by a dedicated L2 neutron agent. > > > > > > > > > > > > > > > > BR, > > > > > > > > Irena > > > > > > > > > > > > > > > > From: Jiang, Yunhong [mailto:yunhong.ji...@intel.com] > > > > Sent: Tuesday, October 29, 2013 9:04 AM > > > > > > > > > > > > To: Robert Li (baoli); Irena Berezovsky; > > > prashant.upadhy...@aricent.com; > > > > chris.frie...@windriver.com; He, Yongli; Itzik Brown > > > > > > > > > > > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); > Kyle > > > Mestery > > > > (kmestery); Sandhya Dasu (sadasu) > > > > Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through > network > > > > support > > > > > > > > > > > > > > > > Robert, is it possible to have a IRC meeting? I'd prefer to IRC meeting > > > > because it's more openstack style and also can keep the minutes > > > clearly. > > > > > > > > > > > > > > > > To your flow, can you give more detailed example. For example, I can > > > > consider user specify the instance with -nic option specify a network > id, > > > > and then how nova device the requirement to the PCI device? I > assume > > > the > > > > network id should define the switches that the device can connect > to , > > > but > > > > how is that information translated to the PCI property requirement? > Will > > > > this translation happen before the nova scheduler make host > decision? > > > > > > > > > > > > > > > > Thanks > > > > > > > > --jyh > > > > > > > > > > > > > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > > > > Sent: Monday, October 28, 2013 12:22 PM > > > > To: Irena Berezovsky; prashant.upadhy...@aricent.com; Jiang, > Yunhong; > > > > chris.frie...@windriver.com; He, Yongli; Itzik Brown > > > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); > Kyle > > > Mestery > > > > (kmestery); Sandhya Dasu (sadasu) > > > > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through > network > > > > support > > > > > > > > > > > > > > > > Hi Irena, > > > > > > > > > > > > > > > > Thank you very much for your comments. See inline. > > > > > > > > > > > > > > > > --Robert > > > > > > > > > > > > > > > > On 10/27/13 3:48 AM, "Irena Berezovsky" <ire...@mellanox.com> > > > wrote: > > > > > > > > > > > > > > > > Hi Robert, > > > > > > > > Thank you very much for sharing the information regarding your > efforts. > > > Can > > > > you please share your idea of the end to end flow? How do you > suggest > > > to > > > > bind Nova and Neutron? > > > > > > > > > > > > > > > > The end to end flow is actually encompassed in the blueprints in a > > > nutshell. > > > > I will reiterate it in below. The binding between Nova and Neutron > > > occurs > > > > with the neutron v2 API that nova invokes in order to provision the > > > neutron > > > > services. The vif driver is responsible for plugging in an instance onto > the > > > > networking setup that neutron has created on the host. > > > > > > > > > > > > > > > > Normally, one will invoke "nova boot" api with the -nic options to > specify > > > > the nic with which the instance will be connected to the network. It > > > > currently allows net-id, fixed ip and/or port-id to be specified for the > > > > option. However, it doesn't allow one to specify special networking > > > > requirements for the instance. Thanks to the nova pci-passthrough > work, > > > one > > > > can specify PCI passthrough device(s) in the nova flavor. But it > doesn't > > > > provide means to tie up these PCI devices in the case of ethernet > > > adpators > > > > with networking services. Therefore the idea is actually simple as > > > indicated > > > > by the blueprint titles, to provide means to tie up SRIOV devices with > > > > neutron services. A work flow would roughly look like this for 'nova > > > boot': > > > > > > > > > > > > > > > > -- Specifies networking requirements in the -nic option. > > > Specifically > > > > for SRIOV, allow the following to be specified in addition to the > existing > > > > required information: > > > > > > > > . PCI alias > > > > > > > > . direct pci-passthrough/macvtap > > > > > > > > . port profileid that is compliant with 802.1Qbh > > > > > > > > > > > > > > > > The above information is optional. In the absence of them, > the > > > > existing behavior remains. > > > > > > > > > > > > > > > > -- if special networking requirements exist, Nova api creates > PCI > > > > requests in the nova instance type for scheduling purpose > > > > > > > > > > > > > > > > -- Nova scheduler schedules the instance based on the > requested > > > flavor > > > > plus the PCI requests that are created for networking. > > > > > > > > > > > > > > > > -- Nova compute invokes neutron services with PCI > passthrough > > > > information if any > > > > > > > > > > > > > > > > -- Neutron performs its normal operations based on the > request, > > > such > > > > as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it > > > > should validate the information such as profileid, and stores them in > its > > > > db. It's also possible to associate a port profileid with a neutron > network > > > > so that port profileid becomes optional in the -nic option. Neutron > > > returns > > > > nova the port information, especially for PCI passthrough related > > > > information in the port binding object. Currently, the port binding > object > > > > contains the following information: > > > > > > > > binding:vif_type > > > > > > > > binding:host_id > > > > > > > > binding:profile > > > > > > > > binding:capabilities > > > > > > > > > > > > > > > > -- nova constructs the domain xml and plug in the instance by > > > calling > > > > the vif driver. The vif driver can build up the interface xml based on > the > > > > port binding information. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The blueprints you registered make sense. On Nova side, there is a > need > > > to > > > > bind between requested virtual network and PCI device/interface to > be > > > > allocated as vNIC. > > > > > > > > On the Neutron side, there is a need to support networking > > > configuration of > > > > the vNIC. Neutron should be able to identify the PCI device/macvtap > > > > interface in order to apply configuration. I think it makes sense to > > > provide > > > > neutron integration via dedicated Modular Layer 2 Mechanism > Driver to > > > allow > > > > PCI pass-through vNIC support along with other networking > technologies. > > > > > > > > > > > > > > > > I haven't sorted through this yet. A neutron port could be associated > > > with a > > > > PCI device or not, which is a common feature, IMHO. However, a > ML2 > > > driver > > > > may be needed specific to a particular SRIOV technology. > > > > > > > > > > > > > > > > > > > > > > > > During the Havana Release, we introduced Mellanox Neutron plugin > that > > > > enables networking via SRIOV pass-through devices or macvtap > > > interfaces. > > > > > > > > We want to integrate our solution with PCI pass-through Nova > support. > > > I > > > > will be glad to share more details if you are interested. > > > > > > > > > > > > > > > > > > > > > > > > Good to know that you already have a SRIOV implementation. I found > out > > > some > > > > information online about the mlnx plugin, but need more time to get > to > > > know > > > > it better. And certainly I'm interested in knowing its details. > > > > > > > > > > > > > > > > The PCI pass-through networking support is planned to be discussed > > > during > > > > the summit: http://summit.openstack.org/cfp/details/129. I think it's > > > worth > > > > to drill down into more detailed proposal and present it during the > > > summit, > > > > especially since it impacts both nova and neutron projects. > > > > > > > > > > > > > > > > I agree. Maybe we can steal some time in that discussion. > > > > > > > > > > > > > > > > Would you be interested in collaboration on this effort? Would you > be > > > > interested to exchange more emails or set an IRC/WebEx meeting > during > > > this > > > > week before the summit? > > > > > > > > > > > > > > > > Sure. If folks want to discuss it before the summit, we can schedule a > > > webex > > > > later this week. Or otherwise, we can continue the discussion with > email. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > Irena > > > > > > > > > > > > > > > > From: Robert Li (baoli) [mailto:ba...@cisco.com] > > > > Sent: Friday, October 25, 2013 11:16 PM > > > > To: prashant.upadhy...@aricent.com; Irena Berezovsky; > > > > yunhong.ji...@intel.com; chris.frie...@windriver.com; > > > yongli...@intel.com > > > > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); > Kyle > > > Mestery > > > > (kmestery); Sandhya Dasu (sadasu) > > > > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through > network > > > > support > > > > > > > > > > > > > > > > Hi Irena, > > > > > > > > > > > > > > > > This is Robert Li from Cisco Systems. Recently, I was tasked to > investigate > > > > such support for Cisco's systems that support VM-FEX, which is a > SRIOV > > > > technology supporting 802-1Qbh. I was able to bring up nova > instances > > > with > > > > SRIOV interfaces, and establish networking in between the instances > that > > > > employes the SRIOV interfaces. Certainly, this was accomplished with > > > hacking > > > > and some manual intervention. Based on this experience and my > study > > > with the > > > > two existing nova pci-passthrough blueprints that have been > > > implemented and > > > > committed into Havana > > > > (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base > and > > > > > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt), I > > > > registered a couple of blueprints (one on Nova side, the other on the > > > > Neutron side): > > > > > > > > > > > > > > > > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov > > > > > > > > > https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov > > > > > > > > > > > > > > > > in order to address SRIOV support in openstack. > > > > > > > > > > > > > > > > Please take a look at them and see if they make sense, and let me > know > > > any > > > > comments and questions. We can also discuss this in the summit, I > > > suppose. > > > > > > > > > > > > > > > > I noticed that there is another thread on this topic, so copy those > folks > > > > from that thread as well. > > > > > > > > > > > > > > > > thanks, > > > > > > > > Robert > > > > > > > > > > > > > > > > On 10/16/13 4:32 PM, "Irena Berezovsky" <ire...@mellanox.com> > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > As one of the next steps for PCI pass-through I would like to discuss is > > > the > > > > support for PCI pass-through vNIC. > > > > > > > > While nova takes care of PCI pass-through device resources > > > management and > > > > VIF settings, neutron should manage their networking configuration. > > > > > > > > I would like to register asummit proposal to discuss the support for > PCI > > > > pass-through networking. > > > > > > > > I am not sure what would be the right topic to discuss the PCI > > > pass-through > > > > networking, since it involve both nova and neutron. > > > > > > > > There is already a session registered by Yongli on nova topic to > discuss > > > the > > > > PCI pass-through next steps. > > > > > > > > I think PCI pass-through networking is quite a big topic and it worth to > > > > have a separate discussion. > > > > > > > > Is there any other people who are interested to discuss it and share > > > their > > > > thoughts and experience? > > > > > > > > > > > > > > > > Regards, > > > > > > > > Irena > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > OpenStack-dev mailing list > > > > OpenStack-dev@lists.openstack.org > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- > Isaku Yamahata <isaku.yamah...@gmail.com> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
