Right, my bad, Nova only did this a long time ago with nova-network network types... Surely it's handled by Neutron nowadays. In my Fuel 9.0-based lab neutron-openvswitch-agent service is managing the rules. I can erase some VM rules, restart the agent (while nova-compute is stopped - to ensure it does not do anything) and they reappear. That's how normal OpenStack installation works:
root@node-3:~# iptables -S | grep 54321 -A neutron-openvswi-i7eeecb33-1 -p tcp -m tcp --dport 54321 -j RETURN root@node-3:~# service nova-compute stop nova-compute stop/waiting root@node-3:~# iptables -D neutron-openvswi-i7eeecb33-1 -p tcp -m tcp --dport 54321 -j RETURN root@node-3:~# restart neutron-openvswitch-agent neutron-openvswitch-agent start/running, process 12541 root@node-3:~# iptables -S | grep 54321 # rule not recreated yet root@node-3:~# iptables -S | grep 54321 # a few moments later it's there -A neutron-openvswi-i7eeecb33-1 -p tcp -m tcp --dport 54321 -j RETURN There are of course also default iptables rules which always get applied for every VM (by the same service). My firewall driver is the same: /etc/neutron/plugins/ml2/openvswitch_agent.ini: firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver On Wed, May 24, 2017 at 12:25 AM, duhongwei <duhong...@qiniu.com> wrote: > > Dmitry Sutyagin also replied, quote: > > *Afaik, iptables are set by Nova, and the driver is set via > firewall_driver option in nova.conf* > > After some investigation, I am thinking both Nova and Neutron provide > security_group implementation, while Nova operates on vm granularity and > Neuton operates on port. > > In my openstack environment, I see *firewall_driver = > nova.virt.firewall.NoopFirewallDriver* in nova.conf and *firewall_driver > = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver* > in ml2_conf.ini. So it seems I'm using Neutron's security_group > implementation. > > Back to my question, I tried to name qbr, qvo, qvb as qbr1234567890, ... > and, container side veth as veth0, qbr side veth as tap1234567890. As a > result, no iptables rules added, therefore packets from veth0 are dropped > on qbr1234567890. > > So, how would Neutron know that it has to add iptables rules for a device, > by device name prefix or what? What can I do to let it happen? > > Regards, > Dastan > > ------------------ Original ------------------ > *From: * "Kevin Benton"<ke...@benton.pub>; > *Date: * Wed, May 24, 2017 05:21 AM > *To: * "duhongwei"<duhong...@qiniu.com>; > *Cc: * "openstack"<openstack@lists.openstack.org>; "Vallachorum > Tyranorum"<ardeleandanflo...@gmail.com>; > *Subject: * Re: [Openstack] How to utilize Neutron independently with > veths > > Neutron sets up the iptables rules if you have security groups enabled and > the agent firewall is set to iptables_hybrid or > neutron.agent.linux.iptables_firewall:OVSHybridIptablesFirewallDriver . > > What are you naming your vNIC? The iptables rules setup by the agent match > specifically on 'tap' + port UUID prefix. So if the bridge is qvb1234567890 > then the vnic you plug into it needs to be named tap1234567890 . > > On Tue, May 23, 2017 at 12:15 AM, duhongwei <duhong...@qiniu.com> wrote: > >> >> Thanks Kevin! I've made a big step forward! >> >> Till now, I've successfully connect *vNIC* directly into *br-int *without >> *qbr*, *qvo*, and *qvb*. And, it works well. >> >> However, following your scripts (connect *vNIC* into *qbr*, then connect >> *qbr* into *br-int*) exposes another problem. In this scenario, *qbr* >> won't forward packets from *vNIC* to *br-int* (packets seem to be >> dropped on *qbr*). >> >> After some troubleshooting, it turns out to be *iptables* who drops >> packets on *qbr*. Reviewing the *FORWARD* chain in *filter* table, >> packets come from *vNIC* won't match any rule of *neutron-filter-top* >> and *neutron-openvswi-FORWARD* so that the default policy *DROP* applies. >> >> So, after setting up all these *qbr*, *qvo*, *qvb*, *vNIC*, it seems >> there're still some *iptables rules* missed. Question is, >> >> Who's adding this iptables rules? (Nova or Neutron?) How can I make it >> happen? >> >> Regards, >> Dastan >> >> >> ------------------ Original ------------------ >> *From: * "Kevin Benton"<ke...@benton.pub>; >> *Date: * Mon, May 22, 2017 10:47 PM >> *To: * "duhongwei"<duhong...@qiniu.com>; >> *Cc: * "openstack"<openstack@lists.openstack.org>; "Vallachorum >> Tyranorum"<ardeleandanflo...@gmail.com>; >> *Subject: * Re: [Openstack] How to utilize Neutron independently with >> veths >> >> Yes, the only thing that needs to use the correct MAC is whatever is >> actually sending traffic. >> >> On May 21, 2017 22:06, "duhongwei" <duhong...@qiniu.com> wrote: >> >>> >>> Thanks for your patient, Kevin. >>> >>> So *qvo *could be any veth whose mac address doesn't matter, but *veth/tap >>> *must have exact the same mac address as *port*, otherwise it will be >>> anti-spoofed. >>> >>> *qvo*'s attributes (external-ids) tell neutron which logical *port* >>> *qvo* is connecting, so neutron knows how to add flows to ovs *br-int *and >>> *br-tun*. >>> >>> Am I correct? >>> >>> Regards, >>> Dastan >>> >>> ------------------ Original ------------------ >>> *From: * "Kevin Benton"<ke...@benton.pub>; >>> *Date: * Sat, May 20, 2017 03:26 AM >>> *To: * "duhongwei"<duhong...@qiniu.com>; >>> *Cc: * "openstack"<openstack@lists.openstack.org>; "Vallachorum >>> Tyranorum"<ardeleandanflo...@gmail.com>; >>> *Subject: * Re: [Openstack] How to utilize Neutron independently with >>> veths >>> >>> >After all these, we create *veth/tap* (as vm/containers vNIC) and >>> plugin it into *qbr* then we're able to talk with other vms/containers >>> on the same network through *veth/tap*, am I understanding it right? >>> >>> Yes, this last step of creating a veth/tap is missing from my script >>> because I didn't need actual dataplane communication for the tests I was >>> doing. >>> >>> >1) isn't it necessary that *veth/tap*'s mac address same as neutron >>> *port*'s mac address? >>> >>> Yeah, if you attach something to qbr to behave like the VM interface, >>> you will need it to be using the mac address of the neutron port, or else >>> the neutron anti-spoofing rules will prevent it from communicating. >>> >>> >>> >2) after we plug *qvo* into ovs *br-int*, neutron just automatically >>> add flows into ovs bridge? >>> >>> Yes, the agent will receive to the new port event from ovs, retrieve >>> port details from the server and then setup the flows. >>> >>> On Fri, May 19, 2017 at 12:09 AM, duhongwei <duhong...@qiniu.com> wrote: >>> >>>> >>>> This script seems easy and cool! >>>> >>>> So first we have to create a logical neutron *port*, then create *qbr*, >>>> *qvo* and *qvb*, and plug *qvb* into *qbr*, finally plug *qvo* into >>>> ovs *br-int*. After all these, we create *veth/tap* (as vm/containers >>>> vNIC) and plugin it into *qbr* then we're able to talk with other >>>> vms/containers on the same network through *veth/tap*, am I >>>> understanding it right? >>>> >>>> Questions, >>>> >>>> 1) isn't it necessary that *veth/tap*'s mac address same as neutron >>>> *port*'s mac address? >>>> 2) after we plug *qvo* into ovs *br-int*, neutron just automatically >>>> add flows into ovs bridge? >>>> >>>> Regards, >>>> Dastan >>>> >>>> ------------------ Original ------------------ >>>> *From: * "Kevin Benton"<ke...@benton.pub>; >>>> *Date: * Sat, May 13, 2017 07:46 AM >>>> *To: * "duhongwei"<duhong...@qiniu.com>; >>>> *Cc: * "openstack"<openstack@lists.openstack.org>; "Vallachorum >>>> Tyranorum"<ardeleandanflo...@gmail.com>; >>>> *Subject: * Re: [Openstack] How to utilize Neutron independently with >>>> veths >>>> >>>> Nova is only responsible for creating the interface and plugging it >>>> into the OVS bridge. It's the neutron agent (or alternative neutron backend >>>> like OVN) responsible for setting up all of the flows. >>>> >>>> Here is a hacky script that I had used to create and delete a bunch of >>>> ports like Nova would that you can probably start with: >>>> http://paste.openstack.org/show/609478/ >>>> >>>> On Fri, May 12, 2017 at 4:25 AM, duhongwei <duhong...@qiniu.com> wrote: >>>> >>>>> >>>>> Thanks Kevin! >>>>> >>>>> I'll dig into neutron.agent.linux.interface to see how it works. >>>>> Before that, would you give me any previews about what steps should be >>>>> taken to add a veth to a existed Neutron network? >>>>> >>>>> Furthermore, is it Neutron who add a veth to ovs bridge or is it the >>>>> Neutron caller? (such as Nova) >>>>> >>>>> Who's adding flows to ovs bridge? Neutron or caller? >>>>> >>>>> Regards, >>>>> Dastan >>>>> >>>>> ------------------ Original ------------------ >>>>> *From: * "Kevin Benton"<ke...@benton.pub>; >>>>> *Date: * Fri, May 12, 2017 10:45 AM >>>>> *To: * "duhongwei"<duhong...@qiniu.com>; >>>>> *Cc: * "openstack"<openstack@lists.openstack.org>; "Vallachorum >>>>> Tyranorum"<ardeleandanflo...@gmail.com>; >>>>> *Subject: * Re: [Openstack] How to utilize Neutron independently with >>>>> veths >>>>> >>>>> You want to look in neutron.agent.linux.interface to see how things >>>>> are plugged into OVS. That's the module used by the L3 agent to plug into >>>>> OVS/linux bridge/etc. >>>>> >>>>> There is a well defined interface name format corresponding to the >>>>> port ID and the port ID, Mac address, and a couple of other things I can't >>>>> recall are set in ovsdb to help the agent identify the port as something >>>>> it >>>>> should care about. >>>>> >>>>> On May 9, 2017 04:49, "duhongwei" <duhong...@qiniu.com> wrote: >>>>> >>>>>> >>>>>> Thanks Dan! >>>>>> >>>>>> I have to write a customized CNI plugin for our product, so it's >>>>>> better if I know more operation details about how to interact with >>>>>> Neutron >>>>>> manually (consider myself as Nova). Therefore Kuryr is not my best option >>>>>> right now, but it's cool! >>>>>> >>>>>> Regards, >>>>>> Dastan >>>>>> >>>>>> ------------------ Original ------------------ >>>>>> *From: * "Vallachorum Tyranorum"<ardeleandanflo...@gmail.com>; >>>>>> *Date: * Tue, May 9, 2017 04:08 PM >>>>>> *To: * "duhongwei"<duhong...@qiniu.com>; "openstack"< >>>>>> openstack@lists.openstack.org>; >>>>>> *Subject: * Re: [Openstack] How to utilize Neutron independently >>>>>> with veths >>>>>> >>>>>> Hi, >>>>>> >>>>>> Please take a look at Kuryr <https://wiki.openstack.org/wiki/Kuryr>. >>>>>> Maybe this is what you are looking for. >>>>>> >>>>>> Dan. >>>>>> >>>>>> On Tue, May 9, 2017 at 10:17 AM duhongwei <duhong...@qiniu.com> >>>>>> wrote: >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> I'm new to OpenStack and currently interested in the Neutron part of >>>>>>> it. What I'm seeking is some advice about how to utilize Neutron >>>>>>> independently, to build a virtual network, for Docker containers maybe. >>>>>>> >>>>>>> Suppose I've already got Neutron and Keystone installed on >>>>>>> controller node and compute nodes. I guess the following steps are >>>>>>> required >>>>>>> to test a virtual network. >>>>>>> >>>>>>> 1) create a *network* >>>>>>> 2) create a *subnet* >>>>>>> 3) create two pairs of veths (each pair represents a vm) >>>>>>> *for each pair of them*: >>>>>>> 4) create a *port *for one end of the veth pair (passing veth's mac >>>>>>> address as a parameter) >>>>>>> 5) attach another end of the veth pair to ovs bridge >>>>>>> 6) ping from one veth pair to another >>>>>>> >>>>>>> The above is my general idea, don't know if it is correct and don't >>>>>>> know the operation details either. >>>>>>> Expecting your suggestions, any links are appreciated. >>>>>>> >>>>>>> Regards, >>>>>>> Dastan >>>>>>> _______________________________________________ >>>>>>> Mailing list: http://lists.openstack.org/cgi >>>>>>> -bin/mailman/listinfo/openstack >>>>>>> Post to : openstack@lists.openstack.org >>>>>>> Unsubscribe : http://lists.openstack.org/cgi >>>>>>> -bin/mailman/listinfo/openstack >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: http://lists.openstack.org/cgi >>>>>> -bin/mailman/listinfo/openstack >>>>>> Post to : openstack@lists.openstack.org >>>>>> Unsubscribe : http://lists.openstack.org/cgi >>>>>> -bin/mailman/listinfo/openstack >>>>>> >>>>>> >>>> >>> > -- Yours sincerely, Dmitrii Sutiagin Cloud Operations Engineer, Mirantis, Inc.
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
