On 18 September 2015 at 08:27, Russell Bryant <rbry...@redhat.com> wrote: > I mentioned in yesterday's OVN meeting that I was finally to the point > where I could start testing Neutron security groups with OVN ACLs but > hadn't done so before the meeting. I worked on it yesterday afternoon > and here's what happened. > > I'm using OVN from Justin's ovn-acl branch. I'm using my > security-groups branch for the Neutron plugin (networking-ovn). > > This test is on Ubuntu 14.04. I do most dev on Fedora 22, but the > kernel modules didn't build there (kernel too new). I used Ubuntu 14.04 > because I thought I remembered that being something Justin had tested > with. I'd actually like to clarify that point so let me know what you > all are using successfully. I was going to switch to CentOS 7 next.
Yes, I most commonly test with Ubuntu14.04. YMMV, when the userspace is reviewed I plan to work on the kernel backport and get it to work on a variety of kernel versions. > The test environment is a simple single hypervisor environment. I > created a Neutron port and booted a VM using this port. From there I > can add/remove/update security groups on this port and see what happens > to the traffic. > > With no security group, things work as usual. I can ping and ssh to the VM. > > I have a security group named "default", which has the default rules > plus two other rules that are supposed to allow me to ping and ssh to > the VM. > >> $ neutron security-group-list >> +--------------------------------------+---------+----------------------------------------------------------------------+ >> | id | name | security_group_rules >> | >> +--------------------------------------+---------+----------------------------------------------------------------------+ >> | 2bdabbcf-cde5-47fb-8a16-f5772545accc | default | egress, IPv4 >> | >> | | | egress, IPv6 >> | > > Allow all outbound traffic from the VM. > >> | | | ingress, IPv4, 22/tcp, >> remote_ip_prefix: 0.0.0.0/0 | >> | | | ingress, IPv4, icmp, >> remote_ip_prefix: 0.0.0.0/0 | > > Allow icmp and ssh from any source. > >> | | | ingress, IPv4, >> remote_group_id: 2bdabbcf-cde5-47fb-8a16-f5772545accc | >> | | | ingress, IPv6, >> remote_group_id: 2bdabbcf-cde5-47fb-8a16-f5772545accc | >> +--------------------------------------+---------+----------------------------------------------------------------------+ > > Allow all traffic from other ports on the same security group. > > > With that configuration, here are the OVN ACLs, the logical flow table > row for ACLs, and the OpenFlow flows for the egress pipeline ACLs: > >> $ ovn-nbctl acl-list e7936efd-1cf3-4198-bff5-cd0abf1164c8 >> >> from-lport 1002 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4) >> allow-related >> from-lport 1002 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip6) >> allow-related >> from-lport 1001 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip) >> drop >> to-lport 1002 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 >> && ip4.dst == 0.0.0.0/0 && icmp4) allow-related >> to-lport 1002 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 >> && ip4.dst == 0.0.0.0/0 && tcp && tcp.dst >= 22 && tcp.dst <= 22) >> allow-related >> to-lport 1001 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip) >> drop > > >> $ ovn-sbctl lflow-list >> >> Datapath: adec5d12-20a8-4a33-9de9-0fed21109118 Pipeline: ingress >> table=1( pre_acl), priority=100, match=(ip), action=(ct_next;) >> table=1( pre_acl), priority= 0, match=(1), action=(next;) >> table=2( acl), priority=65535, match=(!ct.est && ct.rel && !ct.new && >> !ct.inv), action=(next;) >> table=2( acl), priority=65535, match=(ct.est && !ct.rel && !ct.new && >> !ct.inv), action=(next;) >> table=2( acl), priority=65535, match=(ct.inv), action=(drop;) >> table=2( acl), priority=1002, match=(ct.new && inport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4), action=(ct_commit; next;) >> table=2( acl), priority=1002, match=(ct.new && inport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip6), action=(ct_commit; next;) >> table=2( acl), priority=1001, match=(inport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip), action=(drop;) >> table=2( acl), priority= 0, match=(1), action=(next;) >> >> Datapath: adec5d12-20a8-4a33-9de9-0fed21109118 Pipeline: egress >> table=0( pre_acl), priority=100, match=(ip), action=(ct_next;) >> table=0( pre_acl), priority= 0, match=(1), action=(next;) >> table=1( acl), priority=65535, match=(!ct.est && ct.rel && !ct.new && >> !ct.inv), action=(next;) >> table=1( acl), priority=65535, match=(ct.est && !ct.rel && !ct.new && >> !ct.inv), action=(next;) >> table=1( acl), priority=65535, match=(ct.inv), action=(drop;) >> table=1( acl), priority=1002, match=(ct.new && outport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 && ip4.dst == 0.0.0.0/0 && >> icmp4), action=(ct_commit; next;) >> table=1( acl), priority=1002, match=(ct.new && outport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 && ip4.dst == 0.0.0.0/0 && tcp >> && tcp.dst >= 22 && tcp.dst <= 22), action=(ct_commit; next;) >> table=1( acl), priority=1001, match=(outport == >> "fbda6395-71fe-4eb5-abda-531bddf479ba" && ip), action=(drop;) >> table=1( acl), priority= 0, match=(1), action=(next;) > > >> flows for egress pipeline ACLs >> >> cookie=0x0, duration=31.955s, table=48, n_packets=15, n_bytes=1650, >> priority=100,ipv6,metadata=0x1 >> actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[]) >> cookie=0x0, duration=31.952s, table=48, n_packets=2, n_bytes=196, >> priority=100,ip,metadata=0x1 >> actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[]) >> cookie=0x0, duration=31.955s, table=48, n_packets=2, n_bytes=84, >> priority=0,metadata=0x1 actions=resubmit(,49) >> cookie=0x0, duration=31.955s, table=49, n_packets=0, n_bytes=0, >> priority=65535,ct_state=-new+est-rel-inv+trk,metadata=0x1 >> actions=resubmit(,50) >> cookie=0x0, duration=31.952s, table=49, n_packets=0, n_bytes=0, >> priority=65535,ct_state=-new-est+rel-inv+trk,metadata=0x1 >> actions=resubmit(,50) >> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98, >> priority=65535,ct_state=+inv+trk,metadata=0x1 actions=drop >> cookie=0x0, duration=31.955s, table=49, n_packets=0, n_bytes=0, >> priority=1002,ct_state=+new+trk,tcp,reg7=0x4,metadata=0x1,tp_dst=22 >> actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50) >> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98, >> priority=1002,ct_state=+new+trk,icmp,reg7=0x4,metadata=0x1 >> actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50) >> cookie=0x0, duration=31.955s, table=49, n_packets=5, n_bytes=550, >> priority=1001,ipv6,reg7=0x4,metadata=0x1 actions=drop >> cookie=0x0, duration=31.952s, table=49, n_packets=0, n_bytes=0, >> priority=1001,ip,reg7=0x4,metadata=0x1 actions=drop >> cookie=0x0, duration=31.955s, table=49, n_packets=12, n_bytes=1184, >> priority=0,metadata=0x1 actions=resubmit(,50) > > > When I send a single ping from the hypervisor to the VM, I get no > response. Looking at the flows, it seems the ping makes it to the VM, > but the response is dropped by an ACL flow as invalid. > > After the ping, it shows up in conntrack like so: > >> $ sudo conntrack -L >> ... >> icmp 1 21 src=172.24.4.1 dst=10.0.0.4 type=8 code=0 id=9730 src=10.0.0.4 >> dst=172.24.4.1 type=0 code=0 id=9730 mark=0 zone=3 use=1 >> icmp 1 21 src=172.24.4.1 dst=10.0.0.4 type=8 code=0 id=9730 [UNREPLIED] >> src=10.0.0.4 dst=172.24.4.1 type=0 code=0 id=9730 mark=0 use=1 > > The flows that I believe shows what happens are: > >> cookie=0x0, duration=31.952s, table=48, n_packets=2, n_bytes=196, >> priority=100,ip,metadata=0x1 >> actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[]) >> >> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98, >> priority=65535,ct_state=+inv+trk,metadata=0x1 actions=drop >> >> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98, >> priority=1002,ct_state=+new+trk,icmp,reg7=0x4,metadata=0x1 >> actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50) > > > I did a similar test using ssh. After the ssh attempt, conntrack shows: > >> sudo conntracl -L >> tcp 6 27 SYN_RECV src=172.24.4.1 dst=10.0.0.4 sport=47784 dport=22 >> src=10.0.0.4 dst=172.24.4.1 sport=22 dport=47784 mark=0 zone=3 use=1 >> tcp 6 87 SYN_SENT src=172.24.4.1 dst=10.0.0.4 sport=47784 dport=22 >> [UNREPLIED] src=10.0.0.4 dst=172.24.4.1 sport=22 dport=47784 mark=0 use=1 > > It looks like the SYN makes it to the VM, but the return traffic is > dropped as invalid. > > > Any thoughts on this? Is there anything obviously wrong with what I've > done? Are there more details I should gather? It's a little hard to verify your analysis with only the the egress pipeline flows, but looking at the conntrack output it seems about right. There's a bit of a trick here, and I'm guessing that the OVN flows as generated today aren't quite handling this right. When the conntrack module is loaded, and you send packets from the local stack (eg ping from hypervisor -> VM), linux will piggyback conntrack information onto the packet. This is attached during linux stack processing between the application sending the packet and the packet arriving at OVS. The packet's associated conntrack state is in zone=0 if you haven't otherwise configured the local IPtables rules. For the below explanation, I'm going to assume that the zone associated with your hypervisor VIF is zone=2, and the zone associated with your VM is zone=3. It looks like OVN trusts that this conntrack information is correct when the packet arrives (zone=0), and rather than going through the from-lport conntrack processing (which should commit to zone=2), it skips this step and goes straight through to the to-lport processing. In to-lport, it goes through conntrack for zone=3 and commits the connection there. So we have two connections: One in zone=0 and one in zone=3. When the reply comes in the reverse direction, coming from the VM there is no conntrack information associated. OVN correctly passes this to conntrack for zone=3, which says "yep, this is the reply for that connection we saw before". The rules continue processing to go to the to-lport for the original VIF. This time, it submits to zone=2. However, we never committed the connection to zone=2 so conntrack marks it as invalid. Looking at the OVN rules, I think that this should be logically handled correctly so probably it just needs some adjustments on the translation to OpenFlow. _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
