On 21 Sep 2015 2:36 pm, "Russell Bryant" <rbry...@redhat.com> wrote:
>
> On 09/21/2015 02:50 PM, Joe Stringer wrote:
> > On 18 September 2015 at 08:27, Russell Bryant <rbry...@redhat.com>
wrote:
> >> I mentioned in yesterday's OVN meeting that I was finally to the point
> >> where I could start testing Neutron security groups with OVN ACLs but
> >> hadn't done so before the meeting. I worked on it yesterday afternoon
> >> and here's what happened.
> >>
> >> I'm using OVN from Justin's ovn-acl branch. I'm using my
> >> security-groups branch for the Neutron plugin (networking-ovn).
> >>
> >> This test is on Ubuntu 14.04. I do most dev on Fedora 22, but the
> >> kernel modules didn't build there (kernel too new). I used Ubuntu
14.04
> >> because I thought I remembered that being something Justin had tested
> >> with. I'd actually like to clarify that point so let me know what you
> >> all are using successfully. I was going to switch to CentOS 7 next.
> >
> > Yes, I most commonly test with Ubuntu14.04. YMMV, when the userspace
> > is reviewed I plan to work on the kernel backport and get it to work
> > on a variety of kernel versions.
> >
> >> The test environment is a simple single hypervisor environment. I
> >> created a Neutron port and booted a VM using this port. From there I
> >> can add/remove/update security groups on this port and see what happens
> >> to the traffic.
> >>
> >> With no security group, things work as usual. I can ping and ssh to
the VM.
> >>
> >> I have a security group named "default", which has the default rules
> >> plus two other rules that are supposed to allow me to ping and ssh to
> >> the VM.
> >>
> >>> $ neutron security-group-list
> >>>
+--------------------------------------+---------+----------------------------------------------------------------------+
> >>> | id | name |
security_group_rules |
> >>>
+--------------------------------------+---------+----------------------------------------------------------------------+
> >>> | 2bdabbcf-cde5-47fb-8a16-f5772545accc | default | egress, IPv4
|
> >>> | | | egress, IPv6
|
> >>
> >> Allow all outbound traffic from the VM.
> >>
> >>> | | | ingress, IPv4,
22/tcp, remote_ip_prefix: 0.0.0.0/0 |
> >>> | | | ingress, IPv4,
icmp, remote_ip_prefix: 0.0.0.0/0 |
> >>
> >> Allow icmp and ssh from any source.
> >>
> >>> | | | ingress, IPv4,
remote_group_id: 2bdabbcf-cde5-47fb-8a16-f5772545accc |
> >>> | | | ingress, IPv6,
remote_group_id: 2bdabbcf-cde5-47fb-8a16-f5772545accc |
> >>>
+--------------------------------------+---------+----------------------------------------------------------------------+
> >>
> >> Allow all traffic from other ports on the same security group.
> >>
> >>
> >> With that configuration, here are the OVN ACLs, the logical flow table
> >> row for ACLs, and the OpenFlow flows for the egress pipeline ACLs:
> >>
> >>> $ ovn-nbctl acl-list e7936efd-1cf3-4198-bff5-cd0abf1164c8
> >>>
> >>> from-lport 1002 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" &&
ip4) allow-related
> >>> from-lport 1002 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" &&
ip6) allow-related
> >>> from-lport 1001 (inport == "fbda6395-71fe-4eb5-abda-531bddf479ba" &&
ip) drop
> >>> to-lport 1002 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba"
&& ip4 && ip4.dst == 0.0.0.0/0 && icmp4) allow-related
> >>> to-lport 1002 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba"
&& ip4 && ip4.dst == 0.0.0.0/0 && tcp && tcp.dst >= 22 && tcp.dst <= 22)
allow-related
> >>> to-lport 1001 (outport == "fbda6395-71fe-4eb5-abda-531bddf479ba"
&& ip) drop
> >>
> >>
> >>> $ ovn-sbctl lflow-list
> >>>
> >>> Datapath: adec5d12-20a8-4a33-9de9-0fed21109118 Pipeline: ingress
> >>> table=1( pre_acl), priority=100, match=(ip), action=(ct_next;)
> >>> table=1( pre_acl), priority= 0, match=(1), action=(next;)
> >>> table=2( acl), priority=65535, match=(!ct.est && ct.rel &&
!ct.new && !ct.inv), action=(next;)
> >>> table=2( acl), priority=65535, match=(ct.est && !ct.rel &&
!ct.new && !ct.inv), action=(next;)
> >>> table=2( acl), priority=65535, match=(ct.inv), action=(drop;)
> >>> table=2( acl), priority=1002, match=(ct.new && inport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4), action=(ct_commit; next;)
> >>> table=2( acl), priority=1002, match=(ct.new && inport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip6), action=(ct_commit; next;)
> >>> table=2( acl), priority=1001, match=(inport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip), action=(drop;)
> >>> table=2( acl), priority= 0, match=(1), action=(next;)
> >>>
> >>> Datapath: adec5d12-20a8-4a33-9de9-0fed21109118 Pipeline: egress
> >>> table=0( pre_acl), priority=100, match=(ip), action=(ct_next;)
> >>> table=0( pre_acl), priority= 0, match=(1), action=(next;)
> >>> table=1( acl), priority=65535, match=(!ct.est && ct.rel &&
!ct.new && !ct.inv), action=(next;)
> >>> table=1( acl), priority=65535, match=(ct.est && !ct.rel &&
!ct.new && !ct.inv), action=(next;)
> >>> table=1( acl), priority=65535, match=(ct.inv), action=(drop;)
> >>> table=1( acl), priority=1002, match=(ct.new && outport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 && ip4.dst == 0.0.0.0/0 &&
icmp4), action=(ct_commit; next;)
> >>> table=1( acl), priority=1002, match=(ct.new && outport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip4 && ip4.dst == 0.0.0.0/0 &&
tcp && tcp.dst >= 22 && tcp.dst <= 22), action=(ct_commit; next;)
> >>> table=1( acl), priority=1001, match=(outport ==
"fbda6395-71fe-4eb5-abda-531bddf479ba" && ip), action=(drop;)
> >>> table=1( acl), priority= 0, match=(1), action=(next;)
> >>
> >>
> >>> flows for egress pipeline ACLs
> >>>
> >>> cookie=0x0, duration=31.955s, table=48, n_packets=15, n_bytes=1650,
priority=100,ipv6,metadata=0x1
actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[])
> >>> cookie=0x0, duration=31.952s, table=48, n_packets=2, n_bytes=196,
priority=100,ip,metadata=0x1
actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[])
> >>> cookie=0x0, duration=31.955s, table=48, n_packets=2, n_bytes=84,
priority=0,metadata=0x1 actions=resubmit(,49)
> >>> cookie=0x0, duration=31.955s, table=49, n_packets=0, n_bytes=0,
priority=65535,ct_state=-new+est-rel-inv+trk,metadata=0x1
actions=resubmit(,50)
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=0, n_bytes=0,
priority=65535,ct_state=-new-est+rel-inv+trk,metadata=0x1
actions=resubmit(,50)
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98,
priority=65535,ct_state=+inv+trk,metadata=0x1 actions=drop
> >>> cookie=0x0, duration=31.955s, table=49, n_packets=0, n_bytes=0,
priority=1002,ct_state=+new+trk,tcp,reg7=0x4,metadata=0x1,tp_dst=22
actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50)
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98,
priority=1002,ct_state=+new+trk,icmp,reg7=0x4,metadata=0x1
actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50)
> >>> cookie=0x0, duration=31.955s, table=49, n_packets=5, n_bytes=550,
priority=1001,ipv6,reg7=0x4,metadata=0x1 actions=drop
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=0, n_bytes=0,
priority=1001,ip,reg7=0x4,metadata=0x1 actions=drop
> >>> cookie=0x0, duration=31.955s, table=49, n_packets=12, n_bytes=1184,
priority=0,metadata=0x1 actions=resubmit(,50)
> >>
> >>
> >> When I send a single ping from the hypervisor to the VM, I get no
> >> response. Looking at the flows, it seems the ping makes it to the VM,
> >> but the response is dropped by an ACL flow as invalid.
> >>
> >> After the ping, it shows up in conntrack like so:
> >>
> >>> $ sudo conntrack -L
> >>> ...
> >>> icmp 1 21 src=172.24.4.1 dst=10.0.0.4 type=8 code=0 id=9730
src=10.0.0.4 dst=172.24.4.1 type=0 code=0 id=9730 mark=0 zone=3 use=1
> >>> icmp 1 21 src=172.24.4.1 dst=10.0.0.4 type=8 code=0 id=9730
[UNREPLIED] src=10.0.0.4 dst=172.24.4.1 type=0 code=0 id=9730 mark=0 use=1
> >>
> >> The flows that I believe shows what happens are:
> >>
> >>> cookie=0x0, duration=31.952s, table=48, n_packets=2, n_bytes=196,
priority=100,ip,metadata=0x1
actions=ct(recirc,next_table=49,zone_reg=NXM_NX_REG5[])
> >>>
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98,
priority=65535,ct_state=+inv+trk,metadata=0x1 actions=drop
> >>>
> >>> cookie=0x0, duration=31.952s, table=49, n_packets=1, n_bytes=98,
priority=1002,ct_state=+new+trk,icmp,reg7=0x4,metadata=0x1
actions=ct(commit,zone_reg=NXM_NX_REG5[]),resubmit(,50)
> >>
> >>
> >> I did a similar test using ssh. After the ssh attempt, conntrack
shows:
> >>
> >>> sudo conntracl -L
> >>> tcp 6 27 SYN_RECV src=172.24.4.1 dst=10.0.0.4 sport=47784
dport=22 src=10.0.0.4 dst=172.24.4.1 sport=22 dport=47784 mark=0 zone=3
use=1
> >>> tcp 6 87 SYN_SENT src=172.24.4.1 dst=10.0.0.4 sport=47784
dport=22 [UNREPLIED] src=10.0.0.4 dst=172.24.4.1 sport=22 dport=47784
mark=0 use=1
> >>
> >> It looks like the SYN makes it to the VM, but the return traffic is
> >> dropped as invalid.
> >>
> >>
> >> Any thoughts on this? Is there anything obviously wrong with what I've
> >> done? Are there more details I should gather?
> >
> > It's a little hard to verify your analysis with only the the egress
> > pipeline flows, but looking at the conntrack output it seems about
> > right.
>
> Sorry for stripping it. I'll include the full flow table at the end of
> this message.
>
> > There's a bit of a trick here, and I'm guessing that the OVN flows as
> > generated today aren't quite handling this right. When the conntrack
> > module is loaded, and you send packets from the local stack (eg ping
> > from hypervisor -> VM), linux will piggyback conntrack information
> > onto the packet. This is attached during linux stack processing
> > between the application sending the packet and the packet arriving at
> > OVS. The packet's associated conntrack state is in zone=0 if you
> > haven't otherwise configured the local IPtables rules.
> >
> > For the below explanation, I'm going to assume that the zone
> > associated with your hypervisor VIF is zone=2, and the zone associated
> > with your VM is zone=3.
> >
> > It looks like OVN trusts that this conntrack information is correct
> > when the packet arrives (zone=0), and rather than going through the
> > from-lport conntrack processing (which should commit to zone=2), it
> > skips this step and goes straight through to the to-lport processing.
> > In to-lport, it goes through conntrack for zone=3 and commits the
> > connection there. So we have two connections: One in zone=0 and one in
> > zone=3.
> >
> > When the reply comes in the reverse direction, coming from the VM
> > there is no conntrack information associated. OVN correctly passes
> > this to conntrack for zone=3, which says "yep, this is the reply for
> > that connection we saw before". The rules continue processing to go to
> > the to-lport for the original VIF. This time, it submits to zone=2.
> > However, we never committed the connection to zone=2 so conntrack
> > marks it as invalid.
> >
> > Looking at the OVN rules, I think that this should be logically
> > handled correctly so probably it just needs some adjustments on the
> > translation to OpenFlow.
> >
>
> Thanks for the explanation! I'll dig into the CT support to see if I
> can figure out what might need to change. I'm guessing it's something
> like clearing existing CT state, or ignoring state from a different zone?
Roughly, yeah. The translation should be similar for conntrack matching
when transitioning from the from-lport to to-lport pipeline stages; at this
point, conntrack fields are present from the from-lport pipeline but we
wish to ignore it. Similarly, I'd expect us to ignore conntrack fields for
packets coming immediately from a port(from-lport pipeline). Hope that
makes some sense.
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev
