Il 12/11/2014 01:19, Ben Pfaff ha scritto:
Pravin, I think you mentioned that you were doing some work on
bug-fixing in MPLS. Does it sound like it has any relationship to
this bug reported by Stefano?
On Tue, Nov 11, 2014 at 11:25:34PM +0100, Stefano Salsano wrote:
we need to POP out two MPLS labels in an egress node and then send
out the packet as plain IP packet, but apparently this is not
working in Open vSwitch v2.390 operating in kernel mode
Hi all,
we have been progressing in the analysis of the bug and sent a detailed
report to Pravin (attached hereafter)
On 18/11/2014 18:10, Pravin Shelar wrote:
> Thanks for the detailed report.
> I think userspace is not generating correct MPLS action for multiple
> pop. It should use recirc action when it reaches the stack limit. As
> you have seen for kernel datapath it is one. You can force
> check_max_mpls_depth() to return one for all datapath and try same
> test on netdev datapath.
> There must be bug in MPLS action generation, you need to check action
> translation code.
we forced check_max_mpls_depth() to return one also for netdev datapath,
nothing changed for the pop on peo2, while the push has been correctly
modified to forward to slow path
this happens because in function compose_mpls_pop_action
https://github.com/openvswitch/ovs/blob/master/ofproto/ofproto-dpif-xlate.c#L3283
there is no check that if number of labes >= max_mpls_depth the packet
should be forwarded to slow path, the packet is handled in fast path and
it fails for kernel datapath
the rules shown hereafter demonstrate that even if
check_max_mpls_depth() returns one, the pops are handled by the netdev
datapath !
on the other hand, this check is correctly enforced in the
compose_mpls_push_action
sudo ovs-appctl dpif/dump-flows on peo1:
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(tos=0/0xfc,ttl=64,frag=no),
packets:11, bytes:1078, used:0.006s,
actions:userspace(pid=0,slow_path(action))
sudo ovs-appctl dpif/dump-flows on peo2:
recirc_id(0),in_port(4),eth_type(0x8847),mpls(lse0=0x80004040/0xfffff100,lse1=0x40000140/0xffffffff),
packets:80, bytes:8480, used:0.262s,
actions:pop_mpls(eth_type=0x8847),pop_mpls(eth_type=0x800),5
we are now trying to add the correct check in compose_mpls_pop_action:
} else if (n >= ctx->xbridge->max_mpls_depth) {
COVERAGE_INC(xlate_actions_mpls_overflow);
ctx->xout->slow |= SLOW_ACTION;
}
and we will let you know the results
*****************************************************************
Detailed Report of the problem sent on 18/11/2014
1) recirculation does not work properly when there are
two or more mpls_pop, let us compare what happens in our system when
using the kernel datapath (not working) or the userspace netdev datapath
(working)
we are using this reference topology
h1---peo1----peo2---h2
h1 source node, peo1 ingress node, peo2 egress node, h2 dest node
in peo1, we report the rules shown with ovs-ofctl (obviously they are
the same with kernel and netdev datapath)
peo1:
-> cookie=0x0, duration=185.672s, table=0, n_packets=178, n_bytes=17444,
ip,in_port=2
actions=push_mpls:0x8847,set_field:262144->mpls_label,push_mpls:0x8847,set_field:524292->mpls_label,goto_table:1
-> cookie=0x0, duration=185.617s, table=1, n_packets=178, n_bytes=17444,
mpls,in_port=2,mpls_label=524292 actions=output:1
recirculation causes the installation of flows that are not visible with
the ovs-ofctl command, so we use ovs-appctl
(sudo ovs-appctl dpif/dump-flows peo1)
the result is different with kernel and netdev datapath:
kernel datapath:
-> recirc_id(0),in_port(2),eth_type(0x0800),
ipv4(tos=0/0xfc,ttl=64,frag=no), packets:14, bytes:1372, used:0.692s,
actions:userspace(pid=4294959673,slow_path(action))
netdev datapath:
-> recirc_id(0),in_port(2),eth_type(0x0800),
ipv4(tos=0/0xfc,ttl=64,frag=no), packets:17, bytes:1666, used:0.664s,
actions:push_mpls(label=262144,tc=0,ttl=64,bos=1,eth_type=0x8847),push_mpls(label=524292,tc=0,ttl=64,bos=0,eth_type=0x8847),1
we note that with kernel datapath, there is a fallback to userspace
processing that is triggered when multiple PUSH operation are requested
in peo2, we report the rules shown with ovs-ofctl (obviously they are
the same with kernel and netdev datapath)
peo2:
-> cookie=0x0, duration=621.359s, table=0, n_packets=284, n_bytes=30104,
mpls,in_port=1 actions=goto_table:1
-> cookie=0x0, duration=621.299s, table=1, n_packets=284,
n_bytes=30104, mpls,in_port=1,mpls_label=524292
actions=pop_mpls:0x8847,resubmit(,1)
-> cookie=0x0, duration=621.184s, table=1, n_packets=284, n_bytes=30104,
mpls,in_port=1,mpls_label=262144,mpls_bos=1
actions=pop_mpls:0x0800,output:2
recirculation causes the installation of flows that are not visible with
the ovs-ofctl command, so we use ovs-appctl
(sudo ovs-appctl dpif/dump-flows peo1)
the result is different with kernel and netdev datapath:
kernel datapath:
NO RULES ARE SHOWN !!!!
netdev datapath:
-> recirc_id(0),in_port(4),eth_type(0x8847),
mpls(lse0=0x80004040/0xfffff100, lse1=0x40000140/0xffffffff),
packets:5, bytes:530, used:0.720s,
actions:pop_mpls(eth_type=0x8847),pop_mpls(eth_type=0x800),5
peo2 is the egress node where we execute the double pop, it seems that
something is preventing the installation of the "recirculation" flows
when using the kernel datapath
We have also checked the ovs-vswitchd log and when we activate the
kernel datapath, we see these warning messages:
-> |WARN|system@ovs-system: failed to put[create] (Invalid argument)
recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(4),skb_mark(0/0),eth(src=e6:da:61:3a:f5:81/00:00:00:00:00:00,dst=0a:a5:58:7d:1c:12/00:00:00:00:00:00),eth_type(0x8847),mpls(label=524292/0xfffff,tc=0/0,ttl=64/0x0,bos=0/1),
actions:pop_mpls(eth_type=0x8847),pop_mpls(eth_type=0x800),5
-> |WARN|system@ovs-system: execute pop_mpls(eth_type=0x8847),
pop_mpls(eth_type=0x800),5 failed
(Invalid argument) on packet mpls,in_port=0,vlan_tci=0x0000,
dl_src=e6:da:61:3a:f5:81,dl_dst=0a:a5:58:7d:1c:12,mpls_label=524292,
mpls_tc=0,mpls_ttl=64,mpls_bos=0,mpls_lse1=1073742144
These messages explain also the absence of the "recirculation flows",
even if in our mininet emulation scenario there is a single ovs-vswitchd
instance, we can refer these messages to peo2 from the "output port"
We also noted at the beginning of the vswitchd-log these messages:
When using the Kernel datapath:
2014-11-17T19:09:23.698Z|00029|ofproto_dpif|INFO|system@ovs-system:
MPLS label stack length probed as 1
When using the userspace netdev datapath:
2014-11-17T19:17:37.697Z|00069|ofproto_dpif|INFO|netdev@ovs-netdev:
MPLS label stack length probed as 3
We have checked in the ovs code, this print is generated by the
function static size_t check_max_mpls_depth(struct dpif_backer
*backer), this function tries to probe the stack length...
we would like to understand what is behind this limitation... why in our
system the stack length for the kernel datapath is limited to 1 ?
does the stack length depend on some features provided by the
system/kernel? is there some kernel version without this limitation?
assuming that we have this limitation, our understanding is that this
control does not prevent to PUSH three mpls label because the packet is
sent to the userspace ovs-switchd as a fallback (and the packet is
correctly created and forwarded)... is this correct?
if this is the case, it seems that the same fallback is not activated in
case of multiple POP operations (>=2 POPs) the system tries to execute
the multiple POPs in the kernel but it generates the error messages
listed above and no packet is forwarded
End Detailed Report of the problem
*****************************************************************
We have replicated the problem in a simple mininet setup (script
attached) with four nodes connected as follows:
h1---peo1----peo2---h2
h1 source node, peo1 ingress node, peo2 egress node, h2 dest node
- h1 pings h2 (automatic static arp table are used, so there is no ARP)
- peo1 pushes two mpls labels in the packet
- peo2 is supposed to pop the two labels and forward the packet as IP
we have a rule to match the outer label, POP it and resubmit as MPLS packet:
actions=pop_mpls:0x8847,resubmit(,1)
then we have a rule to match the inner label, POP it and output on
a port as an IP packet:
actions=pop_mpls:0x0800,output:11
both rules match (and the first pop works), but then the packet
does not exit from the port 11
we have also tried without success to recirculate the packet
implicitly after the first pop, replacing the first action with:
actions=pop_mpls:0x8847
Can anyone help us to identify the problem ? Are we doing something
wrong or is it a bug?
The mininet script to setup the topology and create the rules in
order to reproduce the problem is attached.
Some details are reported hereafter for the scenario with explicit resubmit.
thank you in advance for your help...
ciao
Stefano
- peo1 flow table
OFPST_FLOW reply (OF1.3) (xid=0x2):
duration=275.553s, table=0, n_packets=58, n_bytes=5684,ip,in_port=2
actions=push_mpls:0x8847,set_field:262144->mpls_label,push_mpls:0x8847,set_field:524292->mpls_label,goto_table:1
duration=275.493s,table=1,n_packets=58,n_bytes=5684,mpls,in_port=2,mpls_label=524292
actions=output:1
- peo2 flow table
OFPST_FLOW reply (OF1.3) (xid=0x2):
duration=317.678s,table=0,n_packets=58,n_bytes=6148,mpls,in_port=1
actions=goto_table:1
duration=317.618s,table=1,n_packets=58,n_bytes=6148,mpls,in_port=1,mpls_label=524292
actions=pop_mpls:0x8847,resubmit(,1)
duration=317.559s,table=1,n_packets=58,n_bytes=6148,mpls,in_port=1,mpls_label=262144,mpls_bos=1
actions=pop_mpls:0x0800,output:2
NB If the interfaces created by Mininet in the root namespace happen
to be down, stop the Network Manager and then restart the experiment
--
*******************************************************************
Stefano Salsano
Professore Associato
Dipartimento Ingegneria Elettronica
Universita' di Roma Tor Vergata
Via del Politecnico, 1 - 00133 Roma - ITALY
http://netgroup.uniroma2.it/Stefano_Salsano/
E-mail : stefano.sals...@uniroma2.it
Cell. : +39 320 4307310
Office : (Tel.) +39 06 72597770 (Fax.) +39 06 72597435
*******************************************************************
#!/usr/bin/python
import subprocess
from mininet.net import Mininet
from mininet.node import Host, OVSKernelSwitch, Node
from mininet.cli import CLI
from mininet.log import lg, info
def setup():
lg.setLogLevel('info')
net = Mininet(switch=OVSKernelSwitch, build=False, autoStaticArp=True )
host1 = net.addHost("h1")
host2 = net.addHost("h2")
peo1 = net.addSwitch("peo1")
peo2 = net.addSwitch("peo2")
net.addLink(peo1, peo2) # 1 | 1
net.addLink(host1, peo1)# N/A | 2
net.addLink(host2, peo2)# N/A | 2
net.start()
#PEO1 Configuration
root = Node( 'root', inNamespace=False )
root.cmd('ovs-vsctl --no-wait set bridge %s protocols=OpenFlow13'
%(peo1.name))
root.cmd('ovs-ofctl -O OpenFlow13 add-flow %s
"table=0,hard_timeout=0,priority=32768,in_port=2,eth_type=0x800,actions=push_mpls:0x8847,set_field:262144->mpls_label,push_mpls:0x8847,set_field:524292->mpls_label,goto_table:1"'
%(peo1.name))
root.cmd('ovs-ofctl -O OpenFlow13 add-flow %s
"table=1,hard_timeout=0,priority=32768,in_port=2,eth_type=0x8847,mpls_label=524292,action=output:1"'
%(peo1.name))
#PEO2 Configuration
root.cmd('ovs-vsctl --no-wait set bridge %s protocols=OpenFlow13'
%(peo2.name))
root.cmd('ovs-ofctl -O OpenFlow13 add-flow %s
"table=0,hard_timeout=0,priority=32768,in_port=1,eth_type=0x8847,actions=goto_table=1"'
%(peo2.name))
root.cmd('ovs-ofctl -O OpenFlow13 add-flow %s
"table=1,hard_timeout=0,priority=32768,in_port=1,eth_type=0x8847,mpls_label=524292,action=pop_mpls:0x8847,resubmit(,1)"'
%(peo2.name))
root.cmd('ovs-ofctl -O OpenFlow13 add-flow %s
"table=1,hard_timeout=0,priority=32768,in_port=1,eth_type=0x8847,mpls_label=262144,mpls_bos=1,action=pop_mpls:0x800,output:2"'
%(peo2.name))
CLI(net)
net.stop()
subprocess.call(["sudo", "mn", "-c"], stdout=None, stderr=None)
if __name__ == '__main__':
setup()
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
--
*******************************************************************
Stefano Salsano
Professore Associato
Dipartimento Ingegneria Elettronica
Universita' di Roma Tor Vergata
Via del Politecnico, 1 - 00133 Roma - ITALY
http://netgroup.uniroma2.it/Stefano_Salsano/
E-mail : stefano.sals...@uniroma2.it
Cell. : +39 320 4307310
Office : (Tel.) +39 06 72597770 (Fax.) +39 06 72597435
*******************************************************************
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
