On 28 Sep 2023, at 5:35, Levente Csikor via discuss wrote:
> Hi all, > > I am playing around with HW offloading and I have observed something > strange that I cannot explain (as of now). > > I have two OvS switches (both supporting and having hw-offload:true) > connected back-to-back through two physical ports. > > I am actually measuring latency with Pktgen-DPDK, so I send latency > measuring packets through my first OvS, which will be returned by the > second OvS. Both are running on a SmartNIC, but the scenario is > somewhat similar of having them as hypervisor switches using kernel > drivers and pktgen-dpdk app is running in a VM "above" the first OvS. > > The flow rules are based on Layer-3 information (instead of simple port > forward). > > OvS 1 flow rules (pfXhpf face the VM, pX face the other OvS): > - ip,in_port=pf0hpf,nw_src=192.168.0.1,nw_dst=192.168.1.1 > actions=output:p0 > - ip,in_port=p1,nw_src=192.168.0.1,nw_dst=192.168.1.1 > actions=output:pf1hpf > > OvS 2 flow rules: > - ip,in_port=p0,nw_src=192.168.0.1,nw_dst=192.168.1.1 actions=output:p1 > > The routing is working as expected, I get back all the packets, but due > to the relatively bad latency I measured, I checked whether these flow > rules, or in fact, their corresponding flow-caches, are indeed > offloaded to the HW. > > In the case of the first OvS, the following command gives the below > output: > # ovs-appctl dpctl/dump-flows -m |grep offloaded > > ufid:66bfe654-11ac-4f01-8dec-414177df78c4, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(p1),packet_type(ns=0/0 > ,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00 > :00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.16 > 8.1.1,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:368243396, > bytes:29459460538, used:0.120s, offloaded:yes, dp:tc, actions:pf1hpf > ufid:89516b0d-bb6f-4df0-ba91-973ef79e6fa3, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(pf0hpf),packet_type(ns > =0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:0 > 0:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=192.168.0.1,dst=19 > 2.168.1.1,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:3630728485, > bytes:275935348872, used:0.080s, offloaded:yes, dp:tc, actions:p0 > ufid:3fbc4e09-9d78-4e3b-8711-541d36828959, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(pf0hpf),packet_type(ns > =0/0,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:0 > 0:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,ds > t=0.0.0.0/128.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), > packets:233401993, bytes:17738549256, used:0.080s, offloaded:yes, > dp:tc, actions:drop > > The key observation here is that the entries offloaded are using tc, > and the ones related to my layer-3 forwarding rules (1st and 2nd entry) > are indeed offloaded to the hardware. > > However, on the second OvS, I see this: > ufid:b88192f5-17d5-461d-ba61-2bd0b7e49b39, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(p1),packet_type(ns=0/0 > ,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00 > :00/00:00:00:00:00:00),eth_type(0x88cc), packets:0, bytes:0, > used:never, offloaded:yes, dp:tc, actions:drop > ufid:82f41402-9d5b-4229-a1c0-d75a35138ebe, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(p0),packet_type(ns=0/0 > ,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00 > :00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.16 > 8.1.1,proto=0/0,tos=0/0,ttl=0/0,frag=no), packets:432344072, > bytes:26805332464, used:0.000s, dp:tc, actions:p1 > ufid:da660698-a31e-47e5-bf30-34ee90486f57, > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0) > ,ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(p0),packet_type(ns=0/0 > ,id=0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00 > :00/00:00:00:00:00:00),eth_type(0x88cc), packets:0, bytes:0, > used:never, offloaded:yes, dp:tc, actions:drop > ufid:68dc355c-0823-4324-8a02-fa104597b8d8, > recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p0),skb_mark(0/0),c > t_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=18:5a:58:0 > c:c8:ca,dst=01:80:c2:00:00:00),eth_type(0/0xffff), packets:67094, > bytes:4025640, used:0.313s, dp:ovs, actions:drop > ufid:39f2ee6d-4a1b-4781-a73e-2d0a8d045df3, > recirc_id(0),dp_hash(0/0),skb_priority(0/0),in_port(p1),skb_mark(0/0),c > t_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=00:00:00:0 > 0:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_ > type(0/0xffff), packets:4112, bytes:246720, used:1.725s, dp:ovs, > actions:drop > > The important part here is that the enrty matching my layer-3 > forwarding rule has only the dp:tc flag, but it does not show the > offloaded:yes tag. However, for other entries (e.g., for a drop entry), > these two flags are shown together. > > Therefore, my question is as follows: > > Does dp:tc on its own mean that HW offloaded is enabled AND working? dp:tc means that the flows are installed in the kernel’s TC framework, and not in the OVS kernel module. If the rule can be offloaded to the hardware by the SmartNIC driver, the offloaded:yes flag is set. > If yes, why do we have an explicit tag as offloaded:yes? > > If no, how can I enforce HW offloading beyond this? It’s the driver of you SmartNIC that decides to offload the flow. You can use the tc command to show the flows being programmed in TC. Looking at the flow that is not offloaded the destination port is p1, is this a port that is offloadable, i.e. is it an sriov port? If it’s a regular kernel port this can not be offloaded as the nic can not send it to this port. > Did anyone encounter something similar before? > I did a presentation a while back with some details; https://www.youtube.com/watch?v=87Mx547WEG8 Or you can look at the ovs_perf test tools’ documentation for how to setup HW offload: https://github.com/chaudron/ovs_perf/tree/RHEL8#open-vswitch-with-linux-kernel-datapath-and-tc-flower-offload > Thank you for considering my request. > > Cheers, > levi > > > _______________________________________________ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
