> -----Original Message----- > From: Franck Baudin [mailto:franck.baudin at qosmos.com] > Sent: Thursday, September 04, 2014 5:24 PM > To: Xie, Huawei; Gray, Mark D; Thomas Monjalon > Cc: dev at dpdk.org; dpdk-ovs at lists.01.org > Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest > (virtIO/vhost) > > Hi Huawei, > > On 09/04/14 04:54, Xie, Huawei wrote: > > Hi Franck: > > I checked your original thread. > > root at linux-native:~# tcpdump -vnei eth0 > > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 > bytes > > 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto > TCP (6), > length 60) > > 1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> > > 0xb2dd), > seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr > 0,nop,wscale 7], length 0 > > 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 > (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto > TCP (6), > length 40) > > 1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq > > 0, ack > 1963818357, win 0, length 0 > > > > The packet from the guest, received on the native machine, has "correct" > checksum 0x1db4. The TCP connection is refused is due to this packet has R > flag. > Could you check if it is the firewall that blocks the > > connection? > > > > Besides, I did ssh tcp test between two guest VMs, the ssh connection could > be established correctly with tx checksum off. I didn't use OVDK, but set the > arp > table and route table manually. > > The packet flow is > > Guest A(virtio)<->vhost example->Guest B(virtio) > I re-made a test between two identical VMs, UDP and ICMP are fine. Some > observations: > on both client & server: > > ethtool -k eth1 | grep ": on" > generic-receive-offload: on > highdma: on [fixed] > rx-vlan-filter: on [fixed] > > ethtool -k eth1 > Features for eth1: > rx-checksumming: off [fixed] > tx-checksumming: off > tx-checksum-ipv4: off [fixed] > tx-checksum-unneeded: off [fixed] > tx-checksum-ip-generic: off [fixed] > tx-checksum-ipv6: off [fixed] > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: off > tx-scatter-gather: off [fixed] > tx-scatter-gather-fraglist: off [fixed] > tcp-segmentation-offload: off > tx-tcp-segmentation: off [fixed] > tx-tcp-ecn-segmentation: off [fixed] > tx-tcp6-segmentation: off [fixed] > udp-fragmentation-offload: off [fixed] > generic-segmentation-offload: off [requested on] > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: on [fixed] > rx-vlan-filter: on [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: off [fixed] > tx-fcoe-segmentation: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > > On the client side: > 11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], > proto TCP > (6), length 60) > 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect -> > 0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val > 4294914387 > ecr 0,nop,wscale 5], length 0 > 0x0000: 4500 003c 6297 4000 4006 d420 0101 0102 E..<b. at .@....... > 0x0010: 0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S..... > 0x0020: a002 3908 0433 0000 0204 05b4 0402 080a ..9..3.......... > 0x0030: ffff 3153 0000 0000 0103 0305 ..1S........ > 11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], > proto TCP > (6), length 40) > 1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0, > ack 3885781957, win 0, length 0 > 0x0000: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..=.... > 0x0010: 0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S. > 0x0020: 5014 0000 b5e6 0000 P....... > > On the server side: > 11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], > proto TCP > (6), length 60) > 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq > 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr > 0,nop,wscale 5], length 0 > 0x0000: 4500 003c 6297 4000 4006 d420 0101 0102 E..<b. at .@....... > 0x0010: 0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S..... > 0x0020: a002 3908 e3bd 0000 0204 05b4 0402 080a ..9............. > 0x0030: ffff 3153 0000 0000 0103 0305 ..1S........ > 11:00:23.632034 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], > proto TCP > (6), length 40) > 1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0, > ack 3885781957, win 0, length 0 > 0x0000: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(.. at .@..=.... > 0x0010: 0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S. > 0x0020: 5014 0000 b5e6 0000 P....... > > > On the client, ethtool announce that TX checksum offload is disabled, > but tcpdump show it's not. On the server side, all packets looks fine > from a checksum perspective but the SYN is answered by a RST. Please > note that the same images behave correctly with a regular openvswitch, > as long as TX checksuming offloading is disabled. In your previous test case, everything looks fine except that server sends a RST packet. In this test, weird the virtio client sends packet with incorrect checksum with tx checksum offloading off. But at least RX side always receives correct checksum which indicates the virtio driver on client side generates checksum for it .
The server receives correct SYN packet with correct checksum, but it sends the RST packet, better root cause this issue first. With vhost example as the backend, nc tests or ssh tests always works fine on my system. If you have bandwidth, you could also test vhost backend. Vhost command line fyi: dpdk/examples/vhost/build/vhost-switch -c 3 -n 4 --huge-dir /dev/hugepages --socket-mem=512,0 -- -p 1 --rx-retry 1 --zero-copy 0 --vm2vm 2 Btw, I don't set any csum...=off in qemu command line. > > If anyone has an equivalent setup, functioning properly, thanks for > sharing its configurations parameters. if not, I might be able to wait > for vhost backend PMD driver: I will then with testpmd instead of > dpdk-ovs, to see where is the problem (maybe in my configuration...). > When so you plan to release the vhost backend PMD driver? > There is no vhost backend PMD driver but a vhost library for integration with OVDK. > Thanks! > Franck
