I have a NATting VM (let's call this vm/nat) sitting in front of another VM (let's call this one vm/wget), with vm/wget residing in a private virtual network, with all network connectivity for vm/wget going through vm/nat. Additionally, I have a web server running on a physical machine from which vm/wget is trying to fetch a file. Graphically, it looks something like this:
web server <-> vm/nat <-> vm/wget Now, when vm/wget tries to wget a file from the web server, it frequently gets a connection reset by peer network error, wget will retry the connection and resume the download, after which the cycle will repeat many times before the file eventually fully downloads. The problem is not limited to wget though, both scp and curl experience the same connection reset by peer issue quite consistently. After some collaborative debugging on IRC, tcpdump revealed that an RST is being generated which is breaking the network connection. Now, from the web server's point of view, the RST is coming from vm/wget and from vm/wget's point of view, the RST is coming from the web server. Further tcpdumping from both NICs on vm/nat reveals that vm/nat itself is generating the RSTs, although it is not clear why. Furthermore, the TCP sequence numbers on the RSTs are totally out of line with the ACKs in the stream, for example: 13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1914061, win 65535, length 0 13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1914061, win 65535, length 0 13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1918441, win 65535, length 0 13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R], seq 1390832295, win 0, length 0 We ran out of ideas on how to further debug the RST issue, but ideas are welcome. One other thing worth noting is that vm/nat is configured with a pair of virtio-net NICs. If I change the NICs to e1000 models, the RST issue vanishes. More specifically, only the NIC facing the web server on vm/nat needs to be changed to e1000 for the issue to go away, the other can remain virtio-net. Also, running a wget directly on vm/nat works in all cases. I tried a number of other things to try and narrow down the issue, none of which helped, including: - vhost=on - qemu versions 2.7.0, 2.3.0 and 2.0.0 - disabling kvm accel (ie, using TCG) The full command line for vm/nat is this: /usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp 4,cores=4,sockets=1,threads=1 -drive file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device virtio-net- pci,mac=52:54:00:42:45:15,netdev=net1 -netdev type=tap,script=/etc/qemu-ifup- br0,downscript=no,id=net1 -device virtio-net- pci,mac=52:54:00:42:45:16,netdev=net2 -netdev type=tap,script=/etc/qemu-ifup- br2,downscript=no,id=net2 -curses -monitor unix:/var/lib/kvm/monitor/lb,server,nowait -name "lb.polldev.com",process=lb.polldev.com I was able to reproduce this with different web server machines, as well as different vm/wget machines, as well as when moving vm/nat to a different host (albeit for the other host, the RST issue presents itself significantly less frequently). Since the problem goes away with e1000, my gut feeling is that virtio-net is somehow slightly mangling the packets which causes iptables to eventually RST the connection, although it's probably more subtle than that, since I'm sure I'm not the first person to try NAT in a virtio-net VM. Any thoughts? - Neil
