On Tue, 26 Nov 2013 21:15:02 -0800
James Yu <ypyu2011 at gmail.com> wrote:
> Running one directional traffic from Spirent traffic generator to l2fwd
> running inside a guest OS on a RHEL 6.2 KVM host, I encountered performance
> issue and need to increase the number of rxd and txd from 256 to 1024.
> There was not enough freeslots for packets to be transmitted in this routine
> virtio_send_packet(){
> ....
> if (tq->freeslots < nseg + 1) {
> return -1;
> }
> ....
> }
>
> How do I solve the performance issue by one of the following
> 1. increase the number of rxd and txd from 256 to 1024
> This should prevent packets could not be stored into the ring due
> to lack of freeslots. But l2fwd fails to run and indicate the number must
> be equal to 256.
> 2. increase the MAX_PKT_BURST
> But this is not ideal since it will increase the delay while
> improving the throughput
> 3. other mechanism that you know can improve it ?
> Is there any other approach to have enough freeslots to store the
> packets before passing down to PCI ?
>
>
> Thanks
>
> James
>
>
> This is the performance numbers I measured on the l2fwd printout for the
> receiving part. I added codes inside l2fwd to do tx part.
> ====================================================================================
> vhost-net is enabled on KVM host, # of cache buffer 4096, Ubuntu 12.04.3
> LTS (3.2.0-53-generic); kvm 1.2.0, libvirtd: 0.9.8
> 64 Bytes/pkt from Spirent @ 223k pps, running test for 10 seconds.
> ====================================================================================
> DPDK 1.3 + virtio + 256 txd/rxd + nice -19 priority (l2fwd, guest kvm
> process)
> bash command: nice -n -19
> /root/dpdk/dpdk-1.3.1r2/examples/l2fwd/build/l2fwd -c 3 -n 1 -b 000:00:03.0
> -b 000:00:07.0 -b 000:00:0a.0 -b 000:00:09.0 -d
> /root/dpdk/virtio-net-pmd-1.1/librte_pmd_virtio.so -- -q 1 -p 1
> ====================================================================================
> Spirent -> l2fwd (receiving 10G) (RX on KVM guest)
> MAX_PKT_BURST 10seconds (<1% loss) Packets Per Second
> -------------------------------------------------------------------------------------------------------------------------------
> 32 74k pps
> 64 80k pps
> 128 126kpps
> 256 133kpps
>
> l2fw -> Spirent (10G port) (transmitting) (using one-directional one port
> (port 0) setup)
> MAX_PKT_BURST < 1% packet loss
> 32 88kpp
>
>
> **********************************
> The same test run on e1000 ports
>
> ====================================================================================
> DPDK 1.3 + e1000 + 1024 txd/rxd + nice -19 priority (l2fwd, guest kvm
> process)
> bash command: nice -n -19
> /root/dpdk/dpdk-1.3.1r2/examples/l2fwd/build/l2fwd -c 3 -n 1 -b 000:00:03.0
> -b 000:00:07.0 -b 000:00:0a.0 -b 000:00:09.0 -- -q 1 -p 1
> ====================================================================================
> Spirent -> l2fwd (RECEIVING 10G)
> MAX_PKT_BURST <= 1% packet loss
> 32 110k pps
>
> l2fw -> Spirent (10G port) (TRANSMITTING) (using one-directional one port
> (port 0) setup)
> MAX_PKT_BURST pkts transmitted on l2fwd
> 32 171k pps (0% dropped)
> 240 203k pps (6% dropped, 130k pps received on
> eth6 (assumed on Spirent)) **
> **: not enough freeslots in tx ring
> ==> this indicate the effects of small txd/rxd (256) when more traffic is
> generated, the packets can not
> be sent due to lack of freeslots in tx ring. I guess this is the
> symptom occurs in the virtio_net
The number of slots with virtio is a parameter negotiated with the host.
So unless the host (KVM) gives the device more slots, then it won't work.
I have a better virtio driver and one of the features being added is multiqueue
and merged TX buffer support which would give a bigger queue.
