JJK, this is actually quite helpful data, as I saw similar results when
doing my internal testing. The falloff rate seems to increase as the
latency increases, suggesting a fixed window or at least one that isn't
scaling properly as latency increases, which causes unusually fast
performance drops with distance increases.
The windowing reference for TCP at http://bradhedlund.com/2008/
12/19/how-to-calculate-tcp-throughput-for-long-distance-links/ is exactly
what I was suspecting was the issue, but it seems to be more complicated
than that based on everyone's feedback. OpenVPN in UDP mode is still
operating a TCP windowing layer somewhere, right? Is it possible that this
code is not windowing well for connections over 20ms? This would also
explain some of the giant performance difference between hardware and
software encryption, because it is actually reducing the latency between
the devices over the OpenVPN link, even though in software for 1Gbit on an
x86 processor you are nowhere near full CPU utilization on a single core.
The drop being asymmetrical is interesting, as I had not thought of
reversing the locations / configs to test. This adds more data to the
mystery.
Excellent data points, thank you!
-Derek
Derek Zimmer
Chief Executive Officer
Open Source Technology Improvement Fund
On Tue, Jun 5, 2018 at 7:50 AM, Jan Just Keijser <janj...@nikhef.nl> wrote:
> Following up on myself....
>
> On 05/06/18 14:25, Jan Just Keijser wrote:
>
>
> On 01/06/18 02:50, Derek Zimmer wrote:
>
> I'm still working on this, as I think it is worthwhile for us to explore
> and get some hard data on how all of these things perform in a real world
> environment.
>
> I've been stalled by transitioning to a new job.
>
> >Same here. I guess this interacts with other properties, like the delay
> >OpenVPN itself adds. And that is where AES-GCM, with it's blazingly
> >fast hardware acceleration, outperforms AES-CBC + HMAC-SHA in orders of
> >magnitude (at the crypto level).
>
> This might be interesting, and it also might be why my real world testing
> doesn't match what we see at https://community.openvpn.net/openvpn
> /wiki/Gigabit_Networks_Linux
>
> It looks like this experiment was conducted on machines on a LAN with
> virtually no latency / forwarding being done, so while it does show us some
> theoretical numbers they don't seem to apply to the real-world use-cases
> that we are hoping to get these types of performance figures for.
>
> As soon as we start adding latency and jitter performance seems to tank
> with these optimizations.
>
> So that I am not chasing phantoms, do we have any real-world examples of
> the claim by janj...@nikhef.nl or are we just going off of the
> Gigabit_Networks_Linux page? If we have real world examples of
> configurations that can push more than 250Mbit (on a 1Gb controller) or
> 2.5Gbit (on a 10Gb controller) over connections with more than 10ms of
> latency then it would allow me to significantly narrow my search for
> problem areas.
>
>
> the experiment *was* conducted on a LAN with virtually no latency at that
> time. However, I've just repeated the experiment going from a university to
> my institute (~ 50 km distance) using a 1 Gbps connection. The results are
> nearly identical:
>
> [ 5] local 10.200.0.2 port 34072 connected with 10.200.0.1 port 5001
> [ 5] 0.0-10.0 sec 707 MBytes 592 Mbits/sec
> [ 4] local 10.200.0.2 port 5001 connected with 10.200.0.1 port 51086
> [ 4] 0.0-10.0 sec 874 MBytes 731 Mbits/sec
>
> (where the LAN performance is ~ 910 Mbps).
> Server: Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz
> Client: Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz
>
> so as you see I am using 4+ year old hardware for this.
>
> I am happy to work with you to figure out what is causing the performance
> to tank. I've got access to 1 & 10 Gbps (and possible higher) internet
> links and am very curious what is causing the jitter on your end.
>
> For higher latency links, the main option to tweak seems to be
> --txqueuelen : increase this to at least 1000 to improve performance.
>
>
> after I add an artificial latency of 20 ms on both ends using
> tc qdisc add dev eth0 root netem delay 20ms
> I do see quite a difference:
> outside the tunnel performance ~ 500 Mbps both ways
> inside the tunnel:
> client->server: 40 Mbps ;
> server->client: ~200 Mbps
>
> It's not an iperf connection issue, if I reverse the client and server
> (i.e iperf -s and iperf -c ) then the numbers are swapped - hence it seems
> that the *tunnel itself* is slower in one direction than the other . Both
> client and server are running the same OS and the same version of OpenVPN.
>
> Ping time between hosts is now 42 ms (2+2x20)
> So for some strange reason an assymmetry is introduced; playing with
> --txqueuelen, --rcvbuf or --sndbuf did not help to remove this assymmetry.
> Also, when I read
> http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-
> throughput-for-long-distance-links/
> I am not surprised that performance drops off with a higher latency ... it
> is always worth investigating how to reduce the latency over a LAN/WAN
> before attempting to tweak a VPN connection.
>
> The assymmetry part is what puzzles me the most - I will keep looking into
> that.
>
> cheers,
>
> JJK
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Openvpn-devel mailing list
Openvpn-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openvpn-devel
