Hi all! It will probably be my last update on the subject.
Short version: I have solved this latency problem by deploying a new Guacamole server on better hypervisors. For those interested in the details, my Guacamole server was hosted on a small VMware cluster of two ESXi servers. The resources (CPU/RAM) of those ESXi servers are OK and all services hosted on them seemed to work fine. Those ESXi have old hardware that is not supported by VMware anymore. Compared to my production ESXi servers, they have fewer CPU (32 vs 96), slower CPU (2~2.6Ghz vs 3Ghz) and less network bandwidth (aggregated 1G vs aggregated 10G). So I deployed a new guacamole server (and its DB) on my production cluster. I have compared the execution time of the cat command on a big file and the improvement was significant. - 56 seconds from the first Guacamole server running on management ESXi. - 10~15 seconds from the new GUacamole server running on production ESXi. The "cat" is still bad practice, but it was a good way to measure the improvement. On my new Guacamole server, I can use the "less" command without seeing any latency I had seen with the previous server. Looking back at it, some of the services hosted on our management cluster have required some tuning to improve their performances. Each of those was addressed separately because they were individual projects. This latency issue has given me the chance to have a holistic understanding of our projects, tools, and infrastructure. I have tried to install a newer version of libssh2 (1.11.0 instead of 1.8.0) but it didn't improve the execution time on the new server. So I'll stick with the version available in the Ubuntu repository. Thanks for all your feedback. Best regards, Julien Lejeune IT administrator +32(0) 81 33 11 11 www.actia.be This email message as well as all attachments are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. Thank you. -----Message d'origine----- De : Luke H <[email protected]> Envoyé : mercredi 18 octobre 2023 11:03 À : [email protected]; Julien Lejeune <[email protected]> Objet : RE: Slow output - Linux connections Agreed on all counts! Nick, thank you so much for regularly reviewing this list, responding, and playing such a large role in what Apache Guacamole is and will be. AG is incredibly well thought out and feature rich. With its license, it will have longevity. Meaning, for us tech people, it is a solid technology to add to one's portfolio. One last detail for you Julien. As stated earlier, this is fascinating, and there is one more result I want to show - VNC. In order of shortest times for the BIGFILE.txt test: $ dd if=/dev/urandom bs=1024k count=1024 | base64 > BIGFILE.txt $ time cat BIGFILE.txt # Guacamole/VNC 0.000u 1.367s 0:24.99 5.4% 0+0k 267856+0io 0pf+0w # Direct SSH connection 0.008u 2.280s 2:18.43 1.6% 0+0k 0+0io 0pf+0w # Guacamole/SSH 0.006u 2.150s 4:39.53 0.7%0+0k 0+0io 0pf+0w I am not exactly sure how to interpret this counterintuitive VNC speed, but I sense that latency you see is governed not by Guacamole, but by the supporting connection library. The short answer to this discussion then, would be: it's probably a library dependency, and not related to Guacamole, nor Linux. Luke PS - These Guacamole numbers were generated with libssh2-1.11.0 and libvncserver-0.9.13. > On 10/18/2023 2:20 AM CDT Julien Lejeune <[email protected]> wrote: > > > Hi Nick, > > Thank you for your feedback. > I understand and I totally agree with everything you wrote. > > On that note, I am very happy with Guacamole, its features, and its > performances so far. > At my company, we manage servers hosted all around the world from the same > Guacamole instance and this « cat » issue is the only latency problem I have > encountered so far. I am impressed by this product and the work that has been > done on it. > > As you said, the real way to fix my problem is to kindly remind my colleagues > not to use « cat » on big files. That’s bad practice. > This latency latency I wrote about is only a technical symptom of this human > problem. > Nonetheless, I am excited to work with Guacamole and to understand it inside > out. I felt digging into this latency issue would be a great opportunity to > learn about this product. And I felt sharing it was a nice way to connect to > this project’s community. > > Best regards, > > Julien Lejeune > IT administrator > > +32(0) 81 33 11 11 > www.actia.be<http://www.actia.be/> [Linkedin] > <https://www.linkedin.com/company/actia-telematics-services> > > > > This email message as well as all attachments are for the sole use of the > intended recipient(s) and may contain confidential and privileged information. > Any unauthorized review, use, disclosure or distribution is prohibited. > If you are not the intended recipient, please contact the sender by reply > e-mail and destroy all copies of the original message. Thank you. > > De : Nick Couchman <[email protected]> > Envoyé : mercredi 18 octobre 2023 01:42 À : [email protected] > Objet : Re: Slow output - Linux connections > > On Tue, Oct 17, 2023 at 2:23 PM Julien Lejeune > <[email protected]<mailto:[email protected]>> wrote: > Hi all! > > Luke and Mark, thank you for your helpful insights. > > I haven't solved the problem yet, but I give you a little feedback anyway. > > > To be clear, if you connect directly via SSH without Guacamole, the cat takes > 4s. When using Guacamole with a defined SSH connection, it takes 1 minute 37 > seconds. > > It's probably worth noting that we don't make any specific claims (that I can > recall) about Guacamole being faster than a native SSH (or RDP) connection - > or, at least, not without some better definitions of the characteristics of > those connections. If I'm wrong about that, and we have made those claims > somewhere, then please someone jump in and correct me :-). Guacamole is > designed to perform reasonably well over connections that have elevated > latency and lower bandwidth, as it can adapt to those conditions, usually > better than the native protocols. It's also designed to give the user a > single clientless interface using only a web browser, with no additional > software, and is able to make those connections available via HTTP/HTTPS/WSS, > which are pretty standard ports and protocols to have open across network > boundaries. And, while, most of the time, performance should be comparable to > - and may be faster than - other SSH clients, there are definitely some > situations you're going to hit where the nature of the connection and what > you're trying to do is better-suited for a native client. > > If your client computer is relatively "close" to the SSH server (in the > single-digit ms range), and you're doing some very display-intensive things > (cat /dev/urandom, or tailing a very verbose log file, etc.), I would fully > expect that the platform-native SSH client is going to render the output of > the SSH connection much more efficiently than the Guacamole combination where > the SSH protocol has to be turned into Guacamole (series of images going > one-way and keystrokes and mouse movements going the other way) and then > rendered by a web browser. > > Now, the difference you're seeing is quite a bit - 4s -> 1m37s is something > like a 2400% increase. That said, I'm not sure there's all that much that can > be done - use "less" instead of cat, so that you're not attempting to stream > the entire contents of the file all at once? > > Aside from the "cat /var/log/huge.log" situations, are there other areas > where you're consistently seeing increased latency as compared to the native > SSH client? > > -Nick
