Thank you Nick for the information, Yes you are correct, i have not really though about the CPU now that you pointed it out I'll have a look at it as well.
On Mon, Sep 22, 2025 at 12:04 PM Nick Couchman <[email protected]> wrote: > On Wed, Jul 23, 2025 at 4:34 PM Gihan Rajapaksha <[email protected]> > wrote: > >> Hi, >> > Hello, Gihan. > > >> I am setting up a production deployment of Apache Guacamole and am >> seeking guidance on optimizing memory allocation, particularly concerning >> the guacd component, for high concurrent user loads. >> >> *Our Production Environment:* We are provisioning two Azure Standard >> E8as v5 virtual machines, each with: >> >> - >> >> *8 vCPUs* >> - >> >> *64 GiB of RAM* >> >> > I'm trying to remember, but if I recall correctly, we had some rough > numbers of 1 vCPU and 2 GB of RAM per 25 concurrent connections. > > >> *Our Goal:* Each of these servers needs to reliably support *1000 >> concurrent RDP sessions at peak time, traffic will be less after hours.* >> >> *Observed guacd Memory Behavior (Critical Concern):* During our testing >> on similar server specifications, we've observed that guacd appears to >> operate in a multi-process mode, where each active Guacamole session >> (specifically RDP) seems to correspond to a separate guacd child process. >> > > Yes, guacd performs a fork() for each connection, spawning a new PID that > is a copy of the parent guacd PID. Each PID spawns several threads that > perform the work of translating the remote protocols (RDP, VNC, SSH, etc.) > to Guacamole and back. > > >> Example: >> [image: image.png] >> >> >> From btop monitoring, we've seen individual guacd child processes >> consuming between *70MB and 120MB of RAM per session*. >> > This seems about right. > > *The Challenge:* If this per-process memory consumption scales linearly, >> then 1000 concurrent RDP sessions would require guacd alone to consume >> approximately *70 GiB to 120 GiB of RAM* (1000×70MB=70 GiB to 1000×120MB= >> 120 GiB). >> > Yep. > > >> This projected guacd memory demand significantly exceeds the total 64 >> GiB of RAM available on each of our target production servers. This raises >> a major concern about potential memory exhaustion, heavy swapping, and >> instability under full load, leaving little to no memory for the Tomcat >> application server, the operating system, Nginx (our reverse proxy), and >> other essential system processes. >> > It's good that you're thinking ahead on this :-). > >> *Our Current Tomcat Memory Allocation (Planned):* We had initially >> planned to allocate a significant portion of memory to Tomcat for high >> concurrency, with settings like: CATALINA_OPTS='-Xms16384M -Xmx32768M >> -server -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 >> -XX:+HeapDumpOnOutOfMemoryError' However, this 32 GiB maximum for Tomcat >> becomes untenable if guacd requires 70 GiB+ on the same 64 GiB server. >> >> *Questions for Guacamole Support:* >> >> 1. >> >> *guacd Scaling Confirmation:* Can you confirm if guacd is indeed >> designed to fork a separate process for each RDP session, and if the >> observed per-process memory consumption (70-120MB) is typical for active >> RDP sessions? >> >> Yes, this is correct. > >> >> 1. >> >> *Memory Sizing Recommendations:* What are your official memory sizing >> recommendations for guacd per active RDP session when planning for >> high concurrency (e.g., 1000+ sessions)? >> >> As mentioned above, 1 vCPU and 2 GB of memory per 25 concurrent > connections. > >> >> 1. >> >> *guacd Optimization:* Are there any recommended guacd configuration >> options or best practices to optimize its memory footprint for high >> concurrent RDP remote app sessions, particularly if there are ways to >> enable a more "threaded" or shared-memory model rather than a strict >> per-process model? >> >> No, there is no optimization to be done for guacd. > > >> >> 1. >> >> *Architectural Guidance:* Given our target of 1000 concurrent RDP >> sessions and the observed guacd behavior, what is your recommended >> architectural approach for these servers? Should we: >> - >> >> Increase the RAM on our servers significantly (e.g., to 128 GiB or >> 256 GiB)? >> - >> >> Plan for a much larger number of smaller 64 GiB servers, each >> handling a reduced number of concurrent sessions (e.g., 200-300 >> sessions >> per server)? >> - >> >> Would it be necessary to provide a large chunk of memory for >> tomcat ? >> >> There are a couple of things to consider as you try to plan this out: > * Neither guacd nor guacamole (Tomcat) currently have any mechanism for HA > or load balancing built into them. This means, if you're intending to scale > out (multiple VMs) instead of scaling up (larger VMs), you need to make > sure that, first, you've architected the components in such a way that you > can reliably get users to the same back-end systems (a user who logs in to > Guacamole Client must get sent to the same back-end server when starting > connections, etc.) and, second, you understand that there are implications > for things like connection sharing that may not or will not work reliably > between users who get shuffled to different back-end systems. > * You're quite fixated on RAM allocation, but you need to factor in CPU > allocation, as well. As I mentioned, guidance in the past is 1 vCPU per 25 > concurrent connections - if you're planning on having 1000 concurrent > connections, then you're going to need something like 40 vCPUs. This can > vary a bit, depending on what each of the users is actually doing on their > session (streaming video over an RDP connection will take up a lot more CPU > than someone using an SSH terminal), but it's the general guidance. > > -Nick >
