Hi Nicolas, Thanks for asking.
I use KVM. 3G seems like a tremendous amount on memory to manage 3 hosts. I am resisting to increase for the time being in my test environment. In production there are 4G minimum, so there it is fine. My test environment meant to catch issues, I run on low resources on purpose. That also allows me to run more tests in parallel. If the problem reappears I would like to compare the memory behaviour between 4.14 and 4.15. Run the manager on exact same JVM settings and perform some black box stress test while recording the JVM behaviour. Also, it is still early for me to start pointing fingers to any piece of software, I have run 4.15 for only a few days and fixed many issues. Regards, Rafael On Thu, 2021-01-21 04:26 AM, Nicolas Vazquez <[email protected]> wrote: > Hi Rafael, > > Was this solved by increasing the RAM on the management server? Which > hypervisor are you using in this test environment? > > > Regards, > > Nicolas Vazquez > > ________________________________ > From: Rohit Yadav " target="_blank"><[email protected]> > Sent: Wednesday, January 20, 2021 6:10 AM > To: [email protected] " > target="_blank"><[email protected]>; [email protected] " > target="_blank"><[email protected]>; Rafael del Valle " > target="_blank"><[email protected]> > Subject: Re: 4.15 Manager out of memory > > This "Ram is 1.5GB + swap", please increase RAM to 3G and try again. > > > Regards. > > ________________________________ > From: Rafael del Valle " target="_blank"><[email protected]> > Sent: Wednesday, January 20, 2021 14:39 > To: [email protected] " > target="_blank"><[email protected]> > Cc: [email protected] " > target="_blank"><[email protected]>; [email protected] " > target="_blank"><[email protected]> > Subject: Re: 4.15 Manager out of memory > > Hi Rohit > > This is not a production environment. This is a test environment run > on minimum resources for my CD/CI cloud release process, so you have to > take this with a pitch of salt: > > > My environment is Ubuntu 18.04 > Ram is 1.5GB + swap > DB is running on the same host. > > However, just in case, note that: > > With 4.14, 150 cloud environments were built and tested and this problem did > not arise. > with 4.15 the problem popped on the first environment. > > I can see the changes made from 4.14 to the JVM options and those are: > > -XX:+UseParallelGC > -XX:MaxGCPauseMillis=500 > > That could explain the difference in behaviour, but better tu remain > suspicious. > I am going to raise memory and continue testing and see if the problem > persists. > > Is memory profiling done as part of ACS release/testing? > > Regards, > Rafael > > > On Wed, 2021-01-20 09:06 AM, Rohit Yadav " > target="_blank"><[email protected]> wrote: > > Can you share your host memory, distro (CentOS or Ubuntu and which version) > > and describe your installation (hypervisor, storage, any special plugin etc > > you may be using). Are you running DB and usage server on the same host? > > The java process requires at least 2G RAM[1] for itself. > > > > > > [1] > > https://github.com/apache/cloudstack/blob/master/packaging/systemd/cloudstack-management.default#L18 > > > > > > Regards. > > > > ________________________________ > > From: [email protected] " target="_blank">" > > target="_blank"><[email protected]> > > Sent: Wednesday, January 20, 2021 13:13 > > To: [email protected] " target="_blank">" > > target="_blank"><[email protected]> > > Cc: [email protected] " target="_blank">" > > target="_blank"><[email protected]> > > Subject: 4.15 Manager out of memory > > > > Hi! > > > > Congrats for the 4.15 release! UI makes a great difference. > > > > I started launching deployment tests with 4.15 and I just noticed that the > > manager is running out of memory were 4.14 did not. > > > > This is in a test environment where the manager host resources are > > constrained yet 100% functional in 4.14. > > > > Anybody experiencing this? > > > > ● cloudstack-management.service - CloudStack Management Server > > Loaded: loaded (/lib/systemd/system/cloudstack-management.service; > > enabled; vendor preset: enabled) > > Drop-In: /etc/systemd/system/cloudstack-management.service.d > > └─mesh-dep.conf, mysql-dep.conf > > Active: failed (Result: exit-code) since Tue 2021-01-19 11:02:44 EST; > > 15h ago > > Process: 6641 ExecStart=/usr/bin/java $JAVA_DEBUG $JAVA_OPTS -cp > > $CLASSPATH $BOOTSTRAP_CLASS (code=exited, status=1/FAILURE) > > Main PID: 6641 (code=exited, status=1/FAILURE) > > > > Jan 19 11:02:40 manager java[6641]: INFO [c.c.c.ConsoleProxyManagerImpl] > > (AgentTaskPool-8:ctx-71ff758f) (logid:58f404cb) Console p > > Jan 19 11:02:40 manager java[6641]: WARN [c.c.a.m.AgentManagerImpl] > > (AgentConnectTaskPool-2683:ctx-abc795fe) (logid:c4046d5d) Reso > > Jan 19 11:02:44 manager java[6641]: OpenJDK 64-Bit Server VM warning: INFO: > > os::commit_memory(0x000000009ea80000, 26738688, 0) fail > > Jan 19 11:02:44 manager java[6641]: # > > Jan 19 11:02:44 manager java[6641]: # There is insufficient memory for the > > Java Runtime Environment to continue. > > Jan 19 11:02:44 manager java[6641]: # Native memory allocation (mmap) > > failed to map 26738688 bytes for committing reserved memory. > > Jan 19 11:02:44 manager java[6641]: # An error report file with more > > information is saved as: > > Jan 19 11:02:44 manager java[6641]: # > > /var/log/cloudstack/management/cloudstack-management.err > > Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Main > > process exited, code=exited, status=1/FAILURE > > Jan 19 11:02:44 manager systemd[1]: cloudstack-management.service: Failed > > with result 'exit-code'. > > > > > > [email protected] > > http://www.shapeblue.com > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > > @shapeblue > > > > > > > > > > [email protected] > http://www.shapeblue.com<http://www.shapeblue.com> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > > [email protected] > http://www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > >
