Yeah, it will work for a little while until it starts swapping. Up until this point most of the bugs filed have been about starting management server or initial zone setup. On Feb 20, 2013 11:09 PM, "Sudha Ponnaganti" <sudha.ponnaga...@citrix.com> wrote:
> I just had a call with some of the QA Team members and starting yesterday, > everyone seem to be running into this issue. I haven't heard anyone > complaining about it earlier, however only little testing has happened > before. > > > -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Wednesday, February 20, 2013 10:05 PM > To: cloudstack-dev@incubator.apache.org > Subject: Re: [DISCUSS] Management Server Memory Requirements > > We began seeing this at the point of the 4.1 cut. Our xen devclouds > stopped working until we increased the dom0 RAM to 1.5GB. Nobody else was > complaining, and most people don't run the mgmt server in devcloud, so I > was just waiting to see where the conversation about memory went on the > MAVEN_OPTS e-mail thread. > > On Wed, Feb 20, 2013 at 10:26 PM, Sudha Ponnaganti < > sudha.ponnaga...@citrix.com> wrote: > > Parth / Sailaja, > > > > Can you update ticket with data points and see if this can be assigned > to Alex to start investigation with Javelin merge unless this can be > associated with a specific check-in. > > > > Talluri - would you be able to narrow down to the check-in or build that > we started to see this?? > > > > Thanks > > /sudha > > > > -----Original Message----- > > From: Parth Jagirdar [mailto:parth.jagir...@citrix.com] > > Sent: Wednesday, February 20, 2013 9:05 PM > > To: cloudstack-dev@incubator.apache.org > > Subject: Re: [DISCUSS] Management Server Memory Requirements > > > > Marcus, > > > > > > I attempted login into UI while running the log. > > > > [root@localhost management]# vmstat 1 > > procs -----------memory---------- ---swap-- -----io---- --system-- > > -----cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > id > > wa st > > 0 0 191132 70904 10340 16192 36 28 40 91 9 1 0 0 > 89 > > 10 0 > > 0 0 191132 70904 10340 16204 0 0 0 0 46 75 0 0 > > 100 0 0 > > 0 0 191132 70904 10356 16204 0 0 0 36 72 221 1 0 > 91 > > 8 0 > > 0 0 191132 70904 10372 16188 0 0 0 44 73 130 0 1 > 88 > > 11 0 > > 0 0 191132 70780 10420 16208 0 0 4 276 83 191 1 0 > 75 > > 24 0 > > 0 0 191132 70780 10452 16192 0 0 0 120 106 309 1 0 > 77 > > 22 0 > > 0 0 191132 70780 10468 16200 0 0 0 40 91 183 1 1 > 90 > > 8 0 > > 0 0 191132 70780 10468 16216 0 0 0 0 47 128 0 0 > > 100 0 0 > > 0 0 191132 70780 10484 16216 0 0 0 36 70 136 0 0 > 94 > > 6 0 > > 0 0 191132 70656 10500 16200 0 0 0 40 66 116 1 0 > 91 > > 8 0 > > 0 0 191132 70656 10500 16216 0 0 0 0 47 94 0 0 > > 100 0 0 > > 0 1 189504 66216 10400 17940 2192 100 3928 172 404 579 9 2 > 5 > > 84 0 > > 1 1 188772 60220 10552 21992 1000 0 5220 68 412 741 7 2 > 21 > > 69 0 > > 1 2 187352 49316 7832 30344 1660 32 10052 32 833 1015 28 3 > 0 > > 69 0 > > 0 4 188816 52420 1392 25716 1488 2872 3168 3240 663 870 19 3 > 0 > > 78 0 > > 1 1 187388 51808 1372 25040 2476 1104 3084 1260 675 813 15 3 > 0 > > 82 0 > > 0 1 187360 54040 1500 24980 32 0 1048 0 447 379 6 1 > 0 > > 93 0 > > 0 1 187360 53916 1516 25004 0 0 924 52 309 283 1 0 > 0 > > 99 0 > > 0 1 195476 64076 1272 20624 0 8116 32 8156 312 308 1 2 > 0 > > 97 0 > > 0 0 203084 71920 1264 19412 0 7608 0 7608 256 173 0 2 > 89 > > 9 0 > > 0 0 203076 71324 1376 20132 64 0 868 40 192 232 2 0 > 65 > > 33 0 > > 0 0 203076 71328 1392 20108 0 0 0 68 75 144 1 0 > 85 > > 14 0 > > 0 0 203076 71084 1392 20392 0 0 268 0 66 132 0 1 > 96 > > 3 0 > > 0 0 203076 71084 1408 20392 0 0 0 36 60 122 0 0 > 94 > > 6 0 > > 0 0 203076 71076 1424 20376 0 0 0 36 77 148 1 0 > 92 > > 7 0 > > 0 1 203072 70696 1472 20460 96 0 168 280 196 1080 7 1 > 66 > > 26 0 > > 0 0 202900 68704 1512 21236 656 0 1432 104 338 760 10 1 > 10 > > 79 0 > > 0 0 201804 65728 1540 21984 1184 0 1904 64 547 1117 26 2 > 40 > > 33 0 > > 0 2 161904 122500 1540 22640 68 0 776 0 407 477 23 2 > 57 > > 18 0 > > 1 0 161384 122132 1556 22748 36 0 92 60 970 200 92 0 > 0 > > 8 0 > > 0 1 160840 119512 1836 23228 676 0 1432 76 772 866 58 2 > 0 > > 40 0 > > 0 0 160776 119636 1836 23516 196 0 500 0 104 199 1 0 > 63 > > 36 0 > > 0 0 160776 119636 1852 23520 0 0 0 44 83 251 2 0 > 92 > > 6 0 > > 0 0 160776 119644 1868 23504 0 0 0 40 64 117 0 1 > 90 > > 9 0 > > 0 0 160776 119644 1868 23520 0 0 0 0 46 91 0 0 > > 100 0 0 > > 0 1 160764 119456 1888 23556 28 0 32 164 71 121 0 0 > 87 > > 13 0 > > 0 0 160764 119208 1952 23572 0 0 4 288 392 1083 4 1 > 66 > > 29 0 > > 0 0 160764 119192 1952 23596 0 0 0 0 42 69 0 0 > > 100 0 0 > > 0 0 160764 119192 1968 23596 0 0 0 40 60 127 1 0 > 92 > > 7 0 > > 0 0 160764 119192 1984 23584 0 0 4 36 71 135 0 1 > 91 > > 8 0 > > 0 0 160764 119192 1984 23600 0 0 0 0 46 89 0 0 > > 100 0 0 > > 0 0 160764 119192 2000 23600 0 0 0 36 59 121 1 0 > 92 > > 7 0 > > 0 0 160764 119192 2016 23584 0 0 0 36 82 196 0 0 > 93 > > 7 0 > > 0 0 160764 119192 2016 23600 0 0 0 0 38 69 0 0 > > 100 0 0 > > 0 0 160764 119192 2032 23600 0 0 0 36 63 130 1 0 > 91 > > 8 0 > > 0 0 160764 119192 2048 23584 0 0 0 36 67 132 0 0 > 94 > > 6 0 > > 0 0 160764 119192 2096 23584 0 0 0 272 89 193 0 0 > 76 > > 24 0 > > > > > > ...Parth > > > > > > On 2/20/13 8:59 PM, "Marcus Sorensen" <shadow...@gmail.com> wrote: > > > >>Well, it doesn't seem to be actively swapping at this point, but I > >>think it's got active memory swapped out and being used as > >>occasionally wait% goes up significantly. At any rate this system is > >>severely memory limited. > >> > >>On Wed, Feb 20, 2013 at 9:52 PM, Parth Jagirdar > >><parth.jagir...@citrix.com> wrote: > >>> Marcus, > >>> > >>> vmstat 1 output > >>> > >>> > >>> [root@localhost management]# vmstat 1 procs > >>>-----------memory---------- ---swap-- -----io---- --system-- > >>> -----cpu----- > >>> r b swpd free buff cache si so bi bo in cs us sy > >>>id > >>> wa st > >>> 0 1 190820 72380 10904 15852 36 28 40 92 9 1 0 0 > >>>89 > >>> 10 0 > >>> 0 0 190820 72256 10932 15828 0 0 0 56 63 130 0 0 > >>>88 > >>> 12 0 > >>> 1 0 190820 72256 10932 15844 0 0 0 0 53 153 1 0 > >>>99 > >>> 0 0 > >>> 0 0 190820 72256 10948 15844 0 0 0 44 89 253 2 0 > >>>88 > >>> 10 0 > >>> 0 0 190820 72256 10964 15828 0 0 0 72 64 135 0 0 > >>>88 > >>> 12 0 > >>> 0 0 190820 72256 10964 15844 0 0 0 0 43 76 0 0 > >>> 100 0 0 > >>> 0 0 190820 72256 10980 15844 0 0 0 36 86 244 1 1 > >>>91 > >>> 7 0 > >>> 0 0 190820 72256 10996 15828 0 0 0 44 57 112 0 1 > >>>88 > >>> 11 0 > >>> 0 0 190820 72256 10996 15844 0 0 0 0 45 88 0 0 > >>> 100 0 0 > >>> 0 0 190820 72256 11012 15844 0 0 0 36 100 264 1 1 > >>>91 > >>> 7 0 > >>> 0 0 190820 72132 11044 15824 0 0 4 96 106 211 1 0 > >>>80 > >>> 19 0 > >>> 0 0 190820 72132 11092 15856 0 0 0 368 81 223 0 1 > >>>74 > >>> 25 0 > >>> 0 0 190820 72132 11108 15856 0 0 0 36 78 145 0 1 > >>>93 > >>> 6 0 > >>> 0 0 190820 72132 11124 15840 0 0 0 40 55 106 1 0 > >>>90 > >>> 9 0 > >>> 0 0 190820 72132 11124 15856 0 0 0 0 47 96 0 0 > >>> 100 0 0 > >>> 0 0 190820 72132 11140 15856 0 0 0 36 61 113 0 0 > >>>85 > >>> 15 0 > >>> 0 0 190820 72008 11156 15840 0 0 0 36 61 158 0 0 > >>>93 > >>> 7 0 > >>> 0 0 190820 72008 11156 15856 0 0 0 0 41 82 0 0 > >>> 100 0 0 > >>> 0 0 190820 72008 11172 15856 0 0 0 36 74 149 1 0 > >>>94 > >>> 5 0 > >>> 0 0 190820 72008 11188 15840 0 0 0 36 60 117 0 0 > >>>93 > >>> 7 0 > >>> 0 0 190820 72008 11188 15856 0 0 0 0 43 91 0 0 > >>> 100 0 0 > >>> 1 0 190820 72008 11252 15860 0 0 4 312 108 243 1 1 > >>>68 > >>> 30 0 > >>> 0 0 190820 72008 11268 15844 0 0 0 36 60 128 0 0 > >>>92 > >>> 8 0 > >>> 0 0 190820 72008 11268 15860 0 0 0 0 36 67 0 0 > >>> 100 0 0 > >>> 0 0 190820 71884 11284 15860 0 0 0 104 84 139 0 1 > >>>83 > >>> 16 0 > >>> 0 0 190820 71884 11300 15844 0 0 0 60 55 111 0 0 > >>>69 > >>> 31 0 > >>> 0 0 190820 71884 11300 15860 0 0 0 0 53 121 1 0 > >>>99 > >>> 0 0 > >>> 0 0 190820 71884 11316 15860 0 0 0 40 67 130 0 0 > >>>87 > >>> 13 0 > >>> 0 0 190820 71884 11332 15844 0 0 0 40 58 130 0 0 > >>>90 > >>> 10 0 > >>> 0 0 190820 71884 11332 15864 0 0 0 0 59 824 1 1 > >>>98 > >>> 0 0 > >>> 1 0 190820 71884 11348 15864 0 0 0 40 113 185 1 0 > >>>67 > >>> 32 0 > >>> 0 0 190820 71744 11412 15852 0 0 4 540 100 238 0 0 > >>>67 > >>> 33 0 > >>> 0 0 190820 71744 11412 15868 0 0 0 0 55 159 1 0 > >>>99 > >>> 0 0 > >>> 0 0 190820 71744 11428 15868 0 0 0 40 89 246 2 1 > >>>90 > >>> 7 0 > >>> 0 0 190820 71620 11444 15852 0 0 0 72 65 135 0 0 > >>>93 > >>> 7 0 > >>> 0 0 190820 71620 11444 15868 0 0 0 0 40 74 0 0 > >>> 100 0 0 > >>> 0 0 190820 71620 11460 15868 0 0 0 52 75 216 1 0 > >>>92 > >>> 7 0 > >>> 0 0 190820 71620 11476 15852 0 0 0 44 53 109 0 0 > >>>89 > >>> 11 0 > >>> 0 0 190820 71620 11476 15868 0 0 0 0 43 87 0 0 > >>> 100 0 0 > >>> 0 0 190820 71620 11496 15868 0 0 4 36 83 143 0 1 > >>>90 > >>> 9 0 > >>> 0 0 190820 71620 11512 15852 0 0 0 40 78 869 1 0 > >>>91 > >>> 8 0 > >>> 0 1 190820 71628 11524 15856 0 0 0 188 94 145 0 0 > >>>87 > >>> 13 0 > >>> 0 0 190820 71496 11576 15872 0 0 4 132 96 214 1 0 > >>>80 > >>> 19 0 > >>> 0 0 190820 71496 11592 15856 0 0 0 36 94 128 1 0 > >>>92 > >>> 7 0 > >>> 0 0 190820 71496 11592 15872 0 0 0 0 115 164 0 0 > >>> 100 0 0 > >>> 0 0 190820 71496 11608 15876 0 0 0 36 130 200 0 0 > >>>87 > >>> 13 0 > >>> 0 0 190820 71496 11624 15860 0 0 0 36 141 218 1 1 > >>>91 > >>> 7 0 > >>> 0 0 190820 71504 11624 15876 0 0 0 0 105 119 0 0 > >>> 100 0 0 > >>> 0 0 190820 71504 11640 15876 0 0 0 36 140 218 1 0 > >>>90 > >>> 9 0 > >>> procs -----------memory---------- ---swap-- -----io---- --system-- > >>> -----cpu----- > >>> r b swpd free buff cache si so bi bo in cs us sy > >>>id > >>> wa st > >>> 0 0 190820 71504 11656 15860 0 0 0 36 131 169 1 0 > >>>92 > >>> 7 0 > >>> 0 0 190820 71504 11656 15876 0 0 0 0 115 146 0 0 > >>> 100 0 0 > >>> 0 0 190820 71380 11672 15876 0 0 0 36 128 173 0 1 > >>>91 > >>> 8 0 > >>> 0 0 190820 71380 11736 15860 0 0 0 308 146 279 1 0 > >>>69 > >>> 30 0 > >>> 0 0 190820 71380 11736 15876 0 0 0 0 59 82 0 0 > >>> 100 0 0 > >>> 0 0 190820 71380 11760 15876 0 0 4 64 90 174 1 1 > >>>86 > >>> 12 0 > >>> > >>> ...Parth > >>> > >>> > >>> > >>> > >>> On 2/20/13 8:46 PM, "Parth Jagirdar" <parth.jagir...@citrix.com> > wrote: > >>> > >>>>JAVA_OPTS="-Djava.awt.headless=true > >>>>-Dcom.sun.management.jmxremote.port=45219 > >>>>-Dcom.sun.management.jmxremote.authenticate=false > >>>>-Dcom.sun.management.jmxremote.ssl=false -Xmx512m -Xms512m > >>>>-XX:+HeapDumpOnOutOfMemoryError > >>>>-XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=256M" > >>>> > >>>>Which did not help. > >>>> > >>>>-------------- > >>>> > >>>>[root@localhost management]# cat /proc/meminfo > >>>>MemTotal: 1016656 kB > >>>>MemFree: 68400 kB > >>>>Buffers: 9108 kB > >>>>Cached: 20984 kB > >>>>SwapCached: 17492 kB > >>>>Active: 424152 kB > >>>>Inactive: 433152 kB > >>>>Active(anon): 409812 kB > >>>>Inactive(anon): 417412 kB > >>>>Active(file): 14340 kB > >>>>Inactive(file): 15740 kB > >>>>Unevictable: 0 kB > >>>>Mlocked: 0 kB > >>>>SwapTotal: 2031608 kB > >>>>SwapFree: 1840900 kB > >>>>Dirty: 80 kB > >>>>Writeback: 0 kB > >>>>AnonPages: 815460 kB > >>>>Mapped: 11408 kB > >>>>Shmem: 4 kB > >>>>Slab: 60120 kB > >>>>SReclaimable: 10368 kB > >>>>SUnreclaim: 49752 kB > >>>>KernelStack: 5216 kB > >>>>PageTables: 6800 kB > >>>>NFS_Unstable: 0 kB > >>>>Bounce: 0 kB > >>>>WritebackTmp: 0 kB > >>>>CommitLimit: 2539936 kB > >>>>Committed_AS: 1596896 kB > >>>>VmallocTotal: 34359738367 kB > >>>>VmallocUsed: 7724 kB > >>>>VmallocChunk: 34359718200 kB > >>>>HardwareCorrupted: 0 kB > >>>>AnonHugePages: 503808 kB > >>>>HugePages_Total: 0 > >>>>HugePages_Free: 0 > >>>>HugePages_Rsvd: 0 > >>>>HugePages_Surp: 0 > >>>>Hugepagesize: 2048 kB > >>>>DirectMap4k: 6144 kB > >>>>DirectMap2M: 1038336 kB > >>>>[root@localhost management]# > >>>>----------------------------- > >>>> > >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >>>> > >>>> > >>>> 9809 cloud 20 0 2215m 785m 4672 S 0.7 79.1 1:59.40 java > >>>> > >>>> > >>>> 1497 mysql 20 0 700m 15m 3188 S 0.3 1.5 23:04.58 mysqld > >>>> > >>>> > >>>> 1 root 20 0 19348 300 296 S 0.0 0.0 0:00.73 init > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>On 2/20/13 8:26 PM, "Sailaja Mada" <sailaja.m...@citrix.com> wrote: > >>>> > >>>>>Hi, > >>>>> > >>>>>Cloudstack Java process statistics are given below when it stops > >>>>>responding are given below : > >>>>> > >>>>>top - 09:52:03 up 4 days, 21:43, 2 users, load average: 0.06, > >>>>>0.05, > >>>>>0.02 > >>>>>Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie > >>>>>Cpu(s): 1.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.3%wa, 0.0%hi, > >>>>>0.0%si, 0.0%st > >>>>>Mem: 1014860k total, 947632k used, 67228k free, 5868k > >>>>>buffers > >>>>>Swap: 2031608k total, 832320k used, 1199288k free, 26764k > cached > >>>>> > >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > >>>>>12559 cloud 20 0 3159m 744m 4440 S 2.3 75.1 6:38.39 java > >>>>> > >>>>>Thanks, > >>>>>Sailaja.M > >>>>> > >>>>>-----Original Message----- > >>>>>From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>>>>Sent: Thursday, February 21, 2013 9:35 AM > >>>>>To: cloudstack-dev@incubator.apache.org > >>>>>Subject: Re: [DISCUSS] Management Server Memory Requirements > >>>>> > >>>>>Yes, these are great data points, but so far nobody has responded > >>>>>on that ticket with the information required to know if the > >>>>>slowness is related to memory settings or swapping. That was just a > >>>>>hunch on my part from being a system admin. > >>>>> > >>>>>How much memory do these systems have that experience issues? What > >>>>>does /proc/meminfo say during the issues? Does adjusting the > >>>>>tomcat6.conf memory settings make a difference (see ticket > >>>>>comments)? How much memory do the java processes list as resident > >>>>>in top? > >>>>>On Feb 20, 2013 8:53 PM, "Parth Jagirdar" > >>>>><parth.jagir...@citrix.com> > >>>>>wrote: > >>>>> > >>>>>> +1 Performance degradation is dramatic and I too have observed > >>>>>> +this > >>>>>>issue. > >>>>>> > >>>>>> I have logged my comments into 1339. > >>>>>> > >>>>>> > >>>>>> ŠParth > >>>>>> > >>>>>> On 2/20/13 7:34 PM, "Srikanteswararao Talluri" > >>>>>> <srikanteswararao.tall...@citrix.com> wrote: > >>>>>> > >>>>>> >To add to what Marcus mentioned, Regarding bug CLOUDSTACK-1339 : > >>>>>> >I have observed this issue within > >>>>>> >5-10 min of starting management server and there has been a lot > >>>>>> >of API requests through automated tests. It is observed that > >>>>>> >Management server not only slows down but also goes down after a > while. > >>>>>> > > >>>>>> >~Talluri > >>>>>> > > >>>>>> >-----Original Message----- > >>>>>> >From: Marcus Sorensen [mailto:shadow...@gmail.com] > >>>>>> >Sent: Thursday, February 21, 2013 7:22 > >>>>>> >To: cloudstack-dev@incubator.apache.org > >>>>>> >Subject: [DISCUSS] Management Server Memory Requirements > >>>>>> > > >>>>>> >When Javelin was merged, there was an email sent out stating > >>>>>> >that devs should set their MAVEN_OPTS to use 2g of heap, and > >>>>>> >512M of permanent memory. Subsequently, there have also been > >>>>>> >several > >>>>>>e-mails > >>>>>> >and issues where devs have echoed this recommendation, and > >>>>>>presumably > >>>>>> >it fixed issues. I've seen the MS run out of memory myself and > >>>>>> >applied those recommendations. > >>>>>> > > >>>>>> >Is this what we want to provide in the tomcat config for a > >>>>>> >package based install as well? It's effectively saying that the > >>>>>> >minimum requirements for the management server are something > >>>>>> >like > >>>>>> >3 or 4 GB (to be safe for other running tasks) of RAM, right? > >>>>>> > > >>>>>> >There is currently a bug filed that may or may not have to do > >>>>>> >with this, CLOUDSTACK-1339. Users report mgmt server slowness, > >>>>>> >going unresponsive for minutes at a time, but the logs seem to > >>>>>> >show business as usual. User reports that java is taking 75% of > >>>>>> >RAM, depending on what else is going on they may be swapping. > >>>>>> >Settings in the code for an install are currently at 2g/512M, > >>>>>> >I've been running this on a 4GB server for awhile now, java is > >>>>>> >at 900M, but I haven't been pounding it with requests or anything. > >>>>>> > > >>>>>> >This bug might not have anything to do with the memory settings, > >>>>>> >but I figured it would be good to nail down what our minimum > >>>>>>requirements > >>>>>> >are for 4.1 > >>>>>> > >>>>>> > >>>> > >>> > > >
