Marcus,
I attempted login into UI while running the log. [root@localhost management]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 191132 70904 10340 16192 36 28 40 91 9 1 0 0 89 10 0 0 0 191132 70904 10340 16204 0 0 0 0 46 75 0 0 100 0 0 0 0 191132 70904 10356 16204 0 0 0 36 72 221 1 0 91 8 0 0 0 191132 70904 10372 16188 0 0 0 44 73 130 0 1 88 11 0 0 0 191132 70780 10420 16208 0 0 4 276 83 191 1 0 75 24 0 0 0 191132 70780 10452 16192 0 0 0 120 106 309 1 0 77 22 0 0 0 191132 70780 10468 16200 0 0 0 40 91 183 1 1 90 8 0 0 0 191132 70780 10468 16216 0 0 0 0 47 128 0 0 100 0 0 0 0 191132 70780 10484 16216 0 0 0 36 70 136 0 0 94 6 0 0 0 191132 70656 10500 16200 0 0 0 40 66 116 1 0 91 8 0 0 0 191132 70656 10500 16216 0 0 0 0 47 94 0 0 100 0 0 0 1 189504 66216 10400 17940 2192 100 3928 172 404 579 9 2 5 84 0 1 1 188772 60220 10552 21992 1000 0 5220 68 412 741 7 2 21 69 0 1 2 187352 49316 7832 30344 1660 32 10052 32 833 1015 28 3 0 69 0 0 4 188816 52420 1392 25716 1488 2872 3168 3240 663 870 19 3 0 78 0 1 1 187388 51808 1372 25040 2476 1104 3084 1260 675 813 15 3 0 82 0 0 1 187360 54040 1500 24980 32 0 1048 0 447 379 6 1 0 93 0 0 1 187360 53916 1516 25004 0 0 924 52 309 283 1 0 0 99 0 0 1 195476 64076 1272 20624 0 8116 32 8156 312 308 1 2 0 97 0 0 0 203084 71920 1264 19412 0 7608 0 7608 256 173 0 2 89 9 0 0 0 203076 71324 1376 20132 64 0 868 40 192 232 2 0 65 33 0 0 0 203076 71328 1392 20108 0 0 0 68 75 144 1 0 85 14 0 0 0 203076 71084 1392 20392 0 0 268 0 66 132 0 1 96 3 0 0 0 203076 71084 1408 20392 0 0 0 36 60 122 0 0 94 6 0 0 0 203076 71076 1424 20376 0 0 0 36 77 148 1 0 92 7 0 0 1 203072 70696 1472 20460 96 0 168 280 196 1080 7 1 66 26 0 0 0 202900 68704 1512 21236 656 0 1432 104 338 760 10 1 10 79 0 0 0 201804 65728 1540 21984 1184 0 1904 64 547 1117 26 2 40 33 0 0 2 161904 122500 1540 22640 68 0 776 0 407 477 23 2 57 18 0 1 0 161384 122132 1556 22748 36 0 92 60 970 200 92 0 0 8 0 0 1 160840 119512 1836 23228 676 0 1432 76 772 866 58 2 0 40 0 0 0 160776 119636 1836 23516 196 0 500 0 104 199 1 0 63 36 0 0 0 160776 119636 1852 23520 0 0 0 44 83 251 2 0 92 6 0 0 0 160776 119644 1868 23504 0 0 0 40 64 117 0 1 90 9 0 0 0 160776 119644 1868 23520 0 0 0 0 46 91 0 0 100 0 0 0 1 160764 119456 1888 23556 28 0 32 164 71 121 0 0 87 13 0 0 0 160764 119208 1952 23572 0 0 4 288 392 1083 4 1 66 29 0 0 0 160764 119192 1952 23596 0 0 0 0 42 69 0 0 100 0 0 0 0 160764 119192 1968 23596 0 0 0 40 60 127 1 0 92 7 0 0 0 160764 119192 1984 23584 0 0 4 36 71 135 0 1 91 8 0 0 0 160764 119192 1984 23600 0 0 0 0 46 89 0 0 100 0 0 0 0 160764 119192 2000 23600 0 0 0 36 59 121 1 0 92 7 0 0 0 160764 119192 2016 23584 0 0 0 36 82 196 0 0 93 7 0 0 0 160764 119192 2016 23600 0 0 0 0 38 69 0 0 100 0 0 0 0 160764 119192 2032 23600 0 0 0 36 63 130 1 0 91 8 0 0 0 160764 119192 2048 23584 0 0 0 36 67 132 0 0 94 6 0 0 0 160764 119192 2096 23584 0 0 0 272 89 193 0 0 76 24 0 ...Parth On 2/20/13 8:59 PM, "Marcus Sorensen" <shadow...@gmail.com> wrote: >Well, it doesn't seem to be actively swapping at this point, but I >think it's got active memory swapped out and being used as >occasionally wait% goes up significantly. At any rate this system is >severely memory limited. > >On Wed, Feb 20, 2013 at 9:52 PM, Parth Jagirdar ><parth.jagir...@citrix.com> wrote: >> Marcus, >> >> vmstat 1 output >> >> >> [root@localhost management]# vmstat 1 >> procs -----------memory---------- ---swap-- -----io---- --system-- >> -----cpu----- >> r b swpd free buff cache si so bi bo in cs us sy >>id >> wa st >> 0 1 190820 72380 10904 15852 36 28 40 92 9 1 0 0 >>89 >> 10 0 >> 0 0 190820 72256 10932 15828 0 0 0 56 63 130 0 0 >>88 >> 12 0 >> 1 0 190820 72256 10932 15844 0 0 0 0 53 153 1 0 >>99 >> 0 0 >> 0 0 190820 72256 10948 15844 0 0 0 44 89 253 2 0 >>88 >> 10 0 >> 0 0 190820 72256 10964 15828 0 0 0 72 64 135 0 0 >>88 >> 12 0 >> 0 0 190820 72256 10964 15844 0 0 0 0 43 76 0 0 >> 100 0 0 >> 0 0 190820 72256 10980 15844 0 0 0 36 86 244 1 1 >>91 >> 7 0 >> 0 0 190820 72256 10996 15828 0 0 0 44 57 112 0 1 >>88 >> 11 0 >> 0 0 190820 72256 10996 15844 0 0 0 0 45 88 0 0 >> 100 0 0 >> 0 0 190820 72256 11012 15844 0 0 0 36 100 264 1 1 >>91 >> 7 0 >> 0 0 190820 72132 11044 15824 0 0 4 96 106 211 1 0 >>80 >> 19 0 >> 0 0 190820 72132 11092 15856 0 0 0 368 81 223 0 1 >>74 >> 25 0 >> 0 0 190820 72132 11108 15856 0 0 0 36 78 145 0 1 >>93 >> 6 0 >> 0 0 190820 72132 11124 15840 0 0 0 40 55 106 1 0 >>90 >> 9 0 >> 0 0 190820 72132 11124 15856 0 0 0 0 47 96 0 0 >> 100 0 0 >> 0 0 190820 72132 11140 15856 0 0 0 36 61 113 0 0 >>85 >> 15 0 >> 0 0 190820 72008 11156 15840 0 0 0 36 61 158 0 0 >>93 >> 7 0 >> 0 0 190820 72008 11156 15856 0 0 0 0 41 82 0 0 >> 100 0 0 >> 0 0 190820 72008 11172 15856 0 0 0 36 74 149 1 0 >>94 >> 5 0 >> 0 0 190820 72008 11188 15840 0 0 0 36 60 117 0 0 >>93 >> 7 0 >> 0 0 190820 72008 11188 15856 0 0 0 0 43 91 0 0 >> 100 0 0 >> 1 0 190820 72008 11252 15860 0 0 4 312 108 243 1 1 >>68 >> 30 0 >> 0 0 190820 72008 11268 15844 0 0 0 36 60 128 0 0 >>92 >> 8 0 >> 0 0 190820 72008 11268 15860 0 0 0 0 36 67 0 0 >> 100 0 0 >> 0 0 190820 71884 11284 15860 0 0 0 104 84 139 0 1 >>83 >> 16 0 >> 0 0 190820 71884 11300 15844 0 0 0 60 55 111 0 0 >>69 >> 31 0 >> 0 0 190820 71884 11300 15860 0 0 0 0 53 121 1 0 >>99 >> 0 0 >> 0 0 190820 71884 11316 15860 0 0 0 40 67 130 0 0 >>87 >> 13 0 >> 0 0 190820 71884 11332 15844 0 0 0 40 58 130 0 0 >>90 >> 10 0 >> 0 0 190820 71884 11332 15864 0 0 0 0 59 824 1 1 >>98 >> 0 0 >> 1 0 190820 71884 11348 15864 0 0 0 40 113 185 1 0 >>67 >> 32 0 >> 0 0 190820 71744 11412 15852 0 0 4 540 100 238 0 0 >>67 >> 33 0 >> 0 0 190820 71744 11412 15868 0 0 0 0 55 159 1 0 >>99 >> 0 0 >> 0 0 190820 71744 11428 15868 0 0 0 40 89 246 2 1 >>90 >> 7 0 >> 0 0 190820 71620 11444 15852 0 0 0 72 65 135 0 0 >>93 >> 7 0 >> 0 0 190820 71620 11444 15868 0 0 0 0 40 74 0 0 >> 100 0 0 >> 0 0 190820 71620 11460 15868 0 0 0 52 75 216 1 0 >>92 >> 7 0 >> 0 0 190820 71620 11476 15852 0 0 0 44 53 109 0 0 >>89 >> 11 0 >> 0 0 190820 71620 11476 15868 0 0 0 0 43 87 0 0 >> 100 0 0 >> 0 0 190820 71620 11496 15868 0 0 4 36 83 143 0 1 >>90 >> 9 0 >> 0 0 190820 71620 11512 15852 0 0 0 40 78 869 1 0 >>91 >> 8 0 >> 0 1 190820 71628 11524 15856 0 0 0 188 94 145 0 0 >>87 >> 13 0 >> 0 0 190820 71496 11576 15872 0 0 4 132 96 214 1 0 >>80 >> 19 0 >> 0 0 190820 71496 11592 15856 0 0 0 36 94 128 1 0 >>92 >> 7 0 >> 0 0 190820 71496 11592 15872 0 0 0 0 115 164 0 0 >> 100 0 0 >> 0 0 190820 71496 11608 15876 0 0 0 36 130 200 0 0 >>87 >> 13 0 >> 0 0 190820 71496 11624 15860 0 0 0 36 141 218 1 1 >>91 >> 7 0 >> 0 0 190820 71504 11624 15876 0 0 0 0 105 119 0 0 >> 100 0 0 >> 0 0 190820 71504 11640 15876 0 0 0 36 140 218 1 0 >>90 >> 9 0 >> procs -----------memory---------- ---swap-- -----io---- --system-- >> -----cpu----- >> r b swpd free buff cache si so bi bo in cs us sy >>id >> wa st >> 0 0 190820 71504 11656 15860 0 0 0 36 131 169 1 0 >>92 >> 7 0 >> 0 0 190820 71504 11656 15876 0 0 0 0 115 146 0 0 >> 100 0 0 >> 0 0 190820 71380 11672 15876 0 0 0 36 128 173 0 1 >>91 >> 8 0 >> 0 0 190820 71380 11736 15860 0 0 0 308 146 279 1 0 >>69 >> 30 0 >> 0 0 190820 71380 11736 15876 0 0 0 0 59 82 0 0 >> 100 0 0 >> 0 0 190820 71380 11760 15876 0 0 4 64 90 174 1 1 >>86 >> 12 0 >> >> ...Parth >> >> >> >> >> On 2/20/13 8:46 PM, "Parth Jagirdar" <parth.jagir...@citrix.com> wrote: >> >>>JAVA_OPTS="-Djava.awt.headless=true >>>-Dcom.sun.management.jmxremote.port=45219 >>>-Dcom.sun.management.jmxremote.authenticate=false >>>-Dcom.sun.management.jmxremote.ssl=false -Xmx512m -Xms512m >>>-XX:+HeapDumpOnOutOfMemoryError >>>-XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=256M" >>> >>>Which did not help. >>> >>>-------------- >>> >>>[root@localhost management]# cat /proc/meminfo >>>MemTotal: 1016656 kB >>>MemFree: 68400 kB >>>Buffers: 9108 kB >>>Cached: 20984 kB >>>SwapCached: 17492 kB >>>Active: 424152 kB >>>Inactive: 433152 kB >>>Active(anon): 409812 kB >>>Inactive(anon): 417412 kB >>>Active(file): 14340 kB >>>Inactive(file): 15740 kB >>>Unevictable: 0 kB >>>Mlocked: 0 kB >>>SwapTotal: 2031608 kB >>>SwapFree: 1840900 kB >>>Dirty: 80 kB >>>Writeback: 0 kB >>>AnonPages: 815460 kB >>>Mapped: 11408 kB >>>Shmem: 4 kB >>>Slab: 60120 kB >>>SReclaimable: 10368 kB >>>SUnreclaim: 49752 kB >>>KernelStack: 5216 kB >>>PageTables: 6800 kB >>>NFS_Unstable: 0 kB >>>Bounce: 0 kB >>>WritebackTmp: 0 kB >>>CommitLimit: 2539936 kB >>>Committed_AS: 1596896 kB >>>VmallocTotal: 34359738367 kB >>>VmallocUsed: 7724 kB >>>VmallocChunk: 34359718200 kB >>>HardwareCorrupted: 0 kB >>>AnonHugePages: 503808 kB >>>HugePages_Total: 0 >>>HugePages_Free: 0 >>>HugePages_Rsvd: 0 >>>HugePages_Surp: 0 >>>Hugepagesize: 2048 kB >>>DirectMap4k: 6144 kB >>>DirectMap2M: 1038336 kB >>>[root@localhost management]# >>>----------------------------- >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> >>> >>> 9809 cloud 20 0 2215m 785m 4672 S 0.7 79.1 1:59.40 java >>> >>> >>> 1497 mysql 20 0 700m 15m 3188 S 0.3 1.5 23:04.58 mysqld >>> >>> >>> 1 root 20 0 19348 300 296 S 0.0 0.0 0:00.73 init >>> >>> >>> >>> >>> >>>On 2/20/13 8:26 PM, "Sailaja Mada" <sailaja.m...@citrix.com> wrote: >>> >>>>Hi, >>>> >>>>Cloudstack Java process statistics are given below when it stops >>>>responding are given below : >>>> >>>>top - 09:52:03 up 4 days, 21:43, 2 users, load average: 0.06, 0.05, >>>>0.02 >>>>Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie >>>>Cpu(s): 1.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.3%wa, 0.0%hi, 0.0%si, >>>>0.0%st >>>>Mem: 1014860k total, 947632k used, 67228k free, 5868k >>>>buffers >>>>Swap: 2031608k total, 832320k used, 1199288k free, 26764k cached >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>12559 cloud 20 0 3159m 744m 4440 S 2.3 75.1 6:38.39 java >>>> >>>>Thanks, >>>>Sailaja.M >>>> >>>>-----Original Message----- >>>>From: Marcus Sorensen [mailto:shadow...@gmail.com] >>>>Sent: Thursday, February 21, 2013 9:35 AM >>>>To: cloudstack-dev@incubator.apache.org >>>>Subject: Re: [DISCUSS] Management Server Memory Requirements >>>> >>>>Yes, these are great data points, but so far nobody has responded on >>>>that >>>>ticket with the information required to know if the slowness is related >>>>to memory settings or swapping. That was just a hunch on my part from >>>>being a system admin. >>>> >>>>How much memory do these systems have that experience issues? What does >>>>/proc/meminfo say during the issues? Does adjusting the tomcat6.conf >>>>memory settings make a difference (see ticket comments)? How much >>>>memory >>>>do the java processes list as resident in top? >>>>On Feb 20, 2013 8:53 PM, "Parth Jagirdar" <parth.jagir...@citrix.com> >>>>wrote: >>>> >>>>> +1 Performance degradation is dramatic and I too have observed this >>>>>issue. >>>>> >>>>> I have logged my comments into 1339. >>>>> >>>>> >>>>> ŠParth >>>>> >>>>> On 2/20/13 7:34 PM, "Srikanteswararao Talluri" >>>>> <srikanteswararao.tall...@citrix.com> wrote: >>>>> >>>>> >To add to what Marcus mentioned, >>>>> >Regarding bug CLOUDSTACK-1339 : I have observed this issue within >>>>> >5-10 min of starting management server and there has been a lot of >>>>> >API requests through automated tests. It is observed that Management >>>>> >server not only slows down but also goes down after a while. >>>>> > >>>>> >~Talluri >>>>> > >>>>> >-----Original Message----- >>>>> >From: Marcus Sorensen [mailto:shadow...@gmail.com] >>>>> >Sent: Thursday, February 21, 2013 7:22 >>>>> >To: cloudstack-dev@incubator.apache.org >>>>> >Subject: [DISCUSS] Management Server Memory Requirements >>>>> > >>>>> >When Javelin was merged, there was an email sent out stating that >>>>> >devs should set their MAVEN_OPTS to use 2g of heap, and 512M of >>>>> >permanent memory. Subsequently, there have also been several >>>>>e-mails >>>>> >and issues where devs have echoed this recommendation, and >>>>>presumably >>>>> >it fixed issues. I've seen the MS run out of memory myself and >>>>> >applied those recommendations. >>>>> > >>>>> >Is this what we want to provide in the tomcat config for a package >>>>> >based install as well? It's effectively saying that the minimum >>>>> >requirements for the management server are something like 3 or 4 GB >>>>> >(to be safe for other running tasks) of RAM, right? >>>>> > >>>>> >There is currently a bug filed that may or may not have to do with >>>>> >this, CLOUDSTACK-1339. Users report mgmt server slowness, going >>>>> >unresponsive for minutes at a time, but the logs seem to show >>>>> >business as usual. User reports that java is taking 75% of RAM, >>>>> >depending on what else is going on they may be swapping. Settings in >>>>> >the code for an install are currently at 2g/512M, I've been running >>>>> >this on a 4GB server for awhile now, java is at 900M, but I haven't >>>>> >been pounding it with requests or anything. >>>>> > >>>>> >This bug might not have anything to do with the memory settings, but >>>>> >I figured it would be good to nail down what our minimum >>>>>requirements >>>>> >are for 4.1 >>>>> >>>>> >>> >>
