Yes, there we see some swap thrashing.
On Wed, Feb 20, 2013 at 10:04 PM, Parth Jagirdar <parth.jagir...@citrix.com> wrote: > Marcus, > > > I attempted login into UI while running the log. > > [root@localhost management]# vmstat 1 > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu----- > r b swpd free buff cache si so bi bo in cs us sy id > wa st > 0 0 191132 70904 10340 16192 36 28 40 91 9 1 0 0 89 > 10 0 > 0 0 191132 70904 10340 16204 0 0 0 0 46 75 0 0 > 100 0 0 > 0 0 191132 70904 10356 16204 0 0 0 36 72 221 1 0 91 > 8 0 > 0 0 191132 70904 10372 16188 0 0 0 44 73 130 0 1 88 > 11 0 > 0 0 191132 70780 10420 16208 0 0 4 276 83 191 1 0 75 > 24 0 > 0 0 191132 70780 10452 16192 0 0 0 120 106 309 1 0 77 > 22 0 > 0 0 191132 70780 10468 16200 0 0 0 40 91 183 1 1 90 > 8 0 > 0 0 191132 70780 10468 16216 0 0 0 0 47 128 0 0 > 100 0 0 > 0 0 191132 70780 10484 16216 0 0 0 36 70 136 0 0 94 > 6 0 > 0 0 191132 70656 10500 16200 0 0 0 40 66 116 1 0 91 > 8 0 > 0 0 191132 70656 10500 16216 0 0 0 0 47 94 0 0 > 100 0 0 > 0 1 189504 66216 10400 17940 2192 100 3928 172 404 579 9 2 5 > 84 0 > 1 1 188772 60220 10552 21992 1000 0 5220 68 412 741 7 2 21 > 69 0 > 1 2 187352 49316 7832 30344 1660 32 10052 32 833 1015 28 3 0 > 69 0 > 0 4 188816 52420 1392 25716 1488 2872 3168 3240 663 870 19 3 0 > 78 0 > 1 1 187388 51808 1372 25040 2476 1104 3084 1260 675 813 15 3 0 > 82 0 > 0 1 187360 54040 1500 24980 32 0 1048 0 447 379 6 1 0 > 93 0 > 0 1 187360 53916 1516 25004 0 0 924 52 309 283 1 0 0 > 99 0 > 0 1 195476 64076 1272 20624 0 8116 32 8156 312 308 1 2 0 > 97 0 > 0 0 203084 71920 1264 19412 0 7608 0 7608 256 173 0 2 89 > 9 0 > 0 0 203076 71324 1376 20132 64 0 868 40 192 232 2 0 65 > 33 0 > 0 0 203076 71328 1392 20108 0 0 0 68 75 144 1 0 85 > 14 0 > 0 0 203076 71084 1392 20392 0 0 268 0 66 132 0 1 96 > 3 0 > 0 0 203076 71084 1408 20392 0 0 0 36 60 122 0 0 94 > 6 0 > 0 0 203076 71076 1424 20376 0 0 0 36 77 148 1 0 92 > 7 0 > 0 1 203072 70696 1472 20460 96 0 168 280 196 1080 7 1 66 > 26 0 > 0 0 202900 68704 1512 21236 656 0 1432 104 338 760 10 1 10 > 79 0 > 0 0 201804 65728 1540 21984 1184 0 1904 64 547 1117 26 2 40 > 33 0 > 0 2 161904 122500 1540 22640 68 0 776 0 407 477 23 2 57 > 18 0 > 1 0 161384 122132 1556 22748 36 0 92 60 970 200 92 0 0 > 8 0 > 0 1 160840 119512 1836 23228 676 0 1432 76 772 866 58 2 0 > 40 0 > 0 0 160776 119636 1836 23516 196 0 500 0 104 199 1 0 63 > 36 0 > 0 0 160776 119636 1852 23520 0 0 0 44 83 251 2 0 92 > 6 0 > 0 0 160776 119644 1868 23504 0 0 0 40 64 117 0 1 90 > 9 0 > 0 0 160776 119644 1868 23520 0 0 0 0 46 91 0 0 > 100 0 0 > 0 1 160764 119456 1888 23556 28 0 32 164 71 121 0 0 87 > 13 0 > 0 0 160764 119208 1952 23572 0 0 4 288 392 1083 4 1 66 > 29 0 > 0 0 160764 119192 1952 23596 0 0 0 0 42 69 0 0 > 100 0 0 > 0 0 160764 119192 1968 23596 0 0 0 40 60 127 1 0 92 > 7 0 > 0 0 160764 119192 1984 23584 0 0 4 36 71 135 0 1 91 > 8 0 > 0 0 160764 119192 1984 23600 0 0 0 0 46 89 0 0 > 100 0 0 > 0 0 160764 119192 2000 23600 0 0 0 36 59 121 1 0 92 > 7 0 > 0 0 160764 119192 2016 23584 0 0 0 36 82 196 0 0 93 > 7 0 > 0 0 160764 119192 2016 23600 0 0 0 0 38 69 0 0 > 100 0 0 > 0 0 160764 119192 2032 23600 0 0 0 36 63 130 1 0 91 > 8 0 > 0 0 160764 119192 2048 23584 0 0 0 36 67 132 0 0 94 > 6 0 > 0 0 160764 119192 2096 23584 0 0 0 272 89 193 0 0 76 > 24 0 > > > ...Parth > > > On 2/20/13 8:59 PM, "Marcus Sorensen" <shadow...@gmail.com> wrote: > >>Well, it doesn't seem to be actively swapping at this point, but I >>think it's got active memory swapped out and being used as >>occasionally wait% goes up significantly. At any rate this system is >>severely memory limited. >> >>On Wed, Feb 20, 2013 at 9:52 PM, Parth Jagirdar >><parth.jagir...@citrix.com> wrote: >>> Marcus, >>> >>> vmstat 1 output >>> >>> >>> [root@localhost management]# vmstat 1 >>> procs -----------memory---------- ---swap-- -----io---- --system-- >>> -----cpu----- >>> r b swpd free buff cache si so bi bo in cs us sy >>>id >>> wa st >>> 0 1 190820 72380 10904 15852 36 28 40 92 9 1 0 0 >>>89 >>> 10 0 >>> 0 0 190820 72256 10932 15828 0 0 0 56 63 130 0 0 >>>88 >>> 12 0 >>> 1 0 190820 72256 10932 15844 0 0 0 0 53 153 1 0 >>>99 >>> 0 0 >>> 0 0 190820 72256 10948 15844 0 0 0 44 89 253 2 0 >>>88 >>> 10 0 >>> 0 0 190820 72256 10964 15828 0 0 0 72 64 135 0 0 >>>88 >>> 12 0 >>> 0 0 190820 72256 10964 15844 0 0 0 0 43 76 0 0 >>> 100 0 0 >>> 0 0 190820 72256 10980 15844 0 0 0 36 86 244 1 1 >>>91 >>> 7 0 >>> 0 0 190820 72256 10996 15828 0 0 0 44 57 112 0 1 >>>88 >>> 11 0 >>> 0 0 190820 72256 10996 15844 0 0 0 0 45 88 0 0 >>> 100 0 0 >>> 0 0 190820 72256 11012 15844 0 0 0 36 100 264 1 1 >>>91 >>> 7 0 >>> 0 0 190820 72132 11044 15824 0 0 4 96 106 211 1 0 >>>80 >>> 19 0 >>> 0 0 190820 72132 11092 15856 0 0 0 368 81 223 0 1 >>>74 >>> 25 0 >>> 0 0 190820 72132 11108 15856 0 0 0 36 78 145 0 1 >>>93 >>> 6 0 >>> 0 0 190820 72132 11124 15840 0 0 0 40 55 106 1 0 >>>90 >>> 9 0 >>> 0 0 190820 72132 11124 15856 0 0 0 0 47 96 0 0 >>> 100 0 0 >>> 0 0 190820 72132 11140 15856 0 0 0 36 61 113 0 0 >>>85 >>> 15 0 >>> 0 0 190820 72008 11156 15840 0 0 0 36 61 158 0 0 >>>93 >>> 7 0 >>> 0 0 190820 72008 11156 15856 0 0 0 0 41 82 0 0 >>> 100 0 0 >>> 0 0 190820 72008 11172 15856 0 0 0 36 74 149 1 0 >>>94 >>> 5 0 >>> 0 0 190820 72008 11188 15840 0 0 0 36 60 117 0 0 >>>93 >>> 7 0 >>> 0 0 190820 72008 11188 15856 0 0 0 0 43 91 0 0 >>> 100 0 0 >>> 1 0 190820 72008 11252 15860 0 0 4 312 108 243 1 1 >>>68 >>> 30 0 >>> 0 0 190820 72008 11268 15844 0 0 0 36 60 128 0 0 >>>92 >>> 8 0 >>> 0 0 190820 72008 11268 15860 0 0 0 0 36 67 0 0 >>> 100 0 0 >>> 0 0 190820 71884 11284 15860 0 0 0 104 84 139 0 1 >>>83 >>> 16 0 >>> 0 0 190820 71884 11300 15844 0 0 0 60 55 111 0 0 >>>69 >>> 31 0 >>> 0 0 190820 71884 11300 15860 0 0 0 0 53 121 1 0 >>>99 >>> 0 0 >>> 0 0 190820 71884 11316 15860 0 0 0 40 67 130 0 0 >>>87 >>> 13 0 >>> 0 0 190820 71884 11332 15844 0 0 0 40 58 130 0 0 >>>90 >>> 10 0 >>> 0 0 190820 71884 11332 15864 0 0 0 0 59 824 1 1 >>>98 >>> 0 0 >>> 1 0 190820 71884 11348 15864 0 0 0 40 113 185 1 0 >>>67 >>> 32 0 >>> 0 0 190820 71744 11412 15852 0 0 4 540 100 238 0 0 >>>67 >>> 33 0 >>> 0 0 190820 71744 11412 15868 0 0 0 0 55 159 1 0 >>>99 >>> 0 0 >>> 0 0 190820 71744 11428 15868 0 0 0 40 89 246 2 1 >>>90 >>> 7 0 >>> 0 0 190820 71620 11444 15852 0 0 0 72 65 135 0 0 >>>93 >>> 7 0 >>> 0 0 190820 71620 11444 15868 0 0 0 0 40 74 0 0 >>> 100 0 0 >>> 0 0 190820 71620 11460 15868 0 0 0 52 75 216 1 0 >>>92 >>> 7 0 >>> 0 0 190820 71620 11476 15852 0 0 0 44 53 109 0 0 >>>89 >>> 11 0 >>> 0 0 190820 71620 11476 15868 0 0 0 0 43 87 0 0 >>> 100 0 0 >>> 0 0 190820 71620 11496 15868 0 0 4 36 83 143 0 1 >>>90 >>> 9 0 >>> 0 0 190820 71620 11512 15852 0 0 0 40 78 869 1 0 >>>91 >>> 8 0 >>> 0 1 190820 71628 11524 15856 0 0 0 188 94 145 0 0 >>>87 >>> 13 0 >>> 0 0 190820 71496 11576 15872 0 0 4 132 96 214 1 0 >>>80 >>> 19 0 >>> 0 0 190820 71496 11592 15856 0 0 0 36 94 128 1 0 >>>92 >>> 7 0 >>> 0 0 190820 71496 11592 15872 0 0 0 0 115 164 0 0 >>> 100 0 0 >>> 0 0 190820 71496 11608 15876 0 0 0 36 130 200 0 0 >>>87 >>> 13 0 >>> 0 0 190820 71496 11624 15860 0 0 0 36 141 218 1 1 >>>91 >>> 7 0 >>> 0 0 190820 71504 11624 15876 0 0 0 0 105 119 0 0 >>> 100 0 0 >>> 0 0 190820 71504 11640 15876 0 0 0 36 140 218 1 0 >>>90 >>> 9 0 >>> procs -----------memory---------- ---swap-- -----io---- --system-- >>> -----cpu----- >>> r b swpd free buff cache si so bi bo in cs us sy >>>id >>> wa st >>> 0 0 190820 71504 11656 15860 0 0 0 36 131 169 1 0 >>>92 >>> 7 0 >>> 0 0 190820 71504 11656 15876 0 0 0 0 115 146 0 0 >>> 100 0 0 >>> 0 0 190820 71380 11672 15876 0 0 0 36 128 173 0 1 >>>91 >>> 8 0 >>> 0 0 190820 71380 11736 15860 0 0 0 308 146 279 1 0 >>>69 >>> 30 0 >>> 0 0 190820 71380 11736 15876 0 0 0 0 59 82 0 0 >>> 100 0 0 >>> 0 0 190820 71380 11760 15876 0 0 4 64 90 174 1 1 >>>86 >>> 12 0 >>> >>> ...Parth >>> >>> >>> >>> >>> On 2/20/13 8:46 PM, "Parth Jagirdar" <parth.jagir...@citrix.com> wrote: >>> >>>>JAVA_OPTS="-Djava.awt.headless=true >>>>-Dcom.sun.management.jmxremote.port=45219 >>>>-Dcom.sun.management.jmxremote.authenticate=false >>>>-Dcom.sun.management.jmxremote.ssl=false -Xmx512m -Xms512m >>>>-XX:+HeapDumpOnOutOfMemoryError >>>>-XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=256M" >>>> >>>>Which did not help. >>>> >>>>-------------- >>>> >>>>[root@localhost management]# cat /proc/meminfo >>>>MemTotal: 1016656 kB >>>>MemFree: 68400 kB >>>>Buffers: 9108 kB >>>>Cached: 20984 kB >>>>SwapCached: 17492 kB >>>>Active: 424152 kB >>>>Inactive: 433152 kB >>>>Active(anon): 409812 kB >>>>Inactive(anon): 417412 kB >>>>Active(file): 14340 kB >>>>Inactive(file): 15740 kB >>>>Unevictable: 0 kB >>>>Mlocked: 0 kB >>>>SwapTotal: 2031608 kB >>>>SwapFree: 1840900 kB >>>>Dirty: 80 kB >>>>Writeback: 0 kB >>>>AnonPages: 815460 kB >>>>Mapped: 11408 kB >>>>Shmem: 4 kB >>>>Slab: 60120 kB >>>>SReclaimable: 10368 kB >>>>SUnreclaim: 49752 kB >>>>KernelStack: 5216 kB >>>>PageTables: 6800 kB >>>>NFS_Unstable: 0 kB >>>>Bounce: 0 kB >>>>WritebackTmp: 0 kB >>>>CommitLimit: 2539936 kB >>>>Committed_AS: 1596896 kB >>>>VmallocTotal: 34359738367 kB >>>>VmallocUsed: 7724 kB >>>>VmallocChunk: 34359718200 kB >>>>HardwareCorrupted: 0 kB >>>>AnonHugePages: 503808 kB >>>>HugePages_Total: 0 >>>>HugePages_Free: 0 >>>>HugePages_Rsvd: 0 >>>>HugePages_Surp: 0 >>>>Hugepagesize: 2048 kB >>>>DirectMap4k: 6144 kB >>>>DirectMap2M: 1038336 kB >>>>[root@localhost management]# >>>>----------------------------- >>>> >>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>> >>>> >>>> 9809 cloud 20 0 2215m 785m 4672 S 0.7 79.1 1:59.40 java >>>> >>>> >>>> 1497 mysql 20 0 700m 15m 3188 S 0.3 1.5 23:04.58 mysqld >>>> >>>> >>>> 1 root 20 0 19348 300 296 S 0.0 0.0 0:00.73 init >>>> >>>> >>>> >>>> >>>> >>>>On 2/20/13 8:26 PM, "Sailaja Mada" <sailaja.m...@citrix.com> wrote: >>>> >>>>>Hi, >>>>> >>>>>Cloudstack Java process statistics are given below when it stops >>>>>responding are given below : >>>>> >>>>>top - 09:52:03 up 4 days, 21:43, 2 users, load average: 0.06, 0.05, >>>>>0.02 >>>>>Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie >>>>>Cpu(s): 1.7%us, 0.7%sy, 0.0%ni, 97.3%id, 0.3%wa, 0.0%hi, 0.0%si, >>>>>0.0%st >>>>>Mem: 1014860k total, 947632k used, 67228k free, 5868k >>>>>buffers >>>>>Swap: 2031608k total, 832320k used, 1199288k free, 26764k cached >>>>> >>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>>>>12559 cloud 20 0 3159m 744m 4440 S 2.3 75.1 6:38.39 java >>>>> >>>>>Thanks, >>>>>Sailaja.M >>>>> >>>>>-----Original Message----- >>>>>From: Marcus Sorensen [mailto:shadow...@gmail.com] >>>>>Sent: Thursday, February 21, 2013 9:35 AM >>>>>To: cloudstack-dev@incubator.apache.org >>>>>Subject: Re: [DISCUSS] Management Server Memory Requirements >>>>> >>>>>Yes, these are great data points, but so far nobody has responded on >>>>>that >>>>>ticket with the information required to know if the slowness is related >>>>>to memory settings or swapping. That was just a hunch on my part from >>>>>being a system admin. >>>>> >>>>>How much memory do these systems have that experience issues? What does >>>>>/proc/meminfo say during the issues? Does adjusting the tomcat6.conf >>>>>memory settings make a difference (see ticket comments)? How much >>>>>memory >>>>>do the java processes list as resident in top? >>>>>On Feb 20, 2013 8:53 PM, "Parth Jagirdar" <parth.jagir...@citrix.com> >>>>>wrote: >>>>> >>>>>> +1 Performance degradation is dramatic and I too have observed this >>>>>>issue. >>>>>> >>>>>> I have logged my comments into 1339. >>>>>> >>>>>> >>>>>> ŠParth >>>>>> >>>>>> On 2/20/13 7:34 PM, "Srikanteswararao Talluri" >>>>>> <srikanteswararao.tall...@citrix.com> wrote: >>>>>> >>>>>> >To add to what Marcus mentioned, >>>>>> >Regarding bug CLOUDSTACK-1339 : I have observed this issue within >>>>>> >5-10 min of starting management server and there has been a lot of >>>>>> >API requests through automated tests. It is observed that Management >>>>>> >server not only slows down but also goes down after a while. >>>>>> > >>>>>> >~Talluri >>>>>> > >>>>>> >-----Original Message----- >>>>>> >From: Marcus Sorensen [mailto:shadow...@gmail.com] >>>>>> >Sent: Thursday, February 21, 2013 7:22 >>>>>> >To: cloudstack-dev@incubator.apache.org >>>>>> >Subject: [DISCUSS] Management Server Memory Requirements >>>>>> > >>>>>> >When Javelin was merged, there was an email sent out stating that >>>>>> >devs should set their MAVEN_OPTS to use 2g of heap, and 512M of >>>>>> >permanent memory. Subsequently, there have also been several >>>>>>e-mails >>>>>> >and issues where devs have echoed this recommendation, and >>>>>>presumably >>>>>> >it fixed issues. I've seen the MS run out of memory myself and >>>>>> >applied those recommendations. >>>>>> > >>>>>> >Is this what we want to provide in the tomcat config for a package >>>>>> >based install as well? It's effectively saying that the minimum >>>>>> >requirements for the management server are something like 3 or 4 GB >>>>>> >(to be safe for other running tasks) of RAM, right? >>>>>> > >>>>>> >There is currently a bug filed that may or may not have to do with >>>>>> >this, CLOUDSTACK-1339. Users report mgmt server slowness, going >>>>>> >unresponsive for minutes at a time, but the logs seem to show >>>>>> >business as usual. User reports that java is taking 75% of RAM, >>>>>> >depending on what else is going on they may be swapping. Settings in >>>>>> >the code for an install are currently at 2g/512M, I've been running >>>>>> >this on a 4GB server for awhile now, java is at 900M, but I haven't >>>>>> >been pounding it with requests or anything. >>>>>> > >>>>>> >This bug might not have anything to do with the memory settings, but >>>>>> >I figured it would be good to nail down what our minimum >>>>>>requirements >>>>>> >are for 4.1 >>>>>> >>>>>> >>>> >>> >
