Hello. We have a very large Jenkins set up that includes on master node with 100+ slaves and 1000+ jobs. We have reasons for keeping just a single master node so it isn't possible split our Hudson master.
Now, we are experiencing performance problems(minor gc happens frequently and it is performed over 1~2 minutes and it made stop-the-world.) However, full gc is performed within 20~30 seconds. Our heap size is over 100G so it is hard to generate and analysis heap dump. Does anyone have any experience with very large Hudson installations like this? Is there any advice for tuning or recommendations for this issue? Also, please do let me know if there is any other data that I can provide that would help with analysis. Thanks for any help you can provide. ---------------------------------------------------------------- Jenkins info ---------------------------------------------------------------- Core ver : 1.424.6 WAS : weblogic 10.3.2 JAVA : jdk1.6.0.34 JVM OPTION : -Xms180g -Xmx180g -XX:NewSize=140g -XX:MaxNewSize=140g -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:-UseGCOverheadLimit -XX:+UseParallelGC -XX:SurvivorRatio=8 -verbosegc -Xloggc:app_gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Djava.awt.headless=true ---------------------------------------------------------------- Server spec ---------------------------------------------------------------- CPU : Intel(R) Xeon(R) CPU E5-2690 2.90GHz * 4 (32 core) RAM : 256GB ---------------------------------------------------------------- Gc log ---------------------------------------------------------------- 60286.042: [GC [PSYoungGen: 8655292K->8714K(40587584K)] 25909185K->17262608K(208359744K), 0.0248360 secs] [Times: user=0.38 sys=0.01, real=0.03 secs] 60286.067: [Full GC (System) [PSYoungGen: 8714K->0K(40587584K)] [ParOldGen: 17253893K->17228395K(167772160K)] 17262608K->17228395K(208359744K) [PSPermGen: 194622K->194622K(2097152K)], 1.8638320 secs] [Times: user=33.26 sys=0.23, real=1.86 secs] 60748.860: [GC [PSYoungGen: 39173056K->532623K(40528512K)] 56401451K->17761019K(208300672K), 0.0837520 secs] [Times: user=1.19 sys=0.00, real=0.08 secs] 61243.483: [GC [PSYoungGen: 39705679K->29759K(40658432K)] 56934075K->17272524K(208430592K), 0.0558890 secs] [Times: user=0.49 sys=0.00, real=0.05 secs] 61805.663: [GC [PSYoungGen: 39346943K->28331K(40601792K)] 56589708K->17275705K(208373952K), 0.0544110 secs] [Times: user=0.49 sys=0.01, real=0.06 secs] 62383.664: [GC [PSYoungGen: 39345515K->33640K(40776640K)] 56592889K->17284373K(208548800K), 0.0592330 secs] [Times: user=0.49 sys=0.00, real=0.06 secs] .......................... 85842.953: [GC [PSYoungGen: 38973565K->1818337K(40038592K)] 80709421K->44276973K(207810752K), 22.0442750 secs] [Times: user=2.44 sys=503.41, real=22.04 secs] 85976.095: [GC [PSYoungGen: 40038561K->1904445K(37126592K)] 82497204K->46320890K(204898752K), 49.0663710 secs] [Times: user=2.88 sys=1117.05, real=49.06 secs] 86147.499: [GC [PSYoungGen: 37126456K->1721075K(38582592K)] 81542901K->48037517K(206354752K), 39.6267960 secs] [Times: user=2.81 sys=904.88, real=39.62 secs] 86265.898: [GC [PSYoungGen: 36943219K->1147657K(38796608K)] 83259661K->49166685K(206568768K), 43.2677960 secs] [Times: user=6.13 sys=985.33, real=43.26 secs] 86435.957: [GC [PSYoungGen: 36592456K->748179K(38591488K)] 84611484K->49915859K(206363648K), 34.1037910 secs] [Times: user=2.48 sys=780.02, real=34.10 secs] 86560.263: [GC [PSYoungGen: 36192756K->448475K(38982464K)] 85360436K->50343633K(206754624K), 27.3025220 secs] [Times: user=1.64 sys=623.52, real=27.29 secs] 86594.685: [GC [PSYoungGen: 36402298K->106372K(38914816K)] 86297455K->50438940K(206686976K), 15.7548480 secs] [Times: user=1.88 sys=359.23, real=15.76 secs]
