They are showing up as completed? Is this correct:
Pool Name Active Pending Completed STREAM-STAGE 0 0 0 RESPONSE-STAGE 0 0 0 ROW-READ-STAGE 0 0 517446 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 0 0 0 GMFD 0 0 0 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 0 ROW-MUTATION-STAGE 0 0 1353622 MESSAGE-STREAMING-POOL 0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 0 FLUSH-WRITER-POOL 0 0 0 AE-SERVICE-STAGE 0 0 0 On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > you need to figure out where the memory is going. check tpstats, if > the pending ops are large somewhere that means you're just generating > insert ops faster than it can handle. > > On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >> note: I'm using the Thrift API to insert. The commitLog directory >> continues to grow. The heap size continues to grow as well. >> >> I decreased MemtableSizeInMB size, but noticed no changes. Any idea >> what is causing this, and/or what property i need to tweek to >> alleviate this? What is the "insert threshold"? >> >> I moved to a more powerful node as well, it still ended up failing >> just after a longer period. >> >> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >>> >>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>>> Currently running on a single node with intensive write operations. >>>> >>>> >>>> After running for a while... >>>> >>>> Client starts outputting: >>>> >>>> TimedOutException() >>>> at >>>> org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232) >>>> at >>>> org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670) >>>> at >>>> org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643) >>>> >>>> Cassandra starts outputting: >>>> >>>> INFO 08:08:49,864 Cassandra starting up... >>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving >>>> 220554976 used; max is 1190723584 >>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving >>>> 230245792 used; max is 1190723584 >>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving >>>> 250127792 used; max is 1190723584 >>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving >>>> 261657720 used; max is 1190723584 >>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving >>>> 273764040 used; max is 1190723584 >>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving >>>> 282912656 used; max is 1190723584 >>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving >>>> 294861824 used; max is 1190723584 >>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving >>>> 304045640 used; max is 1190723584 >>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving >>>> 312964344 used; max is 1190723584 >>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving >>>> 324961320 used; max is 1190723584 >>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving >>>> 345874144 used; max is 1190723584 >>>> INFO 08:20:27,357 Creating new commitlog segment >>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log >>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving >>>> 366587984 used; max is 1190723584 >>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving >>>> 384059832 used; max is 1190723584 >>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving >>>> 383631432 used; max is 1190723584 >>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving >>>> 399697888 used; max is 1190723584 >>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving >>>> 415428952 used; max is 1190723584 >>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving >>>> 433542656 used; max is 1190723584 >>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving >>>> 452030560 used; max is 1190723584 >>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving >>>> 464013992 used; max is 1190723584 >>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving >>>> 471978840 used; max is 1190723584 >>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving >>>> 475912120 used; max is 1190723584 >>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving >>>> 483742344 used; max is 1190723584 >>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving >>>> 491761672 used; max is 1190723584 >>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving >>>> 507442800 used; max is 1190723584 >>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving >>>> 523028304 used; max is 1190723584 >>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving >>>> 530865568 used; max is 1190723584 >>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving >>>> 542690216 used; max is 1190723584 >>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving >>>> 550728232 used; max is 1190723584 >>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving >>>> 573981584 used; max is 1190723584 >>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving >>>> 563891472 used; max is 1190723584 >>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving >>>> 575756040 used; max is 1190723584 >>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving >>>> 583631432 used; max is 1190723584 >>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving >>>> 595405816 used; max is 1190723584 >>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving >>>> 603389432 used; max is 1190723584 >>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving >>>> 615150584 used; max is 1190723584 >>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving >>>> 623125104 used; max is 1190723584 >>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving >>>> 634661008 used; max is 1190723584 >>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving >>>> 646272840 used; max is 1190723584 >>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving >>>> 653978688 used; max is 1190723584 >>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving >>>> 665462760 used; max is 1190723584 >>>> INFO 08:26:44,895 Creating new commitlog segment >>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log >>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving >>>> 673423168 used; max is 1190723584 >>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving >>>> 685277616 used; max is 1190723584 >>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving >>>> 692803112 used; max is 1190723584 >>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving >>>> 700213488 used; max is 1190723584 >>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving >>>> 708002984 used; max is 1190723584 >>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving >>>> 719547672 used; max is 1190723584 >>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving >>>> 727261696 used; max is 1190723584 >>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving >>>> 738760480 used; max is 1190723584 >>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving >>>> 746376392 used; max is 1190723584 >>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving >>>> 754102376 used; max is 1190723584 >>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving >>>> 757926896 used; max is 1190723584 >>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving >>>> 763964048 used; max is 1190723584 >>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving >>>> 771496792 used; max is 1190723584 >>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving >>>> 779088496 used; max is 1190723584 >>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving >>>> 786858536 used; max is 1190723584 >>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving >>>> 794395528 used; max is 1190723584 >>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving >>>> 805862000 used; max is 1190723584 >>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving >>>> 813708392 used; max is 1190723584 >>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving >>>> 821245568 used; max is 1190723584 >>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving >>>> 828614768 used; max is 1190723584 >>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving >>>> 835962544 used; max is 1190723584 >>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving >>>> 844183384 used; max is 1190723584 >>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving >>>> 847924056 used; max is 1190723584 >>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving >>>> 841384040 used; max is 1190723584 >>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving >>>> 839554448 used; max is 1190723584 >>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving >>>> 850851600 used; max is 1190723584 >>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving >>>> 869886032 used; max is 1190723584 >>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving >>>> 881346312 used; max is 1190723584 >>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving >>>> 892677840 used; max is 1190723584 >>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving >>>> 911084096 used; max is 1190723584 >>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving >>>> 922370800 used; max is 1190723584 >>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving >>>> 929916416 used; max is 1190723584 >>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving >>>> 941201152 used; max is 1190723584 >>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving >>>> 948631600 used; max is 1190723584 >>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving >>>> 956424384 used; max is 1190723584 >>>> INFO 08:33:14,864 Creating new commitlog segment >>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log >>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving >>>> 972976288 used; max is 1190723584 >>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving >>>> 981829584 used; max is 1190723584 >>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving >>>> 995907192 used; max is 1190723584 >>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving >>>> 1011914464 used; max is 1190723584 >>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408 >>>> reclaimed leaving 1015729984 used; max is 1190723584 >>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912 >>>> reclaimed leaving 1019176200 used; max is 1190723584 >>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088 >>>> reclaimed leaving 1022673936 used; max is 1190723584 >>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904 >>>> reclaimed leaving 1025979696 used; max is 1190723584 >>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072 >>>> reclaimed leaving 1028992352 used; max is 1190723584 >>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440 >>>> reclaimed leaving 1031961224 used; max is 1190723584 >>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568 >>>> reclaimed leaving 1034536576 used; max is 1190723584 >>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880 >>>> reclaimed leaving 1037266456 used; max is 1190723584 >>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432 >>>> reclaimed leaving 1039709528 used; max is 1190723584 >>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800 >>>> reclaimed leaving 1042230248 used; max is 1190723584 >>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936 >>>> reclaimed leaving 1044527600 used; max is 1190723584 >>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008 >>>> reclaimed leaving 1046621520 used; max is 1190723584 >>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336 >>>> reclaimed leaving 1048677040 used; max is 1190723584 >>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368 >>>> reclaimed leaving 1050563688 used; max is 1190723584 >>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112 >>>> reclaimed leaving 1052423824 used; max is 1190723584 >>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048 >>>> reclaimed leaving 1054138992 used; max is 1190723584 >>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560 >>>> reclaimed leaving 1055761280 used; max is 1190723584 >>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728 >>>> reclaimed leaving 1057201248 used; max is 1190723584 >>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792 >>>> reclaimed leaving 1058496000 used; max is 1190723584 >>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136 >>>> reclaimed leaving 1060011424 used; max is 1190723584 >>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792 >>>> reclaimed leaving 1061388688 used; max is 1190723584 >>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096 >>>> reclaimed leaving 1062658344 used; max is 1190723584 >>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488 >>>> reclaimed leaving 1063834408 used; max is 1190723584 >>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232 >>>> reclaimed leaving 1064800992 used; max is 1190723584 >>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288 >>>> reclaimed leaving 1065798256 used; max is 1190723584 >>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680 >>>> reclaimed leaving 1066867192 used; max is 1190723584 >>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304 >>>> reclaimed leaving 1067925416 used; max is 1190723584 >>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632 >>>> reclaimed leaving 1068886912 used; max is 1190723584 >>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184 >>>> reclaimed leaving 1069705264 used; max is 1190723584 >>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544 >>>> reclaimed leaving 1070568224 used; max is 1190723584 >>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752 >>>> reclaimed leaving 1071366040 used; max is 1190723584 >>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104 >>>> reclaimed leaving 1072168400 used; max is 1190723584 >>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264 >>>> reclaimed leaving 1072743152 used; max is 1190723584 >>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160 >>>> reclaimed leaving 1073334720 used; max is 1190723584 >>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152 >>>> reclaimed leaving 1073930576 used; max is 1190723584 >>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416 >>>> reclaimed leaving 1074548160 used; max is 1190723584 >>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544 >>>> reclaimed leaving 1075122368 used; max is 1190723584 >>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272 >>>> reclaimed leaving 1075699904 used; max is 1190723584 >>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552 >>>> reclaimed leaving 1076227648 used; max is 1190723584 >>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976 >>>> reclaimed leaving 1076747648 used; max is 1190723584 >>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584 >>>> reclaimed leaving 1077152296 used; max is 1190723584 >>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168 >>>> reclaimed leaving 1077567128 used; max is 1190723584 >>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704 >>>> reclaimed leaving 1077987824 used; max is 1190723584 >>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848 >>>> reclaimed leaving 1078393448 used; max is 1190723584 >>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136 >>>> reclaimed leaving 1078708784 used; max is 1190723584 >>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344 >>>> reclaimed leaving 1079074464 used; max is 1190723584 >>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992 >>>> reclaimed leaving 1079409016 used; max is 1190723584 >>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952 >>>> reclaimed leaving 1079749672 used; max is 1190723584 >>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768 >>>> reclaimed leaving 1080014112 used; max is 1190723584 >>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216 >>>> reclaimed leaving 1080278992 used; max is 1190723584 >>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912 >>>> reclaimed leaving 1080574456 used; max is 1190723584 >>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376 >>>> reclaimed leaving 1080884096 used; max is 1190723584 >>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512 >>>> reclaimed leaving 1081076888 used; max is 1190723584 >>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168 >>>> reclaimed leaving 1081313872 used; max is 1190723584 >>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168 >>>> reclaimed leaving 1081528880 used; max is 1190723584 >>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152 >>>> reclaimed leaving 1081704648 used; max is 1190723584 >>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216 >>>> reclaimed leaving 1081889512 used; max is 1190723584 >>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392 >>>> reclaimed leaving 1082072568 used; max is 1190723584 >>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376 >>>> reclaimed leaving 1082270680 used; max is 1190723584 >>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920 >>>> reclaimed leaving 1082522872 used; max is 1190723584 >>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392 >>>> reclaimed leaving 1082632368 used; max is 1190723584 >>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560 >>>> reclaimed leaving 1082794472 used; max is 1190723584 >>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024 >>>> reclaimed leaving 1082963008 used; max is 1190723584 >>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504 >>>> reclaimed leaving 1083117048 used; max is 1190723584 >>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192 >>>> reclaimed leaving 1083213304 used; max is 1190723584 >>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392 >>>> reclaimed leaving 1083332280 used; max is 1190723584 >>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808 >>>> reclaimed leaving 1083448744 used; max is 1190723584 >>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864 >>>> reclaimed leaving 1083601864 used; max is 1190723584 >>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520 >>>> reclaimed leaving 1083680168 used; max is 1190723584 >>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872 >>>> reclaimed leaving 1083814848 used; max is 1190723584 >>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648 >>>> reclaimed leaving 1083871152 used; max is 1190723584 >>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720 >>>> reclaimed leaving 1083984728 used; max is 1190723584 >>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056 >>>> reclaimed leaving 1084063504 used; max is 1190723584 >>>> >>> >> >
