What does OOM stand for? for a given insert the size is small (meaning the a single insert operation only has about a sentence of data) although as the insert process continues, the columns under a given row key could potentially grow to be large. Is that what you mean?
An operation entails: Read Insert row (IE: rowkey: FOO columnName: BLAH value: 10) Delete a row Insert different row (IE: rowkey: FOO columnName: 0000010|Blah value: 10) millions of these operations are performed in sequence as files are read from a directory source. On Wed, Apr 21, 2010 at 2:37 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > then that's not the problem. > > are you writing large rows that OOM during compaction? > > On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >> They are showing up as completed? Is this correct: >> >> >> Pool Name Active Pending Completed >> STREAM-STAGE 0 0 0 >> RESPONSE-STAGE 0 0 0 >> ROW-READ-STAGE 0 0 517446 >> LB-OPERATIONS 0 0 0 >> MESSAGE-DESERIALIZER-POOL 0 0 0 >> GMFD 0 0 0 >> LB-TARGET 0 0 0 >> CONSISTENCY-MANAGER 0 0 0 >> ROW-MUTATION-STAGE 0 0 1353622 >> MESSAGE-STREAMING-POOL 0 0 0 >> LOAD-BALANCER-STAGE 0 0 0 >> FLUSH-SORTER-POOL 0 0 0 >> MEMTABLE-POST-FLUSHER 0 0 0 >> FLUSH-WRITER-POOL 0 0 0 >> AE-SERVICE-STAGE 0 0 0 >> >> >> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >>> you need to figure out where the memory is going. check tpstats, if >>> the pending ops are large somewhere that means you're just generating >>> insert ops faster than it can handle. >>> >>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>>> note: I'm using the Thrift API to insert. The commitLog directory >>>> continues to grow. The heap size continues to grow as well. >>>> >>>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea >>>> what is causing this, and/or what property i need to tweek to >>>> alleviate this? What is the "insert threshold"? >>>> >>>> I moved to a more powerful node as well, it still ended up failing >>>> just after a longer period. >>>> >>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jbel...@gmail.com> wrote: >>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >>>>> >>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>>>>> Currently running on a single node with intensive write operations. >>>>>> >>>>>> >>>>>> After running for a while... >>>>>> >>>>>> Client starts outputting: >>>>>> >>>>>> TimedOutException() >>>>>> at >>>>>> org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232) >>>>>> at >>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670) >>>>>> at >>>>>> org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643) >>>>>> >>>>>> Cassandra starts outputting: >>>>>> >>>>>> INFO 08:08:49,864 Cassandra starting up... >>>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving >>>>>> 220554976 used; max is 1190723584 >>>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving >>>>>> 230245792 used; max is 1190723584 >>>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving >>>>>> 250127792 used; max is 1190723584 >>>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving >>>>>> 261657720 used; max is 1190723584 >>>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving >>>>>> 273764040 used; max is 1190723584 >>>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving >>>>>> 282912656 used; max is 1190723584 >>>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving >>>>>> 294861824 used; max is 1190723584 >>>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving >>>>>> 304045640 used; max is 1190723584 >>>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving >>>>>> 312964344 used; max is 1190723584 >>>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving >>>>>> 324961320 used; max is 1190723584 >>>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving >>>>>> 345874144 used; max is 1190723584 >>>>>> INFO 08:20:27,357 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log >>>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving >>>>>> 366587984 used; max is 1190723584 >>>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving >>>>>> 384059832 used; max is 1190723584 >>>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving >>>>>> 383631432 used; max is 1190723584 >>>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving >>>>>> 399697888 used; max is 1190723584 >>>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving >>>>>> 415428952 used; max is 1190723584 >>>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving >>>>>> 433542656 used; max is 1190723584 >>>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving >>>>>> 452030560 used; max is 1190723584 >>>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving >>>>>> 464013992 used; max is 1190723584 >>>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving >>>>>> 471978840 used; max is 1190723584 >>>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving >>>>>> 475912120 used; max is 1190723584 >>>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving >>>>>> 483742344 used; max is 1190723584 >>>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving >>>>>> 491761672 used; max is 1190723584 >>>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving >>>>>> 507442800 used; max is 1190723584 >>>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving >>>>>> 523028304 used; max is 1190723584 >>>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving >>>>>> 530865568 used; max is 1190723584 >>>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving >>>>>> 542690216 used; max is 1190723584 >>>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving >>>>>> 550728232 used; max is 1190723584 >>>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving >>>>>> 573981584 used; max is 1190723584 >>>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving >>>>>> 563891472 used; max is 1190723584 >>>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving >>>>>> 575756040 used; max is 1190723584 >>>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving >>>>>> 583631432 used; max is 1190723584 >>>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving >>>>>> 595405816 used; max is 1190723584 >>>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving >>>>>> 603389432 used; max is 1190723584 >>>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving >>>>>> 615150584 used; max is 1190723584 >>>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving >>>>>> 623125104 used; max is 1190723584 >>>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving >>>>>> 634661008 used; max is 1190723584 >>>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving >>>>>> 646272840 used; max is 1190723584 >>>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving >>>>>> 653978688 used; max is 1190723584 >>>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving >>>>>> 665462760 used; max is 1190723584 >>>>>> INFO 08:26:44,895 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log >>>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving >>>>>> 673423168 used; max is 1190723584 >>>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving >>>>>> 685277616 used; max is 1190723584 >>>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving >>>>>> 692803112 used; max is 1190723584 >>>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving >>>>>> 700213488 used; max is 1190723584 >>>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving >>>>>> 708002984 used; max is 1190723584 >>>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving >>>>>> 719547672 used; max is 1190723584 >>>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving >>>>>> 727261696 used; max is 1190723584 >>>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving >>>>>> 738760480 used; max is 1190723584 >>>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving >>>>>> 746376392 used; max is 1190723584 >>>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving >>>>>> 754102376 used; max is 1190723584 >>>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving >>>>>> 757926896 used; max is 1190723584 >>>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving >>>>>> 763964048 used; max is 1190723584 >>>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving >>>>>> 771496792 used; max is 1190723584 >>>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving >>>>>> 779088496 used; max is 1190723584 >>>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving >>>>>> 786858536 used; max is 1190723584 >>>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving >>>>>> 794395528 used; max is 1190723584 >>>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving >>>>>> 805862000 used; max is 1190723584 >>>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving >>>>>> 813708392 used; max is 1190723584 >>>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving >>>>>> 821245568 used; max is 1190723584 >>>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving >>>>>> 828614768 used; max is 1190723584 >>>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving >>>>>> 835962544 used; max is 1190723584 >>>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving >>>>>> 844183384 used; max is 1190723584 >>>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving >>>>>> 847924056 used; max is 1190723584 >>>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving >>>>>> 841384040 used; max is 1190723584 >>>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving >>>>>> 839554448 used; max is 1190723584 >>>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving >>>>>> 850851600 used; max is 1190723584 >>>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving >>>>>> 869886032 used; max is 1190723584 >>>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving >>>>>> 881346312 used; max is 1190723584 >>>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving >>>>>> 892677840 used; max is 1190723584 >>>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving >>>>>> 911084096 used; max is 1190723584 >>>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving >>>>>> 922370800 used; max is 1190723584 >>>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving >>>>>> 929916416 used; max is 1190723584 >>>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving >>>>>> 941201152 used; max is 1190723584 >>>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving >>>>>> 948631600 used; max is 1190723584 >>>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving >>>>>> 956424384 used; max is 1190723584 >>>>>> INFO 08:33:14,864 Creating new commitlog segment >>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log >>>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving >>>>>> 972976288 used; max is 1190723584 >>>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving >>>>>> 981829584 used; max is 1190723584 >>>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving >>>>>> 995907192 used; max is 1190723584 >>>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving >>>>>> 1011914464 used; max is 1190723584 >>>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408 >>>>>> reclaimed leaving 1015729984 used; max is 1190723584 >>>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912 >>>>>> reclaimed leaving 1019176200 used; max is 1190723584 >>>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088 >>>>>> reclaimed leaving 1022673936 used; max is 1190723584 >>>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904 >>>>>> reclaimed leaving 1025979696 used; max is 1190723584 >>>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072 >>>>>> reclaimed leaving 1028992352 used; max is 1190723584 >>>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440 >>>>>> reclaimed leaving 1031961224 used; max is 1190723584 >>>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568 >>>>>> reclaimed leaving 1034536576 used; max is 1190723584 >>>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880 >>>>>> reclaimed leaving 1037266456 used; max is 1190723584 >>>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432 >>>>>> reclaimed leaving 1039709528 used; max is 1190723584 >>>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800 >>>>>> reclaimed leaving 1042230248 used; max is 1190723584 >>>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936 >>>>>> reclaimed leaving 1044527600 used; max is 1190723584 >>>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008 >>>>>> reclaimed leaving 1046621520 used; max is 1190723584 >>>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336 >>>>>> reclaimed leaving 1048677040 used; max is 1190723584 >>>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368 >>>>>> reclaimed leaving 1050563688 used; max is 1190723584 >>>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112 >>>>>> reclaimed leaving 1052423824 used; max is 1190723584 >>>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048 >>>>>> reclaimed leaving 1054138992 used; max is 1190723584 >>>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560 >>>>>> reclaimed leaving 1055761280 used; max is 1190723584 >>>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728 >>>>>> reclaimed leaving 1057201248 used; max is 1190723584 >>>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792 >>>>>> reclaimed leaving 1058496000 used; max is 1190723584 >>>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136 >>>>>> reclaimed leaving 1060011424 used; max is 1190723584 >>>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792 >>>>>> reclaimed leaving 1061388688 used; max is 1190723584 >>>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096 >>>>>> reclaimed leaving 1062658344 used; max is 1190723584 >>>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488 >>>>>> reclaimed leaving 1063834408 used; max is 1190723584 >>>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232 >>>>>> reclaimed leaving 1064800992 used; max is 1190723584 >>>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288 >>>>>> reclaimed leaving 1065798256 used; max is 1190723584 >>>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680 >>>>>> reclaimed leaving 1066867192 used; max is 1190723584 >>>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304 >>>>>> reclaimed leaving 1067925416 used; max is 1190723584 >>>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632 >>>>>> reclaimed leaving 1068886912 used; max is 1190723584 >>>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184 >>>>>> reclaimed leaving 1069705264 used; max is 1190723584 >>>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544 >>>>>> reclaimed leaving 1070568224 used; max is 1190723584 >>>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752 >>>>>> reclaimed leaving 1071366040 used; max is 1190723584 >>>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104 >>>>>> reclaimed leaving 1072168400 used; max is 1190723584 >>>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264 >>>>>> reclaimed leaving 1072743152 used; max is 1190723584 >>>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160 >>>>>> reclaimed leaving 1073334720 used; max is 1190723584 >>>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152 >>>>>> reclaimed leaving 1073930576 used; max is 1190723584 >>>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416 >>>>>> reclaimed leaving 1074548160 used; max is 1190723584 >>>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544 >>>>>> reclaimed leaving 1075122368 used; max is 1190723584 >>>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272 >>>>>> reclaimed leaving 1075699904 used; max is 1190723584 >>>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552 >>>>>> reclaimed leaving 1076227648 used; max is 1190723584 >>>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976 >>>>>> reclaimed leaving 1076747648 used; max is 1190723584 >>>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584 >>>>>> reclaimed leaving 1077152296 used; max is 1190723584 >>>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168 >>>>>> reclaimed leaving 1077567128 used; max is 1190723584 >>>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704 >>>>>> reclaimed leaving 1077987824 used; max is 1190723584 >>>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848 >>>>>> reclaimed leaving 1078393448 used; max is 1190723584 >>>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136 >>>>>> reclaimed leaving 1078708784 used; max is 1190723584 >>>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344 >>>>>> reclaimed leaving 1079074464 used; max is 1190723584 >>>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992 >>>>>> reclaimed leaving 1079409016 used; max is 1190723584 >>>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952 >>>>>> reclaimed leaving 1079749672 used; max is 1190723584 >>>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768 >>>>>> reclaimed leaving 1080014112 used; max is 1190723584 >>>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216 >>>>>> reclaimed leaving 1080278992 used; max is 1190723584 >>>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912 >>>>>> reclaimed leaving 1080574456 used; max is 1190723584 >>>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376 >>>>>> reclaimed leaving 1080884096 used; max is 1190723584 >>>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512 >>>>>> reclaimed leaving 1081076888 used; max is 1190723584 >>>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168 >>>>>> reclaimed leaving 1081313872 used; max is 1190723584 >>>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168 >>>>>> reclaimed leaving 1081528880 used; max is 1190723584 >>>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152 >>>>>> reclaimed leaving 1081704648 used; max is 1190723584 >>>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216 >>>>>> reclaimed leaving 1081889512 used; max is 1190723584 >>>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392 >>>>>> reclaimed leaving 1082072568 used; max is 1190723584 >>>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376 >>>>>> reclaimed leaving 1082270680 used; max is 1190723584 >>>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920 >>>>>> reclaimed leaving 1082522872 used; max is 1190723584 >>>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392 >>>>>> reclaimed leaving 1082632368 used; max is 1190723584 >>>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560 >>>>>> reclaimed leaving 1082794472 used; max is 1190723584 >>>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024 >>>>>> reclaimed leaving 1082963008 used; max is 1190723584 >>>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504 >>>>>> reclaimed leaving 1083117048 used; max is 1190723584 >>>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192 >>>>>> reclaimed leaving 1083213304 used; max is 1190723584 >>>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392 >>>>>> reclaimed leaving 1083332280 used; max is 1190723584 >>>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808 >>>>>> reclaimed leaving 1083448744 used; max is 1190723584 >>>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864 >>>>>> reclaimed leaving 1083601864 used; max is 1190723584 >>>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520 >>>>>> reclaimed leaving 1083680168 used; max is 1190723584 >>>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872 >>>>>> reclaimed leaving 1083814848 used; max is 1190723584 >>>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648 >>>>>> reclaimed leaving 1083871152 used; max is 1190723584 >>>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720 >>>>>> reclaimed leaving 1083984728 used; max is 1190723584 >>>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056 >>>>>> reclaimed leaving 1084063504 used; max is 1190723584 >>>>>> >>>>> >>>> >>> >> >
