what i mean by as data is processed is that the column size will grow in cassandra, but my client isn't ever writing large column size under a given row...
Any idea whats going on here? On Wed, Apr 21, 2010 at 3:05 PM, Sonny Heer <sonnyh...@gmail.com> wrote: > What does OOM stand for? > > for a given insert the size is small (meaning the a single insert > operation only has about a sentence of data) although as the insert > process continues, the columns under a given row key could potentially > grow to be large. Is that what you mean? > > An operation entails: > Read > Insert row (IE: rowkey: FOO columnName: BLAH value: 10) > Delete a row > Insert different row (IE: rowkey: FOO columnName: 0000010|Blah value: 10) > > millions of these operations are performed in sequence as files are > read from a directory source. > > On Wed, Apr 21, 2010 at 2:37 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >> then that's not the problem. >> >> are you writing large rows that OOM during compaction? >> >> On Wed, Apr 21, 2010 at 4:34 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>> They are showing up as completed? Is this correct: >>> >>> >>> Pool Name Active Pending Completed >>> STREAM-STAGE 0 0 0 >>> RESPONSE-STAGE 0 0 0 >>> ROW-READ-STAGE 0 0 517446 >>> LB-OPERATIONS 0 0 0 >>> MESSAGE-DESERIALIZER-POOL 0 0 0 >>> GMFD 0 0 0 >>> LB-TARGET 0 0 0 >>> CONSISTENCY-MANAGER 0 0 0 >>> ROW-MUTATION-STAGE 0 0 1353622 >>> MESSAGE-STREAMING-POOL 0 0 0 >>> LOAD-BALANCER-STAGE 0 0 0 >>> FLUSH-SORTER-POOL 0 0 0 >>> MEMTABLE-POST-FLUSHER 0 0 0 >>> FLUSH-WRITER-POOL 0 0 0 >>> AE-SERVICE-STAGE 0 0 0 >>> >>> >>> On Wed, Apr 21, 2010 at 2:09 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >>>> you need to figure out where the memory is going. check tpstats, if >>>> the pending ops are large somewhere that means you're just generating >>>> insert ops faster than it can handle. >>>> >>>> On Wed, Apr 21, 2010 at 4:07 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>>>> note: I'm using the Thrift API to insert. The commitLog directory >>>>> continues to grow. The heap size continues to grow as well. >>>>> >>>>> I decreased MemtableSizeInMB size, but noticed no changes. Any idea >>>>> what is causing this, and/or what property i need to tweek to >>>>> alleviate this? What is the "insert threshold"? >>>>> >>>>> I moved to a more powerful node as well, it still ended up failing >>>>> just after a longer period. >>>>> >>>>> On Wed, Apr 21, 2010 at 10:53 AM, Jonathan Ellis <jbel...@gmail.com> >>>>> wrote: >>>>>> http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts >>>>>> >>>>>> On Wed, Apr 21, 2010 at 12:02 PM, Sonny Heer <sonnyh...@gmail.com> wrote: >>>>>>> Currently running on a single node with intensive write operations. >>>>>>> >>>>>>> >>>>>>> After running for a while... >>>>>>> >>>>>>> Client starts outputting: >>>>>>> >>>>>>> TimedOutException() >>>>>>> at >>>>>>> org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:12232) >>>>>>> at >>>>>>> org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:670) >>>>>>> at >>>>>>> org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:643) >>>>>>> >>>>>>> Cassandra starts outputting: >>>>>>> >>>>>>> INFO 08:08:49,864 Cassandra starting up... >>>>>>> INFO 08:18:09,782 GC for ParNew: 238 ms, 30728008 reclaimed leaving >>>>>>> 220554976 used; max is 1190723584 >>>>>>> INFO 08:18:19,782 GC for ParNew: 231 ms, 30657944 reclaimed leaving >>>>>>> 230245792 used; max is 1190723584 >>>>>>> INFO 08:18:39,782 GC for ParNew: 229 ms, 30567184 reclaimed leaving >>>>>>> 250127792 used; max is 1190723584 >>>>>>> INFO 08:18:59,782 GC for ParNew: 358 ms, 46416776 reclaimed leaving >>>>>>> 261657720 used; max is 1190723584 >>>>>>> INFO 08:19:09,782 GC for ParNew: 205 ms, 46331376 reclaimed leaving >>>>>>> 273764040 used; max is 1190723584 >>>>>>> INFO 08:19:19,783 GC for ParNew: 335 ms, 46354968 reclaimed leaving >>>>>>> 282912656 used; max is 1190723584 >>>>>>> INFO 08:19:30,064 GC for ParNew: 392 ms, 46403400 reclaimed leaving >>>>>>> 294861824 used; max is 1190723584 >>>>>>> INFO 08:19:40,065 GC for ParNew: 326 ms, 46390000 reclaimed leaving >>>>>>> 304045640 used; max is 1190723584 >>>>>>> INFO 08:19:50,064 GC for ParNew: 256 ms, 46460824 reclaimed leaving >>>>>>> 312964344 used; max is 1190723584 >>>>>>> INFO 08:20:00,065 GC for ParNew: 242 ms, 46357104 reclaimed leaving >>>>>>> 324961320 used; max is 1190723584 >>>>>>> INFO 08:20:20,065 GC for ParNew: 336 ms, 46447216 reclaimed leaving >>>>>>> 345874144 used; max is 1190723584 >>>>>>> INFO 08:20:27,357 Creating new commitlog segment >>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271866827357.log >>>>>>> INFO 08:20:40,065 GC for ParNew: 265 ms, 46509608 reclaimed leaving >>>>>>> 366587984 used; max is 1190723584 >>>>>>> INFO 08:21:00,069 GC for ParNew: 321 ms, 46478736 reclaimed leaving >>>>>>> 384059832 used; max is 1190723584 >>>>>>> INFO 08:21:10,069 GC for ParNew: 223 ms, 62235000 reclaimed leaving >>>>>>> 383631432 used; max is 1190723584 >>>>>>> INFO 08:21:30,069 GC for ParNew: 291 ms, 62261104 reclaimed leaving >>>>>>> 399697888 used; max is 1190723584 >>>>>>> INFO 08:21:50,069 GC for ParNew: 245 ms, 62275528 reclaimed leaving >>>>>>> 415428952 used; max is 1190723584 >>>>>>> INFO 08:22:10,248 GC for ParNew: 384 ms, 62219264 reclaimed leaving >>>>>>> 433542656 used; max is 1190723584 >>>>>>> INFO 08:22:30,248 GC for ParNew: 215 ms, 62363608 reclaimed leaving >>>>>>> 452030560 used; max is 1190723584 >>>>>>> INFO 08:22:40,248 GC for ParNew: 318 ms, 62104552 reclaimed leaving >>>>>>> 464013992 used; max is 1190723584 >>>>>>> INFO 08:22:51,039 GC for ParNew: 845 ms, 62218296 reclaimed leaving >>>>>>> 471978840 used; max is 1190723584 >>>>>>> INFO 08:23:01,040 GC for ParNew: 474 ms, 62258080 reclaimed leaving >>>>>>> 475912120 used; max is 1190723584 >>>>>>> INFO 08:23:11,040 GC for ParNew: 738 ms, 62265328 reclaimed leaving >>>>>>> 483742344 used; max is 1190723584 >>>>>>> INFO 08:23:21,040 GC for ParNew: 306 ms, 62218648 reclaimed leaving >>>>>>> 491761672 used; max is 1190723584 >>>>>>> INFO 08:23:41,041 GC for ParNew: 279 ms, 62187536 reclaimed leaving >>>>>>> 507442800 used; max is 1190723584 >>>>>>> INFO 08:24:01,041 GC for ParNew: 557 ms, 62310784 reclaimed leaving >>>>>>> 523028304 used; max is 1190723584 >>>>>>> INFO 08:24:11,041 GC for ParNew: 221 ms, 62268456 reclaimed leaving >>>>>>> 530865568 used; max is 1190723584 >>>>>>> INFO 08:24:21,041 GC for ParNew: 334 ms, 62258720 reclaimed leaving >>>>>>> 542690216 used; max is 1190723584 >>>>>>> INFO 08:24:31,042 GC for ParNew: 262 ms, 62218624 reclaimed leaving >>>>>>> 550728232 used; max is 1190723584 >>>>>>> INFO 08:24:51,045 GC for ParNew: 640 ms, 62235952 reclaimed leaving >>>>>>> 573981584 used; max is 1190723584 >>>>>>> INFO 08:25:01,045 GC for ParNew: 309 ms, 62138776 reclaimed leaving >>>>>>> 563891472 used; max is 1190723584 >>>>>>> INFO 08:25:11,046 GC for ParNew: 242 ms, 62255952 reclaimed leaving >>>>>>> 575756040 used; max is 1190723584 >>>>>>> INFO 08:25:21,047 GC for ParNew: 326 ms, 62264432 reclaimed leaving >>>>>>> 583631432 used; max is 1190723584 >>>>>>> INFO 08:25:31,047 GC for ParNew: 591 ms, 62231816 reclaimed leaving >>>>>>> 595405816 used; max is 1190723584 >>>>>>> INFO 08:25:41,048 GC for ParNew: 478 ms, 62186088 reclaimed leaving >>>>>>> 603389432 used; max is 1190723584 >>>>>>> INFO 08:25:51,049 GC for ParNew: 409 ms, 62264832 reclaimed leaving >>>>>>> 615150584 used; max is 1190723584 >>>>>>> INFO 08:26:01,049 GC for ParNew: 416 ms, 62189952 reclaimed leaving >>>>>>> 623125104 used; max is 1190723584 >>>>>>> INFO 08:26:11,049 GC for ParNew: 430 ms, 62382056 reclaimed leaving >>>>>>> 634661008 used; max is 1190723584 >>>>>>> INFO 08:26:21,094 GC for ParNew: 436 ms, 62319088 reclaimed leaving >>>>>>> 646272840 used; max is 1190723584 >>>>>>> INFO 08:26:31,094 GC for ParNew: 404 ms, 62379896 reclaimed leaving >>>>>>> 653978688 used; max is 1190723584 >>>>>>> INFO 08:26:41,094 GC for ParNew: 568 ms, 62407112 reclaimed leaving >>>>>>> 665462760 used; max is 1190723584 >>>>>>> INFO 08:26:44,895 Creating new commitlog segment >>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867204895.log >>>>>>> INFO 08:26:51,094 GC for ParNew: 682 ms, 62129816 reclaimed leaving >>>>>>> 673423168 used; max is 1190723584 >>>>>>> INFO 08:27:01,094 GC for ParNew: 480 ms, 62284080 reclaimed leaving >>>>>>> 685277616 used; max is 1190723584 >>>>>>> INFO 08:27:11,094 GC for ParNew: 652 ms, 62376688 reclaimed leaving >>>>>>> 692803112 used; max is 1190723584 >>>>>>> INFO 08:27:21,094 GC for ParNew: 335 ms, 62559040 reclaimed leaving >>>>>>> 700213488 used; max is 1190723584 >>>>>>> INFO 08:27:31,094 GC for ParNew: 249 ms, 62288168 reclaimed leaving >>>>>>> 708002984 used; max is 1190723584 >>>>>>> INFO 08:27:41,095 GC for ParNew: 408 ms, 62288040 reclaimed leaving >>>>>>> 719547672 used; max is 1190723584 >>>>>>> INFO 08:27:51,095 GC for ParNew: 378 ms, 62436584 reclaimed leaving >>>>>>> 727261696 used; max is 1190723584 >>>>>>> INFO 08:28:01,095 GC for ParNew: 527 ms, 62380048 reclaimed leaving >>>>>>> 738760480 used; max is 1190723584 >>>>>>> INFO 08:28:11,097 GC for ParNew: 327 ms, 62324200 reclaimed leaving >>>>>>> 746376392 used; max is 1190723584 >>>>>>> INFO 08:28:21,268 GC for ParNew: 310 ms, 62286704 reclaimed leaving >>>>>>> 754102376 used; max is 1190723584 >>>>>>> INFO 08:28:31,268 GC for ParNew: 276 ms, 62366840 reclaimed leaving >>>>>>> 757926896 used; max is 1190723584 >>>>>>> INFO 08:28:41,268 GC for ParNew: 537 ms, 63607904 reclaimed leaving >>>>>>> 763964048 used; max is 1190723584 >>>>>>> INFO 08:28:51,269 GC for ParNew: 224 ms, 62406264 reclaimed leaving >>>>>>> 771496792 used; max is 1190723584 >>>>>>> INFO 08:29:01,269 GC for ParNew: 385 ms, 62443104 reclaimed leaving >>>>>>> 779088496 used; max is 1190723584 >>>>>>> INFO 08:29:11,269 GC for ParNew: 589 ms, 62348680 reclaimed leaving >>>>>>> 786858536 used; max is 1190723584 >>>>>>> INFO 08:29:21,269 GC for ParNew: 396 ms, 62432584 reclaimed leaving >>>>>>> 794395528 used; max is 1190723584 >>>>>>> INFO 08:29:31,269 GC for ParNew: 383 ms, 62326816 reclaimed leaving >>>>>>> 805862000 used; max is 1190723584 >>>>>>> INFO 08:29:41,269 GC for ParNew: 231 ms, 62333928 reclaimed leaving >>>>>>> 813708392 used; max is 1190723584 >>>>>>> INFO 08:29:51,271 GC for ParNew: 503 ms, 62346176 reclaimed leaving >>>>>>> 821245568 used; max is 1190723584 >>>>>>> INFO 08:30:01,270 GC for ParNew: 302 ms, 62400208 reclaimed leaving >>>>>>> 828614768 used; max is 1190723584 >>>>>>> INFO 08:30:11,271 GC for ParNew: 399 ms, 62522104 reclaimed leaving >>>>>>> 835962544 used; max is 1190723584 >>>>>>> INFO 08:30:21,448 GC for ParNew: 896 ms, 62365760 reclaimed leaving >>>>>>> 844183384 used; max is 1190723584 >>>>>>> INFO 08:30:31,448 GC for ParNew: 402 ms, 62489136 reclaimed leaving >>>>>>> 847924056 used; max is 1190723584 >>>>>>> INFO 08:30:41,778 GC for ParNew: 380 ms, 62429648 reclaimed leaving >>>>>>> 841384040 used; max is 1190723584 >>>>>>> INFO 08:30:51,662 GC for ParNew: 333 ms, 62712680 reclaimed leaving >>>>>>> 839554448 used; max is 1190723584 >>>>>>> INFO 08:31:01,664 GC for ParNew: 319 ms, 62219168 reclaimed leaving >>>>>>> 850851600 used; max is 1190723584 >>>>>>> INFO 08:31:21,663 GC for ParNew: 411 ms, 62372424 reclaimed leaving >>>>>>> 869886032 used; max is 1190723584 >>>>>>> INFO 08:31:31,664 GC for ParNew: 238 ms, 62378376 reclaimed leaving >>>>>>> 881346312 used; max is 1190723584 >>>>>>> INFO 08:31:42,026 GC for ParNew: 370 ms, 62375704 reclaimed leaving >>>>>>> 892677840 used; max is 1190723584 >>>>>>> INFO 08:32:02,027 GC for ParNew: 262 ms, 62371288 reclaimed leaving >>>>>>> 911084096 used; max is 1190723584 >>>>>>> INFO 08:32:12,301 GC for ParNew: 495 ms, 62388816 reclaimed leaving >>>>>>> 922370800 used; max is 1190723584 >>>>>>> INFO 08:32:22,301 GC for ParNew: 250 ms, 62360600 reclaimed leaving >>>>>>> 929916416 used; max is 1190723584 >>>>>>> INFO 08:32:32,301 GC for ParNew: 220 ms, 62441712 reclaimed leaving >>>>>>> 941201152 used; max is 1190723584 >>>>>>> INFO 08:32:52,303 GC for ParNew: 321 ms, 62446400 reclaimed leaving >>>>>>> 948631600 used; max is 1190723584 >>>>>>> INFO 08:33:02,304 GC for ParNew: 336 ms, 62221328 reclaimed leaving >>>>>>> 956424384 used; max is 1190723584 >>>>>>> INFO 08:33:14,864 Creating new commitlog segment >>>>>>> /var/lib/cassandra/commitlog/CommitLog-1271867594864.log >>>>>>> INFO 08:33:22,303 GC for ParNew: 319 ms, 62785240 reclaimed leaving >>>>>>> 972976288 used; max is 1190723584 >>>>>>> INFO 08:33:32,304 GC for ParNew: 284 ms, 62525360 reclaimed leaving >>>>>>> 981829584 used; max is 1190723584 >>>>>>> INFO 08:33:52,304 GC for ParNew: 369 ms, 62453624 reclaimed leaving >>>>>>> 995907192 used; max is 1190723584 >>>>>>> INFO 08:34:12,354 GC for ParNew: 247 ms, 63642000 reclaimed leaving >>>>>>> 1011914464 used; max is 1190723584 >>>>>>> INFO 08:34:35,048 GC for ConcurrentMarkSweep: 13449 ms, 67194408 >>>>>>> reclaimed leaving 1015729984 used; max is 1190723584 >>>>>>> INFO 08:34:58,737 GC for ConcurrentMarkSweep: 19035 ms, 62848912 >>>>>>> reclaimed leaving 1019176200 used; max is 1190723584 >>>>>>> INFO 08:35:22,270 GC for ConcurrentMarkSweep: 19822 ms, 62707088 >>>>>>> reclaimed leaving 1022673936 used; max is 1190723584 >>>>>>> INFO 08:35:44,464 GC for ConcurrentMarkSweep: 18461 ms, 59950904 >>>>>>> reclaimed leaving 1025979696 used; max is 1190723584 >>>>>>> INFO 08:36:07,582 GC for ConcurrentMarkSweep: 19597 ms, 56920072 >>>>>>> reclaimed leaving 1028992352 used; max is 1190723584 >>>>>>> INFO 08:36:31,071 GC for ConcurrentMarkSweep: 19924 ms, 53955440 >>>>>>> reclaimed leaving 1031961224 used; max is 1190723584 >>>>>>> INFO 08:36:54,651 GC for ConcurrentMarkSweep: 19851 ms, 51375568 >>>>>>> reclaimed leaving 1034536576 used; max is 1190723584 >>>>>>> INFO 08:37:17,136 GC for ConcurrentMarkSweep: 19554 ms, 48567880 >>>>>>> reclaimed leaving 1037266456 used; max is 1190723584 >>>>>>> INFO 08:37:40,426 GC for ConcurrentMarkSweep: 20502 ms, 46101432 >>>>>>> reclaimed leaving 1039709528 used; max is 1190723584 >>>>>>> INFO 08:38:04,034 GC for ConcurrentMarkSweep: 20880 ms, 43580800 >>>>>>> reclaimed leaving 1042230248 used; max is 1190723584 >>>>>>> INFO 08:38:28,202 GC for ConcurrentMarkSweep: 21714 ms, 41275936 >>>>>>> reclaimed leaving 1044527600 used; max is 1190723584 >>>>>>> INFO 08:38:51,781 GC for ConcurrentMarkSweep: 21180 ms, 39182008 >>>>>>> reclaimed leaving 1046621520 used; max is 1190723584 >>>>>>> INFO 08:39:15,180 GC for ConcurrentMarkSweep: 20973 ms, 37127336 >>>>>>> reclaimed leaving 1048677040 used; max is 1190723584 >>>>>>> INFO 08:39:38,701 GC for ConcurrentMarkSweep: 21160 ms, 35241368 >>>>>>> reclaimed leaving 1050563688 used; max is 1190723584 >>>>>>> INFO 08:39:52,046 GC for ConcurrentMarkSweep: 11468 ms, 33367112 >>>>>>> reclaimed leaving 1052423824 used; max is 1190723584 >>>>>>> INFO 08:40:13,968 GC for ConcurrentMarkSweep: 19517 ms, 31652048 >>>>>>> reclaimed leaving 1054138992 used; max is 1190723584 >>>>>>> INFO 08:40:27,228 GC for ConcurrentMarkSweep: 11430 ms, 30042560 >>>>>>> reclaimed leaving 1055761280 used; max is 1190723584 >>>>>>> INFO 08:40:48,720 GC for ConcurrentMarkSweep: 19716 ms, 28610728 >>>>>>> reclaimed leaving 1057201248 used; max is 1190723584 >>>>>>> INFO 08:41:02,219 GC for ConcurrentMarkSweep: 11780 ms, 27313792 >>>>>>> reclaimed leaving 1058496000 used; max is 1190723584 >>>>>>> INFO 08:41:23,596 GC for ConcurrentMarkSweep: 19641 ms, 25788136 >>>>>>> reclaimed leaving 1060011424 used; max is 1190723584 >>>>>>> INFO 08:41:36,393 GC for ConcurrentMarkSweep: 11441 ms, 24410792 >>>>>>> reclaimed leaving 1061388688 used; max is 1190723584 >>>>>>> INFO 08:41:58,677 GC for ConcurrentMarkSweep: 20674 ms, 23141096 >>>>>>> reclaimed leaving 1062658344 used; max is 1190723584 >>>>>>> INFO 08:42:11,687 GC for ConcurrentMarkSweep: 11533 ms, 21962488 >>>>>>> reclaimed leaving 1063834408 used; max is 1190723584 >>>>>>> INFO 08:42:33,121 GC for ConcurrentMarkSweep: 20099 ms, 20999232 >>>>>>> reclaimed leaving 1064800992 used; max is 1190723584 >>>>>>> INFO 08:42:45,727 GC for ConcurrentMarkSweep: 11518 ms, 20002288 >>>>>>> reclaimed leaving 1065798256 used; max is 1190723584 >>>>>>> INFO 08:43:07,617 GC for ConcurrentMarkSweep: 20496 ms, 18938680 >>>>>>> reclaimed leaving 1066867192 used; max is 1190723584 >>>>>>> INFO 08:43:20,341 GC for ConcurrentMarkSweep: 11540 ms, 17880304 >>>>>>> reclaimed leaving 1067925416 used; max is 1190723584 >>>>>>> INFO 08:43:42,240 GC for ConcurrentMarkSweep: 20687 ms, 16922632 >>>>>>> reclaimed leaving 1068886912 used; max is 1190723584 >>>>>>> INFO 08:43:54,826 GC for ConcurrentMarkSweep: 11644 ms, 16104184 >>>>>>> reclaimed leaving 1069705264 used; max is 1190723584 >>>>>>> INFO 08:44:16,656 GC for ConcurrentMarkSweep: 20778 ms, 15238544 >>>>>>> reclaimed leaving 1070568224 used; max is 1190723584 >>>>>>> INFO 08:44:29,116 GC for ConcurrentMarkSweep: 11682 ms, 14440752 >>>>>>> reclaimed leaving 1071366040 used; max is 1190723584 >>>>>>> INFO 08:44:50,753 GC for ConcurrentMarkSweep: 20628 ms, 13641104 >>>>>>> reclaimed leaving 1072168400 used; max is 1190723584 >>>>>>> INFO 08:45:03,283 GC for ConcurrentMarkSweep: 11815 ms, 13066264 >>>>>>> reclaimed leaving 1072743152 used; max is 1190723584 >>>>>>> INFO 08:45:25,062 GC for ConcurrentMarkSweep: 20760 ms, 12472160 >>>>>>> reclaimed leaving 1073334720 used; max is 1190723584 >>>>>>> INFO 08:45:37,296 GC for ConcurrentMarkSweep: 11580 ms, 11883152 >>>>>>> reclaimed leaving 1073930576 used; max is 1190723584 >>>>>>> INFO 08:45:59,267 GC for ConcurrentMarkSweep: 20994 ms, 11270416 >>>>>>> reclaimed leaving 1074548160 used; max is 1190723584 >>>>>>> INFO 08:46:11,728 GC for ConcurrentMarkSweep: 11801 ms, 10689544 >>>>>>> reclaimed leaving 1075122368 used; max is 1190723584 >>>>>>> INFO 08:46:33,579 GC for ConcurrentMarkSweep: 21071 ms, 10109272 >>>>>>> reclaimed leaving 1075699904 used; max is 1190723584 >>>>>>> INFO 08:46:46,011 GC for ConcurrentMarkSweep: 11723 ms, 9581552 >>>>>>> reclaimed leaving 1076227648 used; max is 1190723584 >>>>>>> INFO 08:47:08,143 GC for ConcurrentMarkSweep: 21184 ms, 9072976 >>>>>>> reclaimed leaving 1076747648 used; max is 1190723584 >>>>>>> INFO 08:47:20,446 GC for ConcurrentMarkSweep: 11730 ms, 8671584 >>>>>>> reclaimed leaving 1077152296 used; max is 1190723584 >>>>>>> INFO 08:47:42,173 GC for ConcurrentMarkSweep: 20917 ms, 8277168 >>>>>>> reclaimed leaving 1077567128 used; max is 1190723584 >>>>>>> INFO 08:47:54,217 GC for ConcurrentMarkSweep: 11590 ms, 7864704 >>>>>>> reclaimed leaving 1077987824 used; max is 1190723584 >>>>>>> INFO 08:48:16,037 GC for ConcurrentMarkSweep: 21135 ms, 7464848 >>>>>>> reclaimed leaving 1078393448 used; max is 1190723584 >>>>>>> INFO 08:48:28,125 GC for ConcurrentMarkSweep: 11709 ms, 7150136 >>>>>>> reclaimed leaving 1078708784 used; max is 1190723584 >>>>>>> INFO 08:48:49,882 GC for ConcurrentMarkSweep: 20807 ms, 6783344 >>>>>>> reclaimed leaving 1079074464 used; max is 1190723584 >>>>>>> INFO 08:49:02,369 GC for ConcurrentMarkSweep: 11706 ms, 6457992 >>>>>>> reclaimed leaving 1079409016 used; max is 1190723584 >>>>>>> INFO 08:49:24,109 GC for ConcurrentMarkSweep: 21370 ms, 6108952 >>>>>>> reclaimed leaving 1079749672 used; max is 1190723584 >>>>>>> INFO 08:49:36,221 GC for ConcurrentMarkSweep: 11721 ms, 5844768 >>>>>>> reclaimed leaving 1080014112 used; max is 1190723584 >>>>>>> INFO 08:49:58,110 GC for ConcurrentMarkSweep: 21327 ms, 5573216 >>>>>>> reclaimed leaving 1080278992 used; max is 1190723584 >>>>>>> INFO 08:50:10,218 GC for ConcurrentMarkSweep: 11762 ms, 5277912 >>>>>>> reclaimed leaving 1080574456 used; max is 1190723584 >>>>>>> INFO 08:50:32,083 GC for ConcurrentMarkSweep: 21350 ms, 4978376 >>>>>>> reclaimed leaving 1080884096 used; max is 1190723584 >>>>>>> INFO 08:50:44,189 GC for ConcurrentMarkSweep: 11842 ms, 4785512 >>>>>>> reclaimed leaving 1081076888 used; max is 1190723584 >>>>>>> INFO 08:51:05,763 GC for ConcurrentMarkSweep: 21020 ms, 4539168 >>>>>>> reclaimed leaving 1081313872 used; max is 1190723584 >>>>>>> INFO 08:51:18,072 GC for ConcurrentMarkSweep: 11856 ms, 4324168 >>>>>>> reclaimed leaving 1081528880 used; max is 1190723584 >>>>>>> INFO 08:51:39,485 GC for ConcurrentMarkSweep: 21025 ms, 4151152 >>>>>>> reclaimed leaving 1081704648 used; max is 1190723584 >>>>>>> INFO 08:51:51,777 GC for ConcurrentMarkSweep: 11838 ms, 3966216 >>>>>>> reclaimed leaving 1081889512 used; max is 1190723584 >>>>>>> INFO 08:52:13,223 GC for ConcurrentMarkSweep: 21004 ms, 3780392 >>>>>>> reclaimed leaving 1082072568 used; max is 1190723584 >>>>>>> INFO 08:52:25,399 GC for ConcurrentMarkSweep: 11785 ms, 3582376 >>>>>>> reclaimed leaving 1082270680 used; max is 1190723584 >>>>>>> INFO 08:52:46,716 GC for ConcurrentMarkSweep: 21108 ms, 3332920 >>>>>>> reclaimed leaving 1082522872 used; max is 1190723584 >>>>>>> INFO 08:52:58,817 GC for ConcurrentMarkSweep: 11925 ms, 3223392 >>>>>>> reclaimed leaving 1082632368 used; max is 1190723584 >>>>>>> INFO 08:53:20,629 GC for ConcurrentMarkSweep: 21396 ms, 3058560 >>>>>>> reclaimed leaving 1082794472 used; max is 1190723584 >>>>>>> INFO 08:53:32,625 GC for ConcurrentMarkSweep: 11626 ms, 2890024 >>>>>>> reclaimed leaving 1082963008 used; max is 1190723584 >>>>>>> INFO 08:53:53,966 GC for ConcurrentMarkSweep: 21147 ms, 2738504 >>>>>>> reclaimed leaving 1083117048 used; max is 1190723584 >>>>>>> INFO 08:54:06,106 GC for ConcurrentMarkSweep: 11786 ms, 2642192 >>>>>>> reclaimed leaving 1083213304 used; max is 1190723584 >>>>>>> INFO 08:54:27,769 GC for ConcurrentMarkSweep: 21436 ms, 2521392 >>>>>>> reclaimed leaving 1083332280 used; max is 1190723584 >>>>>>> INFO 08:54:39,721 GC for ConcurrentMarkSweep: 11792 ms, 2404808 >>>>>>> reclaimed leaving 1083448744 used; max is 1190723584 >>>>>>> INFO 08:55:01,320 GC for ConcurrentMarkSweep: 21250 ms, 2254864 >>>>>>> reclaimed leaving 1083601864 used; max is 1190723584 >>>>>>> INFO 08:55:13,056 GC for ConcurrentMarkSweep: 11608 ms, 2176520 >>>>>>> reclaimed leaving 1083680168 used; max is 1190723584 >>>>>>> INFO 08:55:34,863 GC for ConcurrentMarkSweep: 21455 ms, 2045872 >>>>>>> reclaimed leaving 1083814848 used; max is 1190723584 >>>>>>> INFO 08:55:46,642 GC for ConcurrentMarkSweep: 11670 ms, 1989648 >>>>>>> reclaimed leaving 1083871152 used; max is 1190723584 >>>>>>> INFO 08:56:08,705 GC for ConcurrentMarkSweep: 21669 ms, 1872720 >>>>>>> reclaimed leaving 1083984728 used; max is 1190723584 >>>>>>> INFO 08:56:20,453 GC for ConcurrentMarkSweep: 11647 ms, 1794056 >>>>>>> reclaimed leaving 1084063504 used; max is 1190723584 >>>>>>> >>>>>> >>>>> >>>> >>> >> >
