Hi:ALL
I have 10 nodes cluster ,after inserting many records into the cluster, i
compact each node by nodetool compact.
during the compaciton process ,something wrong with one of the 10 nodes ,
when the size of the compacted temp file rech nearly 100GB( before
compaction ,the size is ~240G)
the GC storm begins,following is the log fragment:
INFO [GC inspection] 2010-06-13 10:02:59,420 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43683 ms, 126983320 reclaimed
leaving 14910532584 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:03:43,571 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43841 ms, 128764776 reclaimed
leaving 14908792344 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:04:18,032 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 34173 ms, 130891824 reclaimed
leaving 14906666280 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:05:02,064 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43754 ms, 128821816 reclaimed
leaving 14908752712 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:05:36,514 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 34151 ms, 125388552 reclaimed
leaving 14912188144 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:06:20,360 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43659 ms, 135293856 reclaimed
leaving 14902289328 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:07:02,049 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 41414 ms, 134790856 reclaimed
leaving 14902771920 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:07:50,410 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 48082 ms, 129526904 reclaimed
leaving 14908040152 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:08:24,489 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 33765 ms, 127936008 reclaimed
leaving 14909622528 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:09:08,312 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43553 ms, 126393960 reclaimed
leaving 14911172104 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:09:42,384 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 33791 ms, 127458944 reclaimed
leaving 14910102920 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:10:26,235 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43587 ms, 129176704 reclaimed
leaving 14908380504 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:11:07,607 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 41063 ms, 118689104 reclaimed
leaving 14918868144 used; max is 15153364992
INFO [GC inspection] 2010-06-13 10:11:51,574 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 43677 ms, 112677232 reclaimed
the configuration of the node : 8 cores, 16GB RAM.
when the insert process end ,i found there are ~5000 sstables in data
directory, the size of most of the files is ~70M
some of the configuration:
-Xms12G \
-Xmx14G \
<DiskAccessMode>standard</DiskAccessMode>
What's wrong with it? Any suggestion will be appreciated.
--
Best Regards
Anty Rao
