Thanks. The only logs I have are system and cassandra. I've included
those. I don't have gcinspector logs. I log gc via munin on other
machines, but I need to install it on these.
On 8/25/11 2:22 PM, Adi wrote:
Ernst,
Can you share the logs just before the crash. Specially the
GCInspector logs.Check the last reported used heap space and whether
it was close to the threshold for full GC.
Also how frequent are your OOM crashes?
The cassandra default for kicking in full GC is 75% (
-XX:CMSInitiatingOccupancyFraction=75). You can try lowering that to
maybe 65 or some lower number and see if that helps. Basically you
will have a little more frequent GCs which will take a little less
time than before. And possibly that unused heap buffer will provide
for the operation which is causing OOM.
I saw an OOM on one node after 2 weeks. The heap used was close to the
GC threshold and full GC takes around 80 seconds.
-Adi
2011/8/24 Ernst D Schoen-René<er...@peoplebrowsr.com>:
So, we're on 8, so I don't think there's a key cache setting. Am I wrong?
here's my newest crash log:
ERROR [Thread-210] 2011-08-24 06:29:53,247 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-210,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [ScheduledTasks:1] 2011-08-24 06:29:53,249 StatusLogger.java (line 65)
InternalResponseStage 0 0
INFO [ScheduledTasks:1] 2011-08-24 06:29:53,250 StatusLogger.java (line 65)
HintedHandoff 1 16
INFO [ScheduledTasks:1] 2011-08-24 06:29:53,250 StatusLogger.java (line 69)
CompactionManager n/a 1298
INFO [ScheduledTasks:1] 2011-08-24 06:29:53,250 StatusLogger.java (line 81)
MessagingService n/a 1,0
INFO [ScheduledTasks:1] 2011-08-24 06:29:53,250 StatusLogger.java (line 85)
ColumnFamily Memtable ops,data Row cache size/cap Key cache
size/cap
On 8/24/11 2:38 AM, aaron morton wrote:
First thing is are you on 0.8 ? It has some automagical memory management
that is both automatic and
magical http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/
Secondly if you are OOM'ing you need to look at how much memory your schema
is taking. See the link above, or just use 0.8
Thirdly, try to reduce the cache as Adi noted.
Finally try to work out when it's OOM'ing. Is it during a compaction, a
repair, during peak load or just regular operations. With 1TB of data i'd be
taking a look at the compaction or repair processes to see whats happening
GC wise. Also check the GC settings as described in cassandra-env.sh
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 24/08/2011, at 12:38 PM, Ernst D Schoen-René wrote:
Thanks,
We had already been running cassandra with a larger heap size, but it
meant that java took way too long between garbage collections. The advice
I'd found was to set the heap size at the 8 we're running at. It was ok for
a while, but now some nodes crash. It's definitely our experience that
adding more memory per node actually makes things worse eventually, as java
starts eating up too many resources for it to handle.
On 8/23/11 5:28 PM, Adi wrote:
2011/8/23 Ernst D Schoen-René<er...@peoplebrowsr.com>:
Hi,
I'm running a 16-node cassandra cluster, with a reasonably large amount of
data per node (~1TB). Nodes have 16G ram, but heap is set to 8G.
The nodes keep stopping with this output in the log. Any ideas?
ERROR [Thread-85] 2011-08-23 21:00:38,723 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-85,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [ReadStage:568] 2011-08-23 21:00:38,723 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[ReadStage:568,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO [HintedHandoff:1] 2011-08-23 21:00:38,720 HintedHandOffManager.java
(line 320) Started hinted handoff for endpoint /10.28.0.184
INFO [GossipStage:2] 2011-08-23 21:00:50,751 Gossiper.java (line 606)
InetAddress /10.29.20.67 is now UP
ERROR [Thread-34] 2011-08-23 21:00:50,525 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-34,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-36] 2011-08-23 21:00:50,518 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-36,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [GossipTasks:1] 2011-08-23 21:00:50,466 Gossiper.java (line 620)
InetAddress /10.29.20.67 is now dead.
INFO [HintedHandoff:1] 2011-08-23 21:00:50,751 HintedHandOffManager.java
(line 376) Finished hinted handoff of 0 rows to endpoint /10.28.0.184
ERROR [Thread-33] 2011-08-23 21:01:05,048 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-33,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-128] 2011-08-23 21:01:05,048 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-128,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
root@cass1:~#
You can try the cargo cult solution of upping the heap to 12GB and
see if the nodes stabilize. We have a 4-node cluster with 2-3 TB data
per node and that was the heap at which it the nodes were managing to
serve requests without running out of memory. Ultimately we ordered
more memory and are running it with 24 GB heap and the cluster has
been stable without complains.
Other things you can do for reducing memory usage if they are
appropriate for your read/write profile:
a) reduce memtable throughput(most reduction in mem footprint)
b) disable row caching
c) reduce/disable key caching(least reduction)
Ultimately you will have to tune based on your
1) row sizes
2) read/write load
-Adi
INFO [CompactionExecutor:46642] 2011-08-25 02:24:07,513
CompactionIterator.java (line 167) Minor@933772622(creds, cred_transactions,
85097783574/87846767242) now compacting at 67108 bytes/ms.
INFO [NonPeriodicTasks:1] 2011-08-25 02:25:04,861 MeteredFlusher.java (line
62) flushing high-traffic column family ColumnFamilyStore(table='creds',
columnFamily='cred_transactions')
INFO [NonPeriodicTasks:1] 2011-08-25 02:25:04,862 ColumnFamilyStore.java (line
1013) Enqueuing flush of
Memtable-cred_transactions@1685154644(9045318/169838270 serialized/live bytes,
71280 ops)
INFO [FlushWriter:1202] 2011-08-25 02:25:04,862 Memtable.java (line 237)
Writing Memtable-cred_transactions@1685154644(9045318/169838270 serialized/live
bytes, 71280 ops)
INFO [FlushWriter:1202] 2011-08-25 02:25:17,727 Memtable.java (line 254)
Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-48859-Data.db (14627100 bytes)
INFO [NonPeriodicTasks:1] 2011-08-25 02:25:58,102 MeteredFlusher.java (line
62) flushing high-traffic column family ColumnFamilyStore(table='creds',
columnFamily='cred_transactions')
INFO [NonPeriodicTasks:1] 2011-08-25 02:25:58,103 ColumnFamilyStore.java (line
1013) Enqueuing flush of
Memtable-cred_transactions@2090889196(8316004/156144397 serialized/live bytes,
65522 ops)
INFO [FlushWriter:1202] 2011-08-25 02:25:58,104 Memtable.java (line 237)
Writing Memtable-cred_transactions@2090889196(8316004/156144397 serialized/live
bytes, 65522 ops)
INFO [FlushWriter:1202] 2011-08-25 02:26:08,332 Memtable.java (line 254)
Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-48860-Data.db (13417621 bytes)
ERROR [CompactionExecutor:46642] 2011-08-25 02:26:54,308
AbstractCassandraDaemon.java (line 113) Fatal exception in thread
Thread[CompactionExecutor:46642,4,main]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
at
org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:374)
at
org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:347)
at
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:88)
at
org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:99)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132)
at
org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:574)
at
org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:504)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:140)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:106)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [Thread-129] 2011-08-25 02:26:54,310 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-129,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-30] 2011-08-25 02:26:54,310 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-30,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-128] 2011-08-25 02:26:54,310 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-128,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-132] 2011-08-25 02:26:54,311 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-132,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-131] 2011-08-25 02:26:54,311 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-131,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-196] 2011-08-25 02:26:54,311 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-196,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-130] 2011-08-25 02:26:54,312 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-130,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-133] 2011-08-25 02:26:54,311 AbstractCassandraDaemon.java (line
113) Fatal exception in thread Thread[Thread-133,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO 05:15:00,235 Writing
Memtable-cred_transactions@1399639513(8721005/156275967 serialized/live bytes,
69248 ops)
INFO 05:15:00,587 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3572-Data.db (14732235 bytes)
INFO 05:15:00,588 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3571-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3569-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3570-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3572-Data.db')]
INFO 05:15:06,367 Compacted to
/disk1/cassandra/data/creds/cred_transactions-tmp-g-3573-Data.db. 59,127,021
to 54,951,639 (~92% of original) bytes for 111,691 keys. Time: 5,779ms.
INFO 05:15:06,368 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3568-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3558-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3563-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3573-Data.db')]
INFO 05:15:21,071 Compacted to
/disk1/cassandra/data/creds/cred_transactions-tmp-g-3574-Data.db. 219,322,605
to 203,954,866 (~92% of original) bytes for 280,263 keys. Time: 14,702ms.
INFO 05:15:33,240 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:15:33,241 Enqueuing flush of
Memtable-cred_transactions@590769509(8600156/154110414 serialized/live bytes,
68302 ops)
INFO 05:15:33,241 Writing
Memtable-cred_transactions@590769509(8600156/154110414 serialized/live bytes,
68302 ops)
INFO 05:15:33,531 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3575-Data.db (14540166 bytes)
INFO 05:15:53,504 Creating new commitlog segment
/disk1/cassandra/commitlog/CommitLog-1312953353504.log
INFO 05:16:03,258 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:16:03,259 Enqueuing flush of
Memtable-cred_transactions@971124888(8879355/159113518 serialized/live bytes,
70527 ops)
INFO 05:16:03,259 Writing
Memtable-cred_transactions@971124888(8879355/159113518 serialized/live bytes,
70527 ops)
INFO 05:16:03,513 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3576-Data.db (14964051 bytes)
INFO 05:16:03,513 Discarding obsolete commit
log:CommitLogSegment(/disk1/cassandra/commitlog/CommitLog-1312953090765.log)
INFO 05:16:33,278 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:16:33,278 Enqueuing flush of
Memtable-cred_transactions@1271412856(8826970/158174806 serialized/live bytes,
70120 ops)
INFO 05:16:33,278 Writing
Memtable-cred_transactions@1271412856(8826970/158174806 serialized/live bytes,
70120 ops)
INFO 05:16:33,547 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3577-Data.db (14973137 bytes)
INFO 05:17:05,283 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:17:05,283 Enqueuing flush of
Memtable-cred_transactions@492408828(8651864/155036995 serialized/live bytes,
68742 ops)
INFO 05:17:05,284 Writing
Memtable-cred_transactions@492408828(8651864/155036995 serialized/live bytes,
68742 ops)
INFO 05:17:05,529 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3578-Data.db (14591718 bytes)
INFO 05:17:05,530 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3576-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3578-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3575-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3577-Data.db')]
INFO 05:17:09,913 Compacted to
/disk1/cassandra/data/creds/cred_transactions-tmp-g-3579-Data.db. 59,069,072
to 54,799,553 (~92% of original) bytes for 110,784 keys. Time: 4,383ms.
INFO 05:17:36,288 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:17:36,289 Enqueuing flush of
Memtable-cred_transactions@2113004586(8724343/156335782 serialized/live bytes,
69346 ops)
INFO 05:17:36,289 Writing
Memtable-cred_transactions@2113004586(8724343/156335782 serialized/live bytes,
69346 ops)
INFO 05:17:36,644 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3580-Data.db (14703492 bytes)
INFO 05:18:04,299 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:18:04,299 Enqueuing flush of
Memtable-cred_transactions@1551995092(9010929/161471257 serialized/live bytes,
71600 ops)
INFO 05:18:04,299 Writing
Memtable-cred_transactions@1551995092(9010929/161471257 serialized/live bytes,
71600 ops)
INFO 05:18:04,579 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3581-Data.db (15153433 bytes)
INFO 05:18:33,304 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:18:33,304 Enqueuing flush of
Memtable-cred_transactions@1946176564(8691693/155750711 serialized/live bytes,
69072 ops)
INFO 05:18:33,304 Writing
Memtable-cred_transactions@1946176564(8691693/155750711 serialized/live bytes,
69072 ops)
INFO 05:18:33,568 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3582-Data.db (14649605 bytes)
INFO 05:19:12,311 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:19:12,311 Enqueuing flush of
Memtable-cred_transactions@370020964(8560728/153403885 serialized/live bytes,
68015 ops)
INFO 05:19:12,312 Writing
Memtable-cred_transactions@370020964(8560728/153403885 serialized/live bytes,
68015 ops)
INFO 05:19:12,848 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3583-Data.db (14405543 bytes)
INFO 05:19:12,850 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3581-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3580-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3583-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3582-Data.db')]
INFO 05:19:16,576 Compacted to
/disk1/cassandra/data/creds/cred_transactions-tmp-g-3584-Data.db. 58,912,073
to 54,598,245 (~92% of original) bytes for 108,179 keys. Time: 3,726ms.
INFO 05:19:41,316 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:19:41,317 Enqueuing flush of
Memtable-cred_transactions@1499935050(8776955/157278562 serialized/live bytes,
69749 ops)
INFO 05:19:41,317 Writing
Memtable-cred_transactions@1499935050(8776955/157278562 serialized/live bytes,
69749 ops)
INFO 05:19:41,705 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3585-Data.db (14662100 bytes)
INFO 05:20:00,706 Creating new commitlog segment
/disk1/cassandra/commitlog/CommitLog-1312953600706.log
INFO 05:20:13,322 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:20:13,323 Enqueuing flush of
Memtable-cred_transactions@937855669(8886021/159232970 serialized/live bytes,
70595 ops)
INFO 05:20:13,323 Writing
Memtable-cred_transactions@937855669(8886021/159232970 serialized/live bytes,
70595 ops)
INFO 05:20:13,838 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3586-Data.db (14799492 bytes)
INFO 05:20:13,838 Discarding obsolete commit
log:CommitLogSegment(/disk1/cassandra/commitlog/CommitLog-1312953353504.log)
INFO 05:20:43,328 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:20:43,328 Enqueuing flush of
Memtable-cred_transactions@45286865(8674483/155442316 serialized/live bytes,
68929 ops)
INFO 05:20:43,328 Writing
Memtable-cred_transactions@45286865(8674483/155442316 serialized/live bytes,
68929 ops)
INFO 05:20:43,618 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3587-Data.db (14512156 bytes)
INFO 05:21:12,897 Opening /disk1/cassandra/data/creds/cred_transactions-g-3393
INFO 05:21:19,333 flushing high-traffic column family
ColumnFamilyStore(table='creds', columnFamily='cred_transactions')
INFO 05:21:19,334 Enqueuing flush of
Memtable-cred_transactions@270497320(8630587/154655722 serialized/live bytes,
68583 ops)
INFO 05:21:19,334 Writing
Memtable-cred_transactions@270497320(8630587/154655722 serialized/live bytes,
68583 ops)
INFO 05:21:19,712 Completed flushing
/disk1/cassandra/data/creds/cred_transactions-g-3588-Data.db (14636755 bytes)
INFO 05:21:19,713 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3586-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3585-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3587-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3588-Data.db')]
INFO 05:21:23,634 Compacted to
/disk1/cassandra/data/creds/cred_transactions-tmp-g-3589-Data.db. 58,610,503
to 53,736,945 (~91% of original) bytes for 99,223 keys. Time: 3,921ms.
INFO 05:21:34,147 Finished streaming session 593802120658571 from /10.28.0.170
INFO 05:21:34,148 Compacting Minor:
[SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3545-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3547-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3549-Data.db'),
SSTableReader(path='/disk1/cassandra/data/creds/cred_transactions-g-3548-Data.db')]
INFO 05:21:34,153 Minor@1712231664(creds, cred_transactions, 373/28553140) now
compacting at 33554 bytes/ms.
INFO 05:21:34,159 Enqueuing flush of Memtable-LocationInfo@1871001581(88/110
serialized/live bytes, 3 ops)
INFO 05:21:34,159 Writing Memtable-LocationInfo@1871001581(88/110
serialized/live bytes, 3 ops)
INFO 05:21:34,181 Completed flushing
/disk1/cassandra/data/system/LocationInfo-g-30-Data.db (252 bytes)
INFO 05:21:34,181 Discarding obsolete commit
log:CommitLogSegment(/disk1/cassandra/commitlog/CommitLog-1312948407999.log)
INFO 05:21:34,189 Bootstrap/move completed! Now serving reads.
INFO 05:21:34,189 Enqueuing flush of Memtable-LocationInfo@942576869(53/66
serialized/live bytes, 2 ops)
INFO 05:21:34,189 Writing Memtable-LocationInfo@942576869(53/66
serialized/live bytes, 2 ops)
INFO 05:21:34,293 Completed flushing
/disk1/cassandra/data/system/LocationInfo-g-31-Data.db (163 bytes)
INFO 05:21:34,368 Minor@779900358(creds, cred_all_link,
13200961066/22357135595) now compacting at 33554 bytes/ms.
INFO 05:21:34,586 mx4j successfuly loaded
HttpAdaptor version 3.0.2 started on port 8081
