From: Pan, Adeline (TR Technology & Ops)
Sent: Tuesday, September 06, 2016 12:34 PM
To: 'user@cassandra.apache.org'
Cc: Yang, Ling (TR Technology & Ops)
Subject: FW: WriteTimeoutException with LOCAL_QUORUM
Hi All,
I hope you are doing well today, and I need your help.
We were using Cassandra 1 before, now we are upgrading to Cassandra 3.4 .
During the integration test, we encountered "WriteTimeoutException" very
frequently (about every other minute), the exception message is as below. The
exception trace is in the attach file.
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency LOCAL_QUORUM (2 replica were required
but only 1 acknowledged the write)
There is some information:
1. It is a six nodes cluster, two data centers, and three nodes for each
datacenter. The consistency level we are using is LOCAL_QUORUM
2. The node info
[BETA:xxxx@xxxx:/local/java/cassandra3/current]$ bin/nodetool -hlocalhost info
ID : ad077318-6531-498e-bf5a-14ac339d1a45
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 23.47 GB
Generation No : 1473065408
Uptime (seconds) : 67180
Heap Memory (MB) : 1679.57 / 4016.00
Off Heap Memory (MB) : 10.34
Data Center : dc1
Rack : rack1
Exceptions : 0
Key Cache : entries 32940, size 3.8 MB, capacity 100 MB, 2124114
hits, 2252348 requests, 0.943 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0
requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0
requests, NaN recent hit rate, 7200 save period in seconds
Token : (invoke with -T/--tokens to see all 256 tokens)
3. We have increased the write_request_timeout_in_ms to 40000, which
didn't work.
4. The memtable size is 4GB.
5. memtable_allocation_type: heap_buffers
6. In the Cassandra server log, we found there are
Native-Transport-Requests pending from time to time. (The server log piece is
in attach file)
INFO [ScheduledTasks:1] 2016-09-01 10:08:47,036 StatusLogger.java:52 - Pool
Name Active Pending Completed Blocked
All Time Blocked
INFO [ScheduledTasks:1] 2016-09-01 10:08:47,043 StatusLogger.java:56 -
Native-Transport-Requests 128 134 300721823 6
28211424
What also need to mention is there is always 1 node will acknowledge the write
.Could you please help me with this situation? Any clue will be appreciated.
Thank you very much!
-------------------------------------------------------------------------------------------
Adeline Pan
Senior Software Engineer
Thomson Reuters
Phone: 62674654
adeline....@thomsonreuters.com<mailto:adeline....@thomsonreuters.com>
ERROR 2016-09-01 01:50:25,185 [pool-17-thread-5] [objectid=100081245
objecttypeid=3 rundatetime=2016-09-01T00:00:00
guid=51ea5d505dab4273aa65f779be71c4e1
sessionguid=132c1d5b6bf142a2ad4475c88dd6595a]
c.p.e.a.l.s.OracleaqObjectEventReader: Error handling message
com.pointcarbon.esb.cassandra.common.CassandraException: Cannot store data for
vscid=114076798 rundatetime=2016-09-01T00:00:00.000Z
at
com.pointcarbon.esb.cassandra.csdao.CassandraDao.storeData(CassandraDao.java:888)
~[esb-cassandra-api-4.0.0.jar:na]
at
com.pointcarbon.esb.cassandra.csdao.CassandraDao.store(CassandraDao.java:494)
~[esb-cassandra-api-4.0.0.jar:na]
at
com.pointcarbon.esb.app.lastupdate.LastUpdateController.processObjectWithCassandra(LastUpdateController.java:286)
~[esb-app-lastupdate-default/:na]
at
com.pointcarbon.esb.app.lastupdate.LastUpdateController.processObject(LastUpdateController.java:111)
~[esb-app-lastupdate-default/:na]
at
com.pointcarbon.esb.app.lastupdate.service.ObjectEventService.process(ObjectEventService.java:149)
~[esb-app-lastupdate-default/:na]
at
com.pointcarbon.esb.app.lastupdate.service.OracleaqObjectEventReader$1.handleMessage(OracleaqObjectEventReader.java:55)
~[esb-app-lastupdate-default/:na]
at
com.pointcarbon.esb.transport.oracleaq.service.AqService.loop(AqService.java:77)
~[esb-transport-oracleaq-3.0.3.jar:na]
at
com.pointcarbon.esb.transport.oracleaq.service.AqService.startSession(AqService.java:38)
~[esb-transport-oracleaq-3.0.3.jar:na]
at
com.pointcarbon.esb.app.lastupdate.service.OracleaqObjectEventReader$2.run(OracleaqObjectEventReader.java:68)
~[esb-app-lastupdate-default/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
~[na:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency LOCAL_QUORUM (2 replica were required
but only 1 acknowledged the write)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:26)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.pointcarbon.esb.cassandra.csdao.CassandraDao.asyncWritesHelper(CassandraDao.java:963)
~[esb-cassandra-api-4.0.0.jar:na]
at
com.pointcarbon.esb.cassandra.csdao.CassandraDao.storeData(CassandraDao.java:885)
~[esb-cassandra-api-4.0.0.jar:na]
... 14 common frames omitted
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency LOCAL_QUORUM (2 replica were required
but only 1 acknowledged the write)
at
com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:100)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Responses$Error.asException(Responses.java:122)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:477)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
~[cassandra-driver-core-3.0.0.jar:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
~[netty-handler-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
~[netty-codec-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
~[netty-codec-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:354)
~[netty-codec-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
~[netty-codec-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
~[netty-transport-4.0.33.Final.jar:4.0.33.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
~[netty-common-4.0.33.Final.jar:4.0.33.Final]
... 1 common frames omitted
Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra
timeout during write query at consistency LOCAL_QUORUM (2 replica were required
but only 1 acknowledged the write)
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:59)
~[cassandra-driver-core-3.0.0.jar:na]
at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:266)
~[cassandra-driver-core-3.0.0.jar:na]
at
com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:246)
~[cassandra-driver-core-3.0.0.jar:na]
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
~[netty-codec-4.0.33.Final.jar:4.0.33.Final]
... 15 common frames omitted
ERROR 2016-09-01 01:50:25,185 [pool-17-thread-5] [objectid=100081245
objecttypeid=3 rundatetime=2016-09-01T00:00:00
guid=51ea5d505dab4273aa65f779be71c4e1
sessionguid=132c1d5b6bf142a2ad4475c88dd6595a] c.p.e.t.o.s.AqService: Responce
from callback was not ok, rolling back
....................
DEBUG [GossipStage:1] 2016-09-01 06:51:46,531 FailureDetector.java:456 -
Ignoring interval time of 2196189000 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:51:46,532 FailureDetector.java:456 -
Ignoring interval time of 2000290754 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:51:50,338 FailureDetector.java:456 -
Ignoring interval time of 2050191200 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:51:50,338 FailureDetector.java:456 -
Ignoring interval time of 2001640657 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:51:54,787 FailureDetector.java:456 -
Ignoring interval time of 2000290870 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:51:57,788 FailureDetector.java:456 -
Ignoring interval time of 2094372662 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:51:58,788 FailureDetector.java:456 -
Ignoring interval time of 2094686638 for /10.51.24.48
INFO [HintsDispatcher:192] 2016-09-01 06:52:05,949 HintsStore.java:126 -
Deleted hint file 6b3d0b0a-7fce-4fa8-88d1-13975268fa6e-1472712703758-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:52:05,955
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
6b3d0b0a-7fce-4fa8-88d1-13975268fa6e-1472712703758-1.hints to endpoint
6b3d0b0a-7fce-4fa8-88d1-13975268fa6e
DEBUG [GossipStage:1] 2016-09-01 06:52:06,575 FailureDetector.java:456 -
Ignoring interval time of 2234051817 for /10.51.24.47
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,067 HintsStore.java:126 -
Deleted hint file 75725304-e02a-4a82-a521-821b80790418-1472712703782-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,075
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
75725304-e02a-4a82-a521-821b80790418-1472712703782-1.hints to endpoint
75725304-e02a-4a82-a521-821b80790418
DEBUG [GossipStage:1] 2016-09-01 06:52:07,341 FailureDetector.java:456 -
Ignoring interval time of 2000447147 for /10.51.24.48
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,769 HintsStore.java:126 -
Deleted hint file a2a99acc-84eb-4f1e-87ba-1fae8449a9e5-1472712703796-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,774
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
a2a99acc-84eb-4f1e-87ba-1fae8449a9e5-1472712703796-1.hints to endpoint
a2a99acc-84eb-4f1e-87ba-1fae8449a9e5
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,801 HintsStore.java:126 -
Deleted hint file ad077318-6531-498e-bf5a-14ac339d1a45-1472712703813-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:52:07,807
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
ad077318-6531-498e-bf5a-14ac339d1a45-1472712703813-1.hints to endpoint
ad077318-6531-498e-bf5a-14ac339d1a45
DEBUG [GossipStage:1] 2016-09-01 06:52:08,695 FailureDetector.java:456 -
Ignoring interval time of 2403623610 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:08,695 FailureDetector.java:456 -
Ignoring interval time of 2119970414 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:11,577 FailureDetector.java:456 -
Ignoring interval time of 2881627341 for /10.51.24.47
DEBUG [GossipStage:1] 2016-09-01 06:52:13,697 FailureDetector.java:456 -
Ignoring interval time of 2119666312 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:52:13,697 FailureDetector.java:456 -
Ignoring interval time of 2119625747 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:17,697 FailureDetector.java:456 -
Ignoring interval time of 2404084742 for /10.235.14.73
DEBUG [SlabPoolCleaner] 2016-09-01 06:52:22,985 ColumnFamilyStore.java:1192 -
Flushing largest CFS(Keyspace='pc_object_data_beta', ColumnFamily='vsc_data')
to free up room. Used total: 0.50/0.00, live: 0.50/0.00, flushing: 0.00/0.00,
this: 0.28/0.28
DEBUG [SlabPoolCleaner] 2016-09-01 06:52:22,985 ColumnFamilyStore.java:846 -
Enqueuing flush of vsc_data: 290612695 (28%) on-heap, 0 (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:52:23,095
Memtable.java:405 - Writing Memtable-vsc_data@1622186130(56.771MiB serialized
bytes, 411187 ops, 28%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
INFO [HintsDispatcher:192] 2016-09-01 06:52:25,119 HintsStore.java:126 -
Deleted hint file ad077318-6531-498e-bf5a-14ac339d1a45-1472712723759-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:52:25,126
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
ad077318-6531-498e-bf5a-14ac339d1a45-1472712723759-1.hints to endpoint
ad077318-6531-498e-bf5a-14ac339d1a45
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:52:26,958
Memtable.java:433 - Completed flushing
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1480-big-Data.db
(21.552MiB) for commitlog position ReplayPosition(segmentId=1471607300705,
position=2625158)
DEBUG [MemtableFlushWriter:791] 2016-09-01 06:52:27,428
ColumnFamilyStore.java:1120 - Flushed to
[BigTableReader(path='/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1480-big-Data.db')]
(1 sstables, 8810717 bytes), biggest 8810717 bytes, smallest 8810717 bytes
DEBUG [GossipStage:1] 2016-09-01 06:52:27,793 FailureDetector.java:456 -
Ignoring interval time of 2000786962 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:28,345 FailureDetector.java:456 -
Ignoring interval time of 2001408660 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:33,700 FailureDetector.java:456 -
Ignoring interval time of 2352663300 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:34,703 FailureDetector.java:456 -
Ignoring interval time of 2326623499 for /10.51.24.47
INFO [Service Thread] 2016-09-01 06:52:37,245 GCInspector.java:284 -
ConcurrentMarkSweep GC in 275ms. CMS Old Gen: 2592185288 -> 477959808; Par
Eden Space: 3167216 -> 342859896; Par Survivor Space: 31866328 -> 50269968
INFO [Service Thread] 2016-09-01 06:52:37,245 StatusLogger.java:52 - Pool Name
Active Pending Completed Blocked All Time Blocked
INFO [Service Thread] 2016-09-01 06:52:37,249 StatusLogger.java:56 -
MutationStage 0 0 697543484 0
0
INFO [Service Thread] 2016-09-01 06:52:37,249 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 - ReadStage
2 0 3449357 0 0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 -
RequestResponseStage 0 0 1135763500 0
0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 -
ReadRepairStage 0 0 28 0
0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 -
CounterMutationStage 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 - MiscStage
0 0 0 0 0
INFO [Service Thread] 2016-09-01 06:52:37,250 StatusLogger.java:56 -
CompactionExecutor 0 0 941118 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
MemtableReclaimMemory 0 0 2165 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
PendingRangeCalculator 0 0 8 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
GossipStage 0 0 4281717 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
SecondaryIndexManagement 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
HintsDispatcher 0 0 8273 0
0
INFO [Service Thread] 2016-09-01 06:52:37,251 StatusLogger.java:56 -
MigrationStage 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
MemtablePostFlush 0 0 2464 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
PerDiskMemtableFlushWriter_0 0 0 2165 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
ValidationExecutor 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 - Sampler
0 0 0 0 0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
MemtableFlushWriter 0 0 2164 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
InternalResponseStage 0 0 13528920 0
0
INFO [Service Thread] 2016-09-01 06:52:37,252 StatusLogger.java:56 -
AntiEntropyStage 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:56 -
CacheCleanupExecutor 0 0 0 0
0
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:56 -
Native-Transport-Requests 128 132 258631922 4
26070160
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:66 -
CompactionManager 0 0
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:78 -
MessagingService n/a 0/0
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:88 - Cache
Type Size Capacity KeysToSave
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:90 - KeyCache
7040008 104857600 all
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:96 - RowCache
0 0 all
INFO [Service Thread] 2016-09-01 06:52:37,253 StatusLogger.java:103 - Table
Memtable ops,data
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
pointconnect_hdc.last_updates 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
pointconnect_hdc.ftp_meta 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system_distributed.parent_repair_history 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system_distributed.repair_history 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.compaction_history 6,1577
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.hints 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.schema_aggregates 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.IndexInfo 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.schema_columnfamilies 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.schema_triggers 0,0
INFO [Service Thread] 2016-09-01 06:52:37,254 StatusLogger.java:106 -
system.size_estimates 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.schema_functions 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.paxos 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.views_builds_in_progress 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.built_views 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.peer_events 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.range_xfers 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.peers 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.batches 13342,111952185
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.schema_keyspaces 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.schema_usertypes 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.local 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.sstable_activity 158,6702
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.available_ranges 0,0
INFO [Service Thread] 2016-09-01 06:52:37,255 StatusLogger.java:106 -
system.batchlog 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system.schema_columns 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.rapido_metadata 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.document_version 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.publisher 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.rapido_schema_version 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.main_document 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
rapidokeyspace.user_details 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.columns 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.types 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.indexes 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.keyspaces 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.dropped_columns 0,0
INFO [Service Thread] 2016-09-01 06:52:37,256 StatusLogger.java:106 -
system_schema.aggregates 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_schema.triggers 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_schema.tables 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_schema.views 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_schema.functions 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
trailblazer.stack_data 5283,83124671
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
trailblazer.stack_market_rundates 17805,904
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_auth.roles 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_auth.role_members 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_auth.resource_role_permissons_index 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_auth.role_permissions 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
pointconnect_dtc.last_updates 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
pointconnect_dtc.ftp_meta 0,0
INFO [Service Thread] 2016-09-01 06:52:37,257 StatusLogger.java:106 -
system_traces.sessions 0,0
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
system_traces.events 0,0
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.versions 12018,238322
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.vsc_data 309009,44765696
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.last_updates 12010,213131
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.vsc_hash 163194,4410704
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.object_model 6,332
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.ftp_meta 0,0
INFO [Service Thread] 2016-09-01 06:52:37,258 StatusLogger.java:106 -
pc_object_data_beta.object_definition 4,32
DEBUG [GossipStage:1] 2016-09-01 06:52:37,797 FailureDetector.java:456 -
Ignoring interval time of 2451477676 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:38,346 FailureDetector.java:456 -
Ignoring interval time of 2229360621 for /10.235.14.72
DEBUG [GossipStage:1] 2016-09-01 06:52:45,799 FailureDetector.java:456 -
Ignoring interval time of 2094607408 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:46,705 FailureDetector.java:456 -
Ignoring interval time of 2000506630 for /10.235.14.72
DEBUG [GossipStage:1] 2016-09-01 06:52:48,713 FailureDetector.java:456 -
Ignoring interval time of 2007742072 for /10.51.24.47
DEBUG [GossipStage:1] 2016-09-01 06:52:48,713 FailureDetector.java:456 -
Ignoring interval time of 2000958300 for /10.235.14.72
DEBUG [GossipStage:1] 2016-09-01 06:52:49,705 FailureDetector.java:456 -
Ignoring interval time of 2122511748 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:51,349 FailureDetector.java:456 -
Ignoring interval time of 2635734580 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:51,706 FailureDetector.java:456 -
Ignoring interval time of 2342871486 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:52:52,714 FailureDetector.java:456 -
Ignoring interval time of 2130325654 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:56,801 FailureDetector.java:456 -
Ignoring interval time of 2094615884 for /10.51.24.46
DEBUG [GossipStage:1] 2016-09-01 06:52:56,801 FailureDetector.java:456 -
Ignoring interval time of 3087031906 for /10.235.14.72
DEBUG [GossipStage:1] 2016-09-01 06:52:57,801 FailureDetector.java:456 -
Ignoring interval time of 2000591900 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:52:57,802 FailureDetector.java:456 -
Ignoring interval time of 2419387841 for /10.51.24.47
DEBUG [SlabPoolCleaner] 2016-09-01 06:53:02,408 ColumnFamilyStore.java:1192 -
Flushing largest CFS(Keyspace='pc_object_data_beta', ColumnFamily='vsc_data')
to free up room. Used total: 0.50/0.00, live: 0.50/0.00, flushing: 0.00/0.00,
this: 0.28/0.28
DEBUG [SlabPoolCleaner] 2016-09-01 06:53:02,408 ColumnFamilyStore.java:846 -
Enqueuing flush of vsc_data: 290550643 (28%) on-heap, 0 (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:53:02,493
Memtable.java:405 - Writing Memtable-vsc_data@129799095(56.830MiB serialized
bytes, 411295 ops, 28%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [GossipStage:1] 2016-09-01 06:53:02,802 FailureDetector.java:456 -
Ignoring interval time of 2000465981 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:53:02,802 FailureDetector.java:456 -
Ignoring interval time of 2000626863 for /10.51.24.46
INFO [HintsDispatcher:192] 2016-09-01 06:53:04,614 HintsStore.java:126 -
Deleted hint file 6b3d0b0a-7fce-4fa8-88d1-13975268fa6e-1472712763759-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:53:04,624
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
6b3d0b0a-7fce-4fa8-88d1-13975268fa6e-1472712763759-1.hints to endpoint
6b3d0b0a-7fce-4fa8-88d1-13975268fa6e
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:53:05,777
Memtable.java:433 - Completed flushing
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1481-big-Data.db
(21.570MiB) for commitlog position ReplayPosition(segmentId=1471607300706,
position=8372249)
DEBUG [CompactionExecutor:6300] 2016-09-01 06:53:06,446 CompactionTask.java:150
- Compacting (bc6d5bd0-7010-11e6-be94-f99b78ee5bd5)
[/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1481-big-Data.db:level=0,
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1478-big-Data.db:level=0,
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1479-big-Data.db:level=0,
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1480-big-Data.db:level=0,
]
DEBUG [MemtableFlushWriter:791] 2016-09-01 06:53:06,448
ColumnFamilyStore.java:1120 - Flushed to
[BigTableReader(path='/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1481-big-Data.db')]
(1 sstables, 8665481 bytes), biggest 8665481 bytes, smallest 8665481 bytes
DEBUG [GossipStage:1] 2016-09-01 06:53:07,589 FailureDetector.java:456 -
Ignoring interval time of 3180651768 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:53:07,732 FailureDetector.java:456 -
Ignoring interval time of 2016069919 for /10.235.14.73
DEBUG [GossipStage:1] 2016-09-01 06:53:09,718 FailureDetector.java:456 -
Ignoring interval time of 2129033578 for /10.51.24.48
DEBUG [GossipStage:1] 2016-09-01 06:53:13,734 FailureDetector.java:456 -
Ignoring interval time of 2015828926 for /10.235.14.72
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,307 HintsStore.java:126 -
Deleted hint file 75725304-e02a-4a82-a521-821b80790418-1472712773766-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,314
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
75725304-e02a-4a82-a521-821b80790418-1472712773766-1.hints to endpoint
75725304-e02a-4a82-a521-821b80790418
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,539 HintsStore.java:126 -
Deleted hint file a2a99acc-84eb-4f1e-87ba-1fae8449a9e5-1472712773793-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,562
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
a2a99acc-84eb-4f1e-87ba-1fae8449a9e5-1472712773793-1.hints to endpoint
a2a99acc-84eb-4f1e-87ba-1fae8449a9e5
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,607 HintsStore.java:126 -
Deleted hint file 017cc31e-321d-4964-9087-ba064c0ca39a-1472712773810-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,614
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
017cc31e-321d-4964-9087-ba064c0ca39a-1472712773810-1.hints to endpoint
017cc31e-321d-4964-9087-ba064c0ca39a
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,665 HintsStore.java:126 -
Deleted hint file ad077318-6531-498e-bf5a-14ac339d1a45-1472712773932-1.hints
INFO [HintsDispatcher:192] 2016-09-01 06:53:15,671
HintsDispatchExecutor.java:263 - Finished hinted handoff of file
ad077318-6531-498e-bf5a-14ac339d1a45-1472712773932-1.hints to endpoint
ad077318-6531-498e-bf5a-14ac339d1a45
DEBUG [SlabPoolCleaner] 2016-09-01 06:53:16,663 ColumnFamilyStore.java:1192 -
Flushing largest CFS(Keyspace='pc_object_data_beta', ColumnFamily='vsc_data')
to free up room. Used total: 0.50/0.00, live: 0.50/0.00, flushing: 0.00/0.00,
this: 0.28/0.28
DEBUG [SlabPoolCleaner] 2016-09-01 06:53:16,664 ColumnFamilyStore.java:846 -
Enqueuing flush of vsc_data: 290440556 (28%) on-heap, 0 (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:53:16,736
Memtable.java:405 - Writing Memtable-vsc_data@987459449(56.795MiB serialized
bytes, 411149 ops, 28%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [GossipStage:1] 2016-09-01 06:53:17,829 FailureDetector.java:456 -
Ignoring interval time of 2419345194 for /10.51.24.48
WARN [SharedPool-Worker-86] 2016-09-01 06:53:19,440 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034567
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-48] 2016-09-01 06:53:19,451 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034565
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-153] 2016-09-01 06:53:19,452 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034563
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-48] 2016-09-01 06:53:19,453 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034565
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-104] 2016-09-01 06:53:19,488 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034566
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-115] 2016-09-01 06:53:19,496 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034568
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-110] 2016-09-01 06:53:19,509 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034568
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-110] 2016-09-01 06:53:19,509 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034568
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-78] 2016-09-01 06:53:19,512 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034564
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-78] 2016-09-01 06:53:19,512 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034564
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-94] 2016-09-01 06:53:19,513 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034563
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-94] 2016-09-01 06:53:19,513 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034563
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-96] 2016-09-01 06:53:19,523 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034564
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-96] 2016-09-01 06:53:19,524 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034564
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-89] 2016-09-01 06:53:19,524 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034565
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-89] 2016-09-01 06:53:19,525 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034565
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-119] 2016-09-01 06:53:19,525 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034567
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-119] 2016-09-01 06:53:19,525 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034567
LIMIT 5000 (see tombstone_warn_threshold)
DEBUG [PerDiskMemtableFlushWriter_0:812] 2016-09-01 06:53:19,569
Memtable.java:433 - Completed flushing
/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1483-big-Data.db
(22.177MiB) for commitlog position ReplayPosition(segmentId=1471607300707,
position=14140036)
DEBUG [GossipStage:1] 2016-09-01 06:53:21,025 FailureDetector.java:456 -
Ignoring interval time of 2431119456 for /10.51.24.47
DEBUG [GossipStage:1] 2016-09-01 06:53:21,025 FailureDetector.java:456 -
Ignoring interval time of 2431274148 for /10.51.24.46
WARN [SharedPool-Worker-75] 2016-09-01 06:53:21,082 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034568
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-130] 2016-09-01 06:53:21,410 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034566
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-130] 2016-09-01 06:53:21,411 ReadCommand.java:481 -
Read 5000 live rows and 10000 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034566
LIMIT 5000 (see tombstone_warn_threshold)
WARN [SharedPool-Worker-76] 2016-09-01 06:53:21,723 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034566
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-76] 2016-09-01 06:53:21,724 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034566
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-127] 2016-09-01 06:53:21,796 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034568
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-127] 2016-09-01 06:53:21,797 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034568
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-115] 2016-09-01 06:53:22,022 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034566
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-115] 2016-09-01 06:53:22,024 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034566
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
DEBUG [MemtableFlushWriter:791] 2016-09-01 06:53:22,191
ColumnFamilyStore.java:1120 - Flushed to
[BigTableReader(path='/data3/pcdtccas05t/data/pc_object_data_beta/vsc_data-2d7c9d703d2311e6a68613bc191c16cf/ma-1483-big-Data.db')]
(1 sstables, 8850095 bytes), biggest 8850095 bytes, smallest 8850095 bytes
WARN [SharedPool-Worker-75] 2016-09-01 06:53:22,419 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034565
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-75] 2016-09-01 06:53:22,419 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034565
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-136] 2016-09-01 06:53:22,428 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034564
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-136] 2016-09-01 06:53:22,428 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034564
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-55] 2016-09-01 06:53:22,593 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034563
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-55] 2016-09-01 06:53:22,597 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472707803000, 111034563
AND forecastdate > 1472707803000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-149] 2016-09-01 06:53:22,693 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034565
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-149] 2016-09-01 06:53:22,694 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034565
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
WARN [SharedPool-Worker-161] 2016-09-01 06:53:23,333 ReadCommand.java:481 -
Read 3760 live rows and 7520 tombstone cells for query SELECT * FROM
pc_object_data_beta.vsc_data WHERE rundate, vscid = 1472653906000, 111034563
AND forecastdate > 1472653906000 AND valuedate > 1501221600000 LIMIT 5000 (see
tombstone_warn_threshold)
INFO [HintsDispatcher:192] 2016-09-01 06:53:24,292 HintsStore.java:126 -
Deleted hint file 6b3d0b0a-7fce-4fa8-88d1-13975268fa6e-1472712783759-1.hints
