Hi Aaron,
At first, before I go with a lot of logs:
I'm considering a problem related to this issue:
https://issues.apache.org/jira/browse/CASSANDRA-4905
Let's say the tombstone on one of the nodes (X) is gcable and was not
compacted (purged) so far. After it was created we re-created this row,
but due some problems it was written only to the second node (Y), so we
have "live" data on node Y which is newer than the gcable tombstone on
replica node X. Some time ago we did NOT repair our cluster for a while
(well, pretty long while), so it's possible that such situation happened.
My concern is: will AntiEntropy ignore this tombstone only, or basically
everything related to the row key that this tombstone was created for?
If it's not the case, here are the answers you asked for :-)
What version are you on ?
1.2.1
(plus CASSANDRA-5298 & CASSANDRA-5299 patches to be exact ;-) )
Can you run a repair on the CF and check:
Does the repair detect differences in the CF and stream changes ?
> After the streaming does it run a secondary index rebuild on the new
sstable ? (Should be in the logs)
I'm attaching a log file (cssa-repair.log).
Just to clarify: the key I use for tests belongs to *:1:7 node and *:2:1
is a replica for that node (checked with nodetool getendpoints).
Yesterday I was repairing this CF cluster-wide, but to (hopefully) make
debugging simplier, what I send you is related only to these two nodes.
So as I understand these logs: no changes have been detected and nothing
was streamed. Indexes have not been rebuilt, obviously.
However, on the other hand I'd expect to get "Nothing to repair for
keyspace production" in nodetool output in this case - am I wrong? I'm a
bit confused with the info I get here ;-)
Can you provide the full query trace ?
I'm attaching two files, as this stack trace is pretty long:
no-index.log (query by row key) and index.log (query by indexed column).
M.
*** When repairing node 1 (requested key belongs to its primary range):
* node 1 log:
INFO [Thread-1780798] 2013-04-04 08:17:37,286 StorageService.java (line 2311) Starting repair command #8, repairing 1 ranges for keyspace production
INFO [AntiEntropySessions:9] 2013-04-04 08:17:37,288 AntiEntropyService.java (line 652) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] new session: will sync cssa01-07/2001:5d19:13:169:0:1:1:7, /2001:5d19:13:169:0:1:2:1 on range (5671372782015641057722910123862803524,11342745564031282115445820247725607048] for production.[Users]
INFO [AntiEntropySessions:9] 2013-04-04 08:17:37,288 AntiEntropyService.java (line 857) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] requesting merkle trees for Users (to [/2001:5d19:13:169:0:1:2:1, cssa01-07/2001:5d19:13:169:0:1:1:7])
INFO [AntiEntropyStage:1] 2013-04-04 08:17:37,326 AntiEntropyService.java (line 214) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] Received merkle tree for Users from /2001:5d19:13:169:0:1:1:7
INFO [AntiEntropyStage:1] 2013-04-04 08:17:37,326 AntiEntropyService.java (line 214) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] Received merkle tree for Users from /2001:5d19:13:169:0:1:2:1
INFO [AntiEntropyStage:1] 2013-04-04 08:17:37,340 AntiEntropyService.java (line 988) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] Endpoints /2001:5d19:13:169:0:1:1:7 and /2001:5d19:13:169:0:1:2:1 are consistent for Users
INFO [AntiEntropyStage:1] 2013-04-04 08:17:37,340 AntiEntropyService.java (line 764) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] Users is fully synced
INFO [AntiEntropySessions:9] 2013-04-04 08:17:37,340 AntiEntropyService.java (line 698) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] session completed successfully
* node 2 log:
INFO [AntiEntropyStage:1] 2013-04-04 08:17:37,302 AntiEntropyService.java (line 246) [repair #1c2df170-9d00-11e2-938f-11f9b91aba37] Sending completed merkle tree to /2001:5d19:13:169:0:1:1:7 for (production,Users)
*** When repairing node 2 (replica of node 1):
* node 2 log:
INFO [Thread-1778894] 2013-04-04 08:22:27,727 StorageService.java (line 2311) Starting repair command #8, repairing 1 ranges for keyspace production
INFO [AntiEntropySessions:9] 2013-04-04 08:22:27,728 AntiEntropyService.java (line 652) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] new session: will sync cssa02-01/2001:5d19:13:169:0:1:2:1, /2001:5d19:13:169:0:1:1:8 on range (11342745564031282115445820247725607048,17014118346046923173168730371588410572] for production.[Users]
INFO [AntiEntropySessions:9] 2013-04-04 08:22:27,728 AntiEntropyService.java (line 857) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] requesting merkle trees for Users (to [/2001:5d19:13:169:0:1:1:8, cssa02-01/2001:5d19:13:169:0:1:2:1])
INFO [ValidationExecutor:20] 2013-04-04 08:22:27,729 ColumnFamilyStore.java (line 640) Enqueuing flush of Memtable-Users.Users_active_idx@1715128181(28/180 serialized/live bytes, 4 ops)
INFO [ValidationExecutor:20] 2013-04-04 08:22:27,730 ColumnFamilyStore.java (line 640) Enqueuing flush of Memtable-Users.Users_email_idx@1740076506(28/1251 serialized/live bytes, 4 ops)
INFO [ValidationExecutor:20] 2013-04-04 08:22:27,730 ColumnFamilyStore.java (line 640) Enqueuing flush of Memtable-Users.Users_username_idx@1033531496(28/1103 serialized/live bytes, 4 ops)
INFO [FlushWriter:51] 2013-04-04 08:22:27,730 Memtable.java (line 447) Writing Memtable-Users.Users_active_idx@1715128181(28/180 serialized/live bytes, 4 ops)
INFO [FlushWriter:51] 2013-04-04 08:22:27,735 Memtable.java (line 481) Completed flushing /cassandra/production/Users/production-Users.Users_active_idx-ib-23-Data.db (63 bytes) for commitlog position ReplayPosition(segmentId=1362644396554, position=25894629)
INFO [FlushWriter:51] 2013-04-04 08:22:27,737 Memtable.java (line 447) Writing Memtable-Users.Users_email_idx@1740076506(28/1251 serialized/live bytes, 4 ops)
INFO [FlushWriter:51] 2013-04-04 08:22:27,740 Memtable.java (line 481) Completed flushing /cassandra/production/Users/production-Users.Users_email_idx-ib-23-Data.db (84 bytes) for commitlog position ReplayPosition(segmentId=1362644396554, position=25894629)
INFO [FlushWriter:51] 2013-04-04 08:22:27,740 Memtable.java (line 447) Writing Memtable-Users.Users_username_idx@1033531496(28/1103 serialized/live bytes, 4 ops)
INFO [FlushWriter:51] 2013-04-04 08:22:27,747 Memtable.java (line 481) Completed flushing /cassandra/production/Users/production-Users.Users_username_idx-ib-23-Data.db (84 bytes) for commitlog position ReplayPosition(segmentId=1362644396554, position=25894629)
INFO [AntiEntropyStage:1] 2013-04-04 08:22:27,766 AntiEntropyService.java (line 214) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] Received merkle tree for Users from /2001:5d19:13:169:0:1:1:8
INFO [AntiEntropyStage:1] 2013-04-04 08:22:27,782 AntiEntropyService.java (line 214) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] Received merkle tree for Users from /2001:5d19:13:169:0:1:2:1
INFO [AntiEntropyStage:1] 2013-04-04 08:22:27,798 AntiEntropyService.java (line 988) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] Endpoints /2001:5d19:13:169:0:1:1:8 and /2001:5d19:13:169:0:1:2:1 are consistent for Users
INFO [AntiEntropyStage:1] 2013-04-04 08:22:27,798 AntiEntropyService.java (line 764) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] Users is fully synced
INFO [AntiEntropySessions:9] 2013-04-04 08:22:27,798 AntiEntropyService.java (line 698) [repair #c94b91f0-9d00-11e2-843a-aba1caf27753] session completed successfully
cqlsh:production> select * from "Users" WHERE username = 'somesec...@email.com';
Tracing session: ba1b8f80-9d02-11e2-8f39-13ee1d303762
activity | timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------------------------------+--------------+---------------------------+----------------
execute_cql3_query | 10:36:21,277 | 2001:5d19:13:169:0:1:1:1 | 0
Parsing statement | 10:36:21,277 | 2001:5d19:13:169:0:1:1:1 | 31
Peparing statement | 10:36:21,278 | 2001:5d19:13:169:0:1:1:1 | 186
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,278 | 2001:5d19:13:169:0:1:2:7 | 16
Determining replicas to query | 10:36:21,278 | 2001:5d19:13:169:0:1:1:1 | 386
Sending message to /2001:5d19:13:169:0:1:2:7 | 10:36:21,278 | 2001:5d19:13:169:0:1:1:1 | 645
Message received from /2001:5d19:13:169:0:1:2:7 | 10:36:21,279 | 2001:5d19:13:169:0:1:1:1 | 1848
Executing indexed scan for [min(-1), max(5671372782015641057722910123862803524)] | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 444
Executing single-partition query on Users.Users_username_idx | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 508
Acquiring sstable references | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 515
Merging memtable contents | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 532
Merging data from memtables and 0 sstables | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 554
Read 0 live cells and 0 tombstoned | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 573
Scanned 0 rows and matched 0 | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 609
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 620
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,279 | 2001:5d19:13:169:0:1:2:7 | 739
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,280 | 2001:5d19:13:169:0:1:2:1 | 16
Processing response from /2001:5d19:13:169:0:1:2:7 | 10:36:21,280 | 2001:5d19:13:169:0:1:1:1 | 2126
Executing indexed scan for (max(5671372782015641057722910123862803524), max(17014118346046923173168730371588410572)] | 10:36:21,280 | 2001:5d19:13:169:0:1:2:1 | 553
Sending message to /2001:5d19:13:169:0:1:2:1 | 10:36:21,280 | 2001:5d19:13:169:0:1:1:1 | 2280
Executing single-partition query on Users.Users_username_idx | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 629
Acquiring sstable references | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 634
Merging memtable contents | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 649
Partition index lookup complete for sstable 23 | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 771
Merging data from memtables and 1 sstables | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 1237
Read 0 live cells and 1 tombstoned | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 1275
Scanned 0 rows and matched 0 | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 1320
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 1330
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,281 | 2001:5d19:13:169:0:1:2:1 | 1422
Message received from /2001:5d19:13:169:0:1:2:1 | 10:36:21,282 | 2001:5d19:13:169:0:1:1:1 | 4094
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,282 | 2001:5d19:13:169:0:1:2:8 | 25
Processing response from /2001:5d19:13:169:0:1:2:1 | 10:36:21,282 | 2001:5d19:13:169:0:1:1:1 | 4407
Sending message to /2001:5d19:13:169:0:1:2:8 | 10:36:21,282 | 2001:5d19:13:169:0:1:1:1 | 4577
Message received from /2001:5d19:13:169:0:1:2:8 | 10:36:21,283 | 2001:5d19:13:169:0:1:1:1 | 5934
Executing indexed scan for (max(17014118346046923173168730371588410572), max(28356863910078205288614550619314017621)] | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 644
Executing single-partition query on Users.Users_username_idx | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 699
Acquiring sstable references | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 707
Merging memtable contents | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 720
Merging data from memtables and 0 sstables | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 742
Read 0 live cells and 0 tombstoned | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 759
Scanned 0 rows and matched 0 | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 794
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 804
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,283 | 2001:5d19:13:169:0:1:2:8 | 952
Processing response from /2001:5d19:13:169:0:1:2:8 | 10:36:21,284 | 2001:5d19:13:169:0:1:1:1 | 6194
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,284 | 2001:5d19:13:169:0:1:2:9 | 18
Sending message to /2001:5d19:13:169:0:1:2:9 | 10:36:21,284 | 2001:5d19:13:169:0:1:1:1 | 6303
Message received from /2001:5d19:13:169:0:1:2:9 | 10:36:21,285 | 2001:5d19:13:169:0:1:1:1 | 7536
Executing indexed scan for (max(28356863910078205288614550619314017621), max(39699609474109487404060370867039624669)] | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 548
Processing response from /2001:5d19:13:169:0:1:2:9 | 10:36:21,285 | 2001:5d19:13:169:0:1:1:1 | 7810
Executing single-partition query on Users.Users_username_idx | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 603
Sending message to /2001:5d19:13:169:0:1:2:2 | 10:36:21,285 | 2001:5d19:13:169:0:1:1:1 | 7970
Acquiring sstable references | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 610
Merging memtable contents | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 623
Merging data from memtables and 0 sstables | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 641
Read 0 live cells and 0 tombstoned | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 660
Scanned 0 rows and matched 0 | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 693
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 703
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,285 | 2001:5d19:13:169:0:1:2:9 | 838
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 25
Executing indexed scan for (max(39699609474109487404060370867039624669), max(51042355038140769519506191114765231716)] | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 553
Executing single-partition query on Users.Users_username_idx | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 608
Acquiring sstable references | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 615
Merging memtable contents | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 629
Merging data from memtables and 0 sstables | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 646
Read 0 live cells and 0 tombstoned | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 663
Scanned 0 rows and matched 0 | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 696
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,286 | 2001:5d19:13:169:0:1:2:2 | 706
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,287 | 2001:5d19:13:169:0:1:2:10 | 25
Message received from /2001:5d19:13:169:0:1:2:2 | 10:36:21,287 | 2001:5d19:13:169:0:1:1:1 | 9237
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,287 | 2001:5d19:13:169:0:1:2:2 | 856
Processing response from /2001:5d19:13:169:0:1:2:2 | 10:36:21,287 | 2001:5d19:13:169:0:1:1:1 | 9498
Sending message to /2001:5d19:13:169:0:1:2:10 | 10:36:21,287 | 2001:5d19:13:169:0:1:1:1 | 9667
Executing indexed scan for (max(51042355038140769519506191114765231716), max(62385100602172051634952011362490838766)] | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 584
Message received from /2001:5d19:13:169:0:1:2:10 | 10:36:21,288 | 2001:5d19:13:169:0:1:1:1 | 10941
Executing single-partition query on Users.Users_username_idx | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 643
Acquiring sstable references | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 650
Merging memtable contents | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 663
Merging data from memtables and 0 sstables | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 681
Read 0 live cells and 0 tombstoned | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 697
Scanned 0 rows and matched 0 | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 735
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 746
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,288 | 2001:5d19:13:169:0:1:2:10 | 891
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,289 | 2001:5d19:13:169:0:1:2:11 | 27
Processing response from /2001:5d19:13:169:0:1:2:10 | 10:36:21,289 | 2001:5d19:13:169:0:1:1:1 | 11194
Sending message to /2001:5d19:13:169:0:1:2:11 | 10:36:21,289 | 2001:5d19:13:169:0:1:1:1 | 11288
Executing indexed scan for (max(62385100602172051634952011362490838766), max(73727846166203333750397831610216445815)] | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 560
Message received from /2001:5d19:13:169:0:1:2:11 | 10:36:21,290 | 2001:5d19:13:169:0:1:1:1 | 12419
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,290 | 2001:5d19:13:169:0:1:2:3 | 19
Executing single-partition query on Users.Users_username_idx | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 614
Processing response from /2001:5d19:13:169:0:1:2:11 | 10:36:21,290 | 2001:5d19:13:169:0:1:1:1 | 12543
Acquiring sstable references | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 620
Sending message to /2001:5d19:13:169:0:1:2:3 | 10:36:21,290 | 2001:5d19:13:169:0:1:1:1 | 12698
Merging memtable contents | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 634
Merging data from memtables and 0 sstables | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 655
Read 0 live cells and 0 tombstoned | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 669
Scanned 0 rows and matched 0 | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 701
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 711
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,290 | 2001:5d19:13:169:0:1:2:11 | 755
Message received from /2001:5d19:13:169:0:1:2:3 | 10:36:21,291 | 2001:5d19:13:169:0:1:1:1 | 13834
Executing indexed scan for (max(73727846166203333750397831610216445815), max(85070591730234615865843651857942052860)] | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 479
Processing response from /2001:5d19:13:169:0:1:2:3 | 10:36:21,291 | 2001:5d19:13:169:0:1:1:1 | 13876
Executing single-partition query on Users.Users_username_idx | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 536
Sending message to /2001:5d19:13:169:0:1:2:12 | 10:36:21,291 | 2001:5d19:13:169:0:1:1:1 | 13993
Acquiring sstable references | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 544
Merging memtable contents | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 559
Merging data from memtables and 0 sstables | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 576
Read 0 live cells and 0 tombstoned | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 590
Scanned 0 rows and matched 0 | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 626
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 636
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,291 | 2001:5d19:13:169:0:1:2:3 | 751
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 18
Executing indexed scan for (max(85070591730234615865843651857942052860), max(96413337294265897981289472105667659912)] | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 582
Executing single-partition query on Users.Users_username_idx | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 641
Acquiring sstable references | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 651
Merging memtable contents | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 667
Merging data from memtables and 0 sstables | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 686
Read 0 live cells and 0 tombstoned | 10:36:21,292 | 2001:5d19:13:169:0:1:2:12 | 702
Message received from /2001:5d19:13:169:0:1:2:12 | 10:36:21,293 | 2001:5d19:13:169:0:1:1:1 | 15305
Scanned 0 rows and matched 0 | 10:36:21,293 | 2001:5d19:13:169:0:1:2:12 | 738
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,293 | 2001:5d19:13:169:0:1:2:13 | 14
Processing response from /2001:5d19:13:169:0:1:2:12 | 10:36:21,293 | 2001:5d19:13:169:0:1:1:1 | 15436
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,293 | 2001:5d19:13:169:0:1:2:12 | 748
Sending message to /2001:5d19:13:169:0:1:2:13 | 10:36:21,293 | 2001:5d19:13:169:0:1:1:1 | 15616
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,293 | 2001:5d19:13:169:0:1:2:12 | 942
Message received from /2001:5d19:13:169:0:1:2:13 | 10:36:21,294 | 2001:5d19:13:169:0:1:1:1 | 16459
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,294 | 2001:5d19:13:169:0:1:2:5 | 16
Executing indexed scan for (max(96413337294265897981289472105667659912), max(107756082858297180096735292353393266961)] | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 377
Processing response from /2001:5d19:13:169:0:1:2:13 | 10:36:21,294 | 2001:5d19:13:169:0:1:1:1 | 16597
Executing single-partition query on Users.Users_username_idx | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 425
Sending message to /2001:5d19:13:169:0:1:2:5 | 10:36:21,294 | 2001:5d19:13:169:0:1:1:1 | 16704
Acquiring sstable references | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 431
Merging memtable contents | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 444
Merging data from memtables and 0 sstables | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 462
Read 0 live cells and 0 tombstoned | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 476
Scanned 0 rows and matched 0 | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 508
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 517
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,294 | 2001:5d19:13:169:0:1:2:13 | 560
Message received from /2001:5d19:13:169:0:1:2:5 | 10:36:21,295 | 2001:5d19:13:169:0:1:1:1 | 17929
Executing indexed scan for (max(107756082858297180096735292353393266961), max(119098828422328462212181112601118874004)] | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 571
Processing response from /2001:5d19:13:169:0:1:2:5 | 10:36:21,295 | 2001:5d19:13:169:0:1:1:1 | 17975
Executing single-partition query on Users.Users_username_idx | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 622
Acquiring sstable references | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 630
Merging memtable contents | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 645
Merging data from memtables and 0 sstables | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 664
Read 0 live cells and 0 tombstoned | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 677
Scanned 0 rows and matched 0 | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 709
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 719
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,295 | 2001:5d19:13:169:0:1:2:5 | 835
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,296 | 2001:5d19:13:169:0:1:2:14 | 14
Sending message to /2001:5d19:13:169:0:1:2:14 | 10:36:21,296 | 2001:5d19:13:169:0:1:1:1 | 18068
Executing indexed scan for (max(119098828422328462212181112601118874004), max(130441573986359744327626932848844481058)] | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 549
Message received from /2001:5d19:13:169:0:1:2:14 | 10:36:21,297 | 2001:5d19:13:169:0:1:1:1 | 19364
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,297 | 2001:5d19:13:169:0:1:2:15 | 28
Executing single-partition query on Users.Users_username_idx | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 597
Processing response from /2001:5d19:13:169:0:1:2:14 | 10:36:21,297 | 2001:5d19:13:169:0:1:1:1 | 19481
Acquiring sstable references | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 605
Sending message to /2001:5d19:13:169:0:1:2:15 | 10:36:21,297 | 2001:5d19:13:169:0:1:1:1 | 19571
Merging memtable contents | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 619
Merging data from memtables and 0 sstables | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 641
Read 0 live cells and 0 tombstoned | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 658
Scanned 0 rows and matched 0 | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 688
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 698
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,297 | 2001:5d19:13:169:0:1:2:14 | 840
Message received from /2001:5d19:13:169:0:1:2:15 | 10:36:21,298 | 2001:5d19:13:169:0:1:1:1 | 20837
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,298 | 2001:5d19:13:169:0:1:2:6 | 26
Executing indexed scan for (max(130441573986359744327626932848844481058), max(141784319550391026443072753096570088106)] | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 522
Processing response from /2001:5d19:13:169:0:1:2:15 | 10:36:21,298 | 2001:5d19:13:169:0:1:1:1 | 20897
Executing single-partition query on Users.Users_username_idx | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 590
Sending message to /2001:5d19:13:169:0:1:2:6 | 10:36:21,298 | 2001:5d19:13:169:0:1:1:1 | 21014
Acquiring sstable references | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 599
Merging memtable contents | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 614
Merging data from memtables and 0 sstables | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 636
Read 0 live cells and 0 tombstoned | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 652
Scanned 0 rows and matched 0 | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 697
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 708
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,298 | 2001:5d19:13:169:0:1:2:15 | 886
Executing indexed scan for (max(141784319550391026443072753096570088106), max(153127065114422308558518573344295695148)] | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 564
Executing single-partition query on Users.Users_username_idx | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 620
Acquiring sstable references | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 628
Merging memtable contents | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 646
Merging data from memtables and 0 sstables | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 663
Read 0 live cells and 0 tombstoned | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 678
Scanned 0 rows and matched 0 | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 711
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 722
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,299 | 2001:5d19:13:169:0:1:2:6 | 836
Message received from /2001:5d19:13:169:0:1:2:6 | 10:36:21,300 | 2001:5d19:13:169:0:1:1:1 | 22246
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,300 | 2001:5d19:13:169:0:1:2:16 | 23
Processing response from /2001:5d19:13:169:0:1:2:6 | 10:36:21,300 | 2001:5d19:13:169:0:1:1:1 | 22389
Sending message to /2001:5d19:13:169:0:1:2:16 | 10:36:21,300 | 2001:5d19:13:169:0:1:1:1 | 22558
Message received from /2001:5d19:13:169:0:1:2:16 | 10:36:21,301 | 2001:5d19:13:169:0:1:1:1 | 23798
Executing indexed scan for (max(153127065114422308558518573344295695148), max(164469810678453590673964393592021302203)] | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 509
Message received from /2001:5d19:13:169:0:1:1:1 | 10:36:21,301 | 2001:5d19:13:169:0:1:2:7 | 12
Processing response from /2001:5d19:13:169:0:1:2:16 | 10:36:21,301 | 2001:5d19:13:169:0:1:1:1 | 23881
Executing single-partition query on Users.Users_username_idx | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 581
Sending message to /2001:5d19:13:169:0:1:2:7 | 10:36:21,301 | 2001:5d19:13:169:0:1:1:1 | 23953
Acquiring sstable references | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 589
Merging memtable contents | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 604
Merging data from memtables and 0 sstables | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 626
Read 0 live cells and 0 tombstoned | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 645
Scanned 0 rows and matched 0 | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 685
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 697
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,301 | 2001:5d19:13:169:0:1:2:16 | 826
Message received from /2001:5d19:13:169:0:1:2:7 | 10:36:21,302 | 2001:5d19:13:169:0:1:1:1 | 24708
Executing indexed scan for (max(164469810678453590673964393592021302203), min(-1)] | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 307
Processing response from /2001:5d19:13:169:0:1:2:7 | 10:36:21,302 | 2001:5d19:13:169:0:1:1:1 | 24829
Executing single-partition query on Users.Users_username_idx | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 341
Acquiring sstable references | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 346
Merging memtable contents | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 354
Merging data from memtables and 0 sstables | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 364
Read 0 live cells and 0 tombstoned | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 377
Scanned 0 rows and matched 0 | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 394
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 403
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:36:21,302 | 2001:5d19:13:169:0:1:2:7 | 520
Request complete | 10:36:21,302 | 2001:5d19:13:169:0:1:1:1 | 25121
cqlsh:production> select * from "Users" WHERE key = '97080e62-09ce-431a-8a3e-9235817d9ebf';
key | active | date_created | default_account_id | email | username
-------------------------------------------------+--------+--------------+--------------------+---------------------------+---------------------------
\x97\x08\x0eb\t\xceC\x1a\x8a>\x925\x81}\x9e\xbf | 1 | 1361360828 | null | somesec...@email.com | somesec...@email.com
Tracing session: 917d4e60-9d02-11e2-8f39-13ee1d303762
activity | timestamp | source | source_elapsed
----------------------------------------------------+--------------+--------------------------+----------------
execute_cql3_query | 10:35:13,134 | 2001:5d19:13:169:0:1:1:1 | 0
Parsing statement | 10:35:13,134 | 2001:5d19:13:169:0:1:1:1 | 31
Peparing statement | 10:35:13,134 | 2001:5d19:13:169:0:1:1:1 | 174
Sending message to /2001:5d19:13:169:0:1:2:1 | 10:35:13,134 | 2001:5d19:13:169:0:1:1:1 | 430
Sending message to /2001:5d19:13:169:0:1:1:7 | 10:35:13,134 | 2001:5d19:13:169:0:1:1:1 | 454
Message received from /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 16
Message received from /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 21
Executing single-partition query on Users | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 444
Executing single-partition query on Users | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 476
Acquiring sstable references | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 464
Acquiring sstable references | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 497
Merging memtable contents | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 479
Merging memtable contents | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 509
Key cache hit for sstable 12 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 528
Key cache hit for sstable 14 | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 546
Merging data from sstable 12 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 550
Merging data from sstable 14 | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 571
Key cache hit for sstable 14 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 567
Collating all results | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 583
Merging data from sstable 14 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 578
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 638
Key cache hit for sstable 13 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 597
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:1:7 | 733
Merging data from sstable 13 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 606
Collating all results | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 616
Enqueuing response to /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 680
Sending message to /2001:5d19:13:169:0:1:1:1 | 10:35:13,135 | 2001:5d19:13:169:0:1:2:1 | 799
Message received from /2001:5d19:13:169:0:1:1:7 | 10:35:13,136 | 2001:5d19:13:169:0:1:1:1 | 1717
Message received from /2001:5d19:13:169:0:1:2:1 | 10:35:13,136 | 2001:5d19:13:169:0:1:1:1 | 1823
Processing response from /2001:5d19:13:169:0:1:1:7 | 10:35:13,136 | 2001:5d19:13:169:0:1:1:1 | 2235
Processing response from /2001:5d19:13:169:0:1:2:1 | 10:35:13,136 | 2001:5d19:13:169:0:1:1:1 | 2443
Request complete | 10:35:13,136 | 2001:5d19:13:169:0:1:1:1 | 2417
