LS,
We are currently in the process of migrating data from an old Cassandra
cluster to a new one. When quering data from a table that was copied
using sstableloader, we find that even at consistency level ALL, results
contain 200% duplicate entries, or worse, keep paginating and repeating
themselves forever.
To make matters more confusing, when we query the same tables using
thrift, 3.0.14 responds promptly, and the data returned is correct and
unique.
We've initiated full repairs of the column family several time, but
since Thrift does return consistent results, we doubt that the data
itself is corrupt.
Could anyone advise a way to debug this problem further? Should we
investigate the table metadata, or the network layer?
Thanks in advance,
Michiel
Attached below are the table description and trace log of a query that
repeats itself forever.
CREATE TABLE boring.medium_sized_table (
key text,
column1 text,
value varint,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 DESC)
AND bloom_filter_fp_chance = 0.1
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = 'Contains the boring statistics per hour'
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
'sstable_size_in_mb': '160'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.1
AND speculative_retry = '99PERCENTILE';
READ message received from /10.10.0.70 on
/10.10.0.66[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20
11:09:26 CEST 2017
Parsing SELECT * FROM medium_sized_table WHERE key=?; on
/10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
Preparing statement on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20
11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.66[SharedPool-Worker-8] at Wed
Sep 20 11:09:26 CEST 2017
Read-repair GLOBAL on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20
11:09:26 CEST 2017
Bloom filter allows skipping sstable 164014 on
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
reading data from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-8] at Wed
Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 163275 on /10.10.0.66[SharedPool-Worker-8] at
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 163187 on /10.10.0.66[SharedPool-Worker-8] at
Wed Sep 20 11:09:26 CEST 2017
Skipped 0/3 non-slice-intersecting sstables, included 0 due to
tombstones on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26
CEST 2017
Merging data from memtables and 2 sstables on
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
reading data from /10.10.0.78 on /10.10.0.70[SharedPool-Worker-8] at Wed
Sep 20 11:09:26 CEST 2017
READ message received from /10.10.0.70 on
/10.10.0.78[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20
11:09:26 CEST 2017
Sending READ message to /10.10.0.66 on
/10.10.0.70[MessagingService-Outgoing-/10.10.0.66] at Wed Sep 20
11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.78[SharedPool-Worker-1] at Wed
Sep 20 11:09:26 CEST 2017
Sending READ message to /10.10.0.78 on
/10.10.0.70[MessagingService-Outgoing-/10.10.0.78] at Wed Sep 20
11:09:26 CEST 2017
Bloom filter allows skipping sstable 128083 on
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.70[SharedPool-Worker-6] at Wed
Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 127687 on
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 125855 on
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 126764 on /10.10.0.78[SharedPool-Worker-1] at
Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 124137 on
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 127128 on /10.10.0.78[SharedPool-Worker-1] at
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 125183 on /10.10.0.70[SharedPool-Worker-6] at
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 127637 on /10.10.0.78[SharedPool-Worker-1] at
Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 124188 on
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Skipped 0/5 non-slice-intersecting sstables, included 0 due to
tombstones on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26
CEST 2017
Skipped 0/4 non-slice-intersecting sstables, included 0 due to
tombstones on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26
CEST 2017
Merging data from memtables and 3 sstables on
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Merging data from memtables and 1 sstables on
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.70[SharedPool-Worker-6]
at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.66[SharedPool-Worker-8]
at Wed Sep 20 11:09:26 CEST 2017
Enqueuing response to /10.10.0.70 on /10.10.0.66[SharedPool-Worker-8] at
Wed Sep 20 11:09:26 CEST 2017
Sending REQUEST_RESPONSE message to cassh01/10.10.0.70 on
/10.10.0.66[MessagingService-Outgoing-cassh01/10.10.0.70] at Wed Sep 20
11:09:26 CEST 2017
REQUEST_RESPONSE message received from /10.10.0.66 on
/10.10.0.70[MessagingService-Incoming-/10.10.0.66] at Wed Sep 20
11:09:26 CEST 2017
Processing response from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-7]
at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.78[SharedPool-Worker-1]
at Wed Sep 20 11:09:26 CEST 2017
Enqueuing response to /10.10.0.70 on /10.10.0.78[SharedPool-Worker-1] at
Wed Sep 20 11:09:26 CEST 2017
Sending REQUEST_RESPONSE message to /10.10.0.70 on
/10.10.0.78[MessagingService-Outgoing-/10.10.0.70] at Wed Sep 20
11:09:26 CEST 2017
REQUEST_RESPONSE message received from /10.10.0.78 on
/10.10.0.70[MessagingService-Incoming-/10.10.0.78] at Wed Sep 20
11:09:26 CEST 2017
Processing response from /10.10.0.78 on
/10.10.0.70[SharedPool-Worker-10] at Wed Sep 20 11:09:26 CEST 2017
Initiating read-repair on /10.10.0.70[SharedPool-Worker-10] at Wed Sep
20 11:09:26 CEST 2017
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
