[
https://issues.apache.org/jira/browse/CASSANDRA-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863161#comment-13863161
]
Michael Shuler edited comment on CASSANDRA-6517 at 1/6/14 6:20 PM:
-------------------------------------------------------------------
I reproduced on 2.0.4 with a simple 3-node RF=3 ccm cluster. Repair did not
change the results and querying a different node has the same result.
{code}
cqlsh:mwerrch> select key,node,computer from "B4Container_Demo" where
computer=50;
(0 rows)
cqlsh:mwerrch> select * from "B4Container_Demo";
key | archived | bytes | computer | deleted |
description | doarchive | filename | first | frames | ifversion | imported |
jobid | keepuntil | nextchunk | node | recordingkey | recstart | recstop |
simulationid | systemstart | systemstop | tapelabel | version
--------------------------------------+----------+-------+----------+---------+-------------+-----------+----------+-------+--------+-----------+----------+-------+-----------+-----------+------+--------------+----------+---------+--------------+-------------+------------+-----------+---------
78c70562-1f98-3971-9c28-2c3d8e09c10f | null | null | 50 | null |
null | null | null | null | null | null | null |
null | null | null | 50 | null | null | null |
null | null | null | null | null
(1 rows)
cqlsh:mwerrch>
{code}
----
Update1: full cluster restart shows the same results
----
Update2: debug logs, query run on node1 "... where computer=50" resulting in (0
rows)
{code}
DEBUG [Thrift:1] 2014-01-06 12:11:42,122 CassandraServer.java (line 1954)
execute_cql3_query
DEBUG [Thrift:1] 2014-01-06 12:11:42,136 AbstractReplicationStrategy.java (line
86) clearing cached endpoints
DEBUG [WRITE-/127.0.0.2] 2014-01-06 12:11:42,270 OutboundTcpConnection.java
(line 290) attempting to connect to /127.0.0.2
INFO [HANDSHAKE-/127.0.0.2] 2014-01-06 12:11:42,271 OutboundTcpConnection.java
(line 386) Handshaking version with /127.0.0.2
==> .ccm/test/node2/logs/system.log <==
DEBUG [ACCEPT-/127.0.0.2] 2014-01-06 12:11:42,271 MessagingService.java (line
850) Connection version 7 from /127.0.0.1
DEBUG [Thread-7] 2014-01-06 12:11:42,272 IncomingTcpConnection.java (line 107)
Upgrading incoming connection to be compressed
DEBUG [Thread-7] 2014-01-06 12:11:42,274 IncomingTcpConnection.java (line 115)
Max version for /127.0.0.1 is 7
DEBUG [Thread-7] 2014-01-06 12:11:42,274 MessagingService.java (line 743)
Setting version 7 for /127.0.0.1
DEBUG [Thread-7] 2014-01-06 12:11:42,274 IncomingTcpConnection.java (line 124)
set version for /127.0.0.1 to 7
DEBUG [ReadStage:1] 2014-01-06 12:11:42,283 KeysSearcher.java (line 69)
Most-selective indexed predicate is 'B4Container_Demo.computer EQ 50'
DEBUG [ReadStage:1] 2014-01-06 12:11:42,285 FileCacheService.java (line 70)
Evicting cold readers for
/home/mshuler/.ccm/test/node2/data/system/schema_columns/system-schema_columns-jb-7-Data.db
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11033316 compared to actual usage 0
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11164665 compared to actual usage 131349
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11296014 compared to actual usage 262698
==> .ccm/test/node1/logs/system.log <==
DEBUG [Thrift:1] 2014-01-06 12:11:42,287 Tracing.java (line 159) request
complete
{code}
'select * ..." query debug log is just:
{code}
DEBUG [Thrift:1] 2014-01-06 12:18:07,087 CassandraServer.java (line 1954)
execute_cql3_query
DEBUG [Thrift:1] 2014-01-06 12:18:07,122 Tracing.java (line 159) request
complete
{code}
was (Author: mshuler):
I reproduced on 2.0.4 with a simple 3-node RF=3 ccm cluster. Repair did not
change the results and querying a different node has the same result.
{code}
cqlsh:mwerrch> select key,node,computer from "B4Container_Demo" where
computer=50;
(0 rows)
cqlsh:mwerrch> select * from "B4Container_Demo";
key | archived | bytes | computer | deleted |
description | doarchive | filename | first | frames | ifversion | imported |
jobid | keepuntil | nextchunk | node | recordingkey | recstart | recstop |
simulationid | systemstart | systemstop | tapelabel | version
--------------------------------------+----------+-------+----------+---------+-------------+-----------+----------+-------+--------+-----------+----------+-------+-----------+-----------+------+--------------+----------+---------+--------------+-------------+------------+-----------+---------
78c70562-1f98-3971-9c28-2c3d8e09c10f | null | null | 50 | null |
null | null | null | null | null | null | null |
null | null | null | 50 | null | null | null |
null | null | null | null | null
(1 rows)
cqlsh:mwerrch>
{code}
----
Update1: full cluster restart shows the same results
----
Update2: debug logs, query run on node1 "... where computer=50" resulting in (0
rows)
{code}
DEBUG [Thrift:1] 2014-01-06 12:11:42,122 CassandraServer.java (line 1954)
execute_cql3_query
DEBUG [Thrift:1] 2014-01-06 12:11:42,136 AbstractReplicationStrategy.java (line
86) clearing cached endpoints
DEBUG [WRITE-/127.0.0.2] 2014-01-06 12:11:42,270 OutboundTcpConnection.java
(line 290) attempting to connect to /127.0.0.2
INFO [HANDSHAKE-/127.0.0.2] 2014-01-06 12:11:42,271 OutboundTcpConnection.java
(line 386) Handshaking version with /127.0.0.2
==> .ccm/test/node2/logs/system.log <==
DEBUG [ACCEPT-/127.0.0.2] 2014-01-06 12:11:42,271 MessagingService.java (line
850) Connection version 7 from /127.0.0.1
DEBUG [Thread-7] 2014-01-06 12:11:42,272 IncomingTcpConnection.java (line 107)
Upgrading incoming connection to be compressed
DEBUG [Thread-7] 2014-01-06 12:11:42,274 IncomingTcpConnection.java (line 115)
Max version for /127.0.0.1 is 7
DEBUG [Thread-7] 2014-01-06 12:11:42,274 MessagingService.java (line 743)
Setting version 7 for /127.0.0.1
DEBUG [Thread-7] 2014-01-06 12:11:42,274 IncomingTcpConnection.java (line 124)
set version for /127.0.0.1 to 7
DEBUG [ReadStage:1] 2014-01-06 12:11:42,283 KeysSearcher.java (line 69)
Most-selective indexed predicate is 'B4Container_Demo.computer EQ 50'
DEBUG [ReadStage:1] 2014-01-06 12:11:42,285 FileCacheService.java (line 70)
Evicting cold readers for
/home/mshuler/.ccm/test/node2/data/system/schema_columns/system-schema_columns-jb-7-Data.db
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11033316 compared to actual usage 0
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11164665 compared to actual usage 131349
DEBUG [ReadStage:1] 2014-01-06 12:11:42,286 FileCacheService.java (line 115)
Estimated memory usage is 11296014 compared to actual usage 262698
==> .ccm/test/node1/logs/system.log <==
DEBUG [Thrift:1] 2014-01-06 12:11:42,287 Tracing.java (line 159) request
complete
{code}
> Loose of secondary index entries if nodetool cleanup called before compaction
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-6517
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6517
> Project: Cassandra
> Issue Type: Bug
> Components: API
> Environment: Ubuntu 12.0.4 with 8+ GB RAM and 40GB hard disk for data
> directory.
> Reporter: Christoph Werres
> Assignee: Michael Shuler
>
> From time to time we had the feeling of not getting all results that should
> have been returned using secondary indexes. Now we tracked down some
> situations and found out, it happened:
> 1) To primary keys that were already deleted and have been re-created later on
> 2) After our nightly maintenance scripts were running
> We can reproduce now the following szenario:
> - create a row entry with an indexed column included
> - query it and use the secondary index criteria -> Success
> - delete it, query again -> entry gone as expected
> - re-create it with the same key, query it -> success again
> Now use in exactly that sequence
> nodetool cleanup
> nodetool flush
> nodetool compact
> When issuing the query now, we don't get the result using the index. The
> entry is indeed available in it's table when I just ask for the key. Below is
> the exact copy-paste output from CQL when I reproduced the problem with an
> example entry on on of our tables.
> mwerrch@mstc01401:/opt/cassandra$ current/bin/cqlsh Connected to
> 14-15-Cluster at localhost:9160.
> [cqlsh 4.1.0 | Cassandra 2.0.3 | CQL spec 3.1.1 | Thrift protocol 19.38.0]
> Use HELP for help.
> cqlsh> use mwerrch;
> cqlsh:mwerrch> desc tables;
> B4Container_Demo
> cqlsh:mwerrch> desc table "B4Container_Demo";
> CREATE TABLE "B4Container_Demo" (
> key uuid,
> archived boolean,
> bytes int,
> computer int,
> deleted boolean,
> description text,
> doarchive boolean,
> filename text,
> first boolean,
> frames int,
> ifversion int,
> imported boolean,
> jobid int,
> keepuntil bigint,
> nextchunk text,
> node int,
> recordingkey blob,
> recstart bigint,
> recstop bigint,
> simulationid bigint,
> systemstart bigint,
> systemstop bigint,
> tapelabel bigint,
> version blob,
> PRIMARY KEY (key)
> ) WITH COMPACT STORAGE AND
> bloom_filter_fp_chance=0.010000 AND
> caching='KEYS_ONLY' AND
> comment='demo' AND
> dclocal_read_repair_chance=0.000000 AND
> gc_grace_seconds=604800 AND
> index_interval=128 AND
> read_repair_chance=1.000000 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='NONE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> CREATE INDEX mwerrch_Demo_computer ON "B4Container_Demo" (computer);
> CREATE INDEX mwerrch_Demo_node ON "B4Container_Demo" (node);
> CREATE INDEX mwerrch_Demo_recordingkey ON "B4Container_Demo" (recordingkey);
> cqlsh:mwerrch> INSERT INTO "B4Container_Demo" (key,computer,node) VALUES
> (78c70562-1f98-3971-9c28-2c3d8e09c10f, 50, 50); cqlsh:mwerrch> select
> key,node,computer from "B4Container_Demo" where computer=50;
> key | node | computer
> --------------------------------------+------+----------
> 78c70562-1f98-3971-9c28-2c3d8e09c10f | 50 | 50
> (1 rows)
> cqlsh:mwerrch> DELETE FROM "B4Container_Demo" WHERE
> key=78c70562-1f98-3971-9c28-2c3d8e09c10f;
> cqlsh:mwerrch> select key,node,computer from "B4Container_Demo" where
> computer=50;
> (0 rows)
> cqlsh:mwerrch> INSERT INTO "B4Container_Demo" (key,computer,node) VALUES
> (78c70562-1f98-3971-9c28-2c3d8e09c10f, 50, 50); cqlsh:mwerrch> select
> key,node,computer from "B4Container_Demo" where computer=50;
> key | node | computer
> --------------------------------------+------+----------
> 78c70562-1f98-3971-9c28-2c3d8e09c10f | 50 | 50
> (1 rows)
> **********************************
> Now we execute (maybe from a different shell so we don't have to close this
> session) from /opt/cassandra/current/bin directory:
> ./nodetool cleanup
> ./nodetool flush
> ./nodetool compact
> Going back to our CQL session the result will no longer be available if
> queried via the index:
> *********************************
> cqlsh:mwerrch> select key,node,computer from "B4Container_Demo" where
> computer=50;
> (0 rows)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
