Large number of tombstones without delete or update

[Ralf Steppacher]

Hi all,

I started running “stress tests” against my application which writes event data 
to a local single-node Cassandra cluster (C* v2.2.4). After inserting a couple 
thousand rows (all for the same partition key) I am starting to get warnings in 
the C* log informing me about large numbers of tombstones:

WARN  [SharedPool-Worker-2] 2016-03-23 16:54:43,134 SliceQueryFilter.java:307 - 
Read 2410 live and 4820 tombstone cells in 
event_log_system.event_by_patient_timestamp for key: 100013866046895035 (see 
tombstone_warn_threshold). 5000 columns were requested, slices=[2040-06-02 
05\:57+0200:!-]
WARN  [SharedPool-Worker-2] 2016-03-23 16:55:01,991 SliceQueryFilter.java:307 - 
Read 2551 live and 5102 tombstone cells in 
event_log_system.event_by_patient_timestamp for key: 100013866046895035 (see 
tombstone_warn_threshold). 5000 columns were requested, slices=[2040-06-02 
05\:57+0200:!-]

Which leaves me stumped because I never delete or update any data during the 
test. It is all inserts, each insert creating a new row. What the stress test 
does is 

- drop the keyspace
- ramp up the application (which re-creates the schema)
- start inserting events for a single partition key
- periodically query for the number of rows for that partition key via the 
application

The query counting rows causes C* to log above warning. The application uses 
the Java driver (v.3.0.0). The query is:

select count(*) from event_by_patient_timestamp where patient_id = 
'100013866046895035' AND created_at > '2001-09-09T03:00:46+0200’;

The table contains a set of a UDT and a map<text,text>, if that should make any 
difference:

CREATE TABLE event_by_patient_timestamp (
    id                  uuid,
    version             text,
    created_at          timestamp,
   — [...]
    user         frozen <t_participant>,
   — [...]
    participants frozen <set<t_participant>>,
    — [...]
    extensions          map<text,text>,
    PRIMARY KEY (patient_id, created_at, id)
)
WITH CLUSTERING ORDER BY (created_at DESC);

All inserted events contain the exact same data, except for the event id and a 
creation timestamp.
Now, to add to my confusion: If I execute the exact same query via cqlsh I get 
no warnings at all!?


How come I end up with that large a number of tombstones? 
And how can I possibly avoid them? 
And why would they not be reported when queried via cqlsh?


Thanks,
Ralf