Re: Cassandra compaction stuck? Should I disable?

[PenguinWhispererThe .]

Hi Rob,

Thanks for your reply.

2015-11-09 23:17 GMT+01:00 Robert Coli :

> On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com> wrote:
>>
>> In Opscenter I see one of the nodes is orange. It seems like it's working
>> on compaction. I used nodetool compactionstats and whenever I did this the
>> Completed nad percentage stays the same (even with hours in between).
>>
> Are you the same person from IRC, or a second report today of compaction
> hanging in this way?
>
Same person ;) Just didn't had things to work with from the chat there. I
want to understand the issue more, see what I can tune or fix. I want to do
nodetool repair before upgrading to 2.1.11 but the compaction is blocking
it.

>
>
>
What version of Cassandra?
>
2.0.9

> I currently don't see cpu load from cassandra on that node. So it seems
>> stuck (somewhere mid 60%). Also some other nodes have compaction on the
>> same columnfamily. I don't see any progress.
>>
>>  WARN [RMI TCP Connection(554)-192.168.0.68] 2015-11-09 17:18:13,677 
>> ColumnFamilyStore.java (line 2101) Unable to cancel in-progress compactions 
>> for usage_record_ptd.  Probably there is an unusually large row in progress 
>> somewhere.  It is also possible that buggy code left some sstables 
>> compacting after it was done with them
>>
>>
>>- How can I assure that nothing is happening?
>>
>> Find the thread that is doing compaction and strace it. Generally it is
> one of the threads with a lower thread priority.
>

I have 141 threads. Not sure if that's normal.

This seems to be the one:
 61404 cassandr  24   4 8948m 4.3g 820m R 90.2 36.8 292:54.47 java

In the strace I see basically this part repeating (with once in a while the
"resource temporarily unavailable"):
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
getpriority(PRIO_PROCESS, 61404)= 16
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1233854, FUTEX_WAIT_PRIVATE, 494045, NULL) = -1 EAGAIN (Resource
temporarily unavailable)
futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1233854, FUTEX_WAIT_PRIVATE, 494047, NULL) = 0
futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
getpriority(PRIO_PROCESS, 61404)= 16
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x1233854, FUTEX_WAIT_PRIVATE, 494049, NULL) = 0
futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
getpriority(PRIO_PROCESS, 61404)= 16

But wait!
I also see this:
futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x1233854, FUTEX_WAIT_PRIVATE, 494055, NULL) = 0
futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

This doesn't seem to happen that often though.

>
> Compaction often appears hung when decompressing a very large row, but
> usually not for "hours".
>
>>
>>- Is it recommended to disable compaction from a certain data size?
>>(I believe 25GB on each node).
>>
>> It is almost never recommended to disable compaction.
>
>>
>>- Can I stop this compaction? nodetool stop compaction doesn't seem
>>to work.
>>
>> Killing the JVM ("the dungeon collapses!") would certainly stop it, but
> it'd likely just start again when you restart the node.
>
>>
>>- Is stopping the compaction dangerous?
>>
>>  Not if you're in a version that properly cleans up partial compactions,
> which is most of them.
>
>>
>>- Is killing the cassandra process dangerous while compacting(I did
>>nodetool drain on one node)?
>>
>> No. But probably nodetool drain couldn't actually stop the in-progress
> compaction either, FWIW.
>
>> This is output of nodetool compactionstats grepped for the keyspace that
>> seems stuck.
>>
>> Do you have gigantic rows in that keyspace? What does cfstats say about
> the largest row compaction has seen/do you have log messages about
> compacting large rows?
>

I don't know about the gigantic rows. How can I check?

I've checked the logs and found this:
 INFO [CompactionExecutor:67] 2015-11-10 02:34:19,077
CompactionController.java (line 192) Compacting large row
billing/usage_record_ptd:177727:2015-10-14 00\:00Z (243992466 bytes)
incrementally
So this is from 6 hours ago.

I also see a lot of messages like this:
INFO [OptionalTasks:1] 2015-11-10 06:36:06,395 MeteredFlusher.java (line
58) flushing high-traffic column family CFS(Key

Re: Cassandra compaction stuck? Should I disable?

[PenguinWhispererThe .]

Correction...
I was grepping on Segmentation on the strace and it happens a lot.

Do I need to run a scrub?

2015-11-10 9:30 GMT+01:00 PenguinWhispererThe . <
th3penguinwhispe...@gmail.com>:

> Hi Rob,
>
> Thanks for your reply.
>
> 2015-11-09 23:17 GMT+01:00 Robert Coli :
>
>> On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
>> th3penguinwhispe...@gmail.com> wrote:
>>>
>>> In Opscenter I see one of the nodes is orange. It seems like it's
>>> working on compaction. I used nodetool compactionstats and whenever I did
>>> this the Completed nad percentage stays the same (even with hours in
>>> between).
>>>
>> Are you the same person from IRC, or a second report today of compaction
>> hanging in this way?
>>
> Same person ;) Just didn't had things to work with from the chat there. I
> want to understand the issue more, see what I can tune or fix. I want to do
> nodetool repair before upgrading to 2.1.11 but the compaction is blocking
> it.
>
>>
>>
>>
> What version of Cassandra?
>>
> 2.0.9
>
>> I currently don't see cpu load from cassandra on that node. So it seems
>>> stuck (somewhere mid 60%). Also some other nodes have compaction on the
>>> same columnfamily. I don't see any progress.
>>>
>>>  WARN [RMI TCP Connection(554)-192.168.0.68] 2015-11-09 17:18:13,677 
>>> ColumnFamilyStore.java (line 2101) Unable to cancel in-progress compactions 
>>> for usage_record_ptd.  Probably there is an unusually large row in progress 
>>> somewhere.  It is also possible that buggy code left some sstables 
>>> compacting after it was done with them
>>>
>>>
>>>- How can I assure that nothing is happening?
>>>
>>> Find the thread that is doing compaction and strace it. Generally it is
>> one of the threads with a lower thread priority.
>>
>
> I have 141 threads. Not sure if that's normal.
>
> This seems to be the one:
>  61404 cassandr  24   4 8948m 4.3g 820m R 90.2 36.8 292:54.47 java
>
> In the strace I see basically this part repeating (with once in a while
> the "resource temporarily unavailable"):
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
> getpriority(PRIO_PROCESS, 61404)= 16
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494045, NULL) = -1 EAGAIN (Resource
> temporarily unavailable)
> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494047, NULL) = 0
> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
> getpriority(PRIO_PROCESS, 61404)= 16
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494049, NULL) = 0
> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
> getpriority(PRIO_PROCESS, 61404)= 16
>
> But wait!
> I also see this:
> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494055, NULL) = 0
> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
>
> This doesn't seem to happen that often though.
>
>>
>> Compaction often appears hung when decompressing a very large row, but
>> usually not for "hours".
>>
>>>
>>>- Is it recommended to disable compaction from a certain data size?
>>>(I believe 25GB on each node).
>>>
>>> It is almost never recommended to disable compaction.
>>
>>>
>>>- Can I stop this compaction? nodetool stop compaction doesn't seem
>>>to work.
>>>
>>> Killing the JVM ("the dungeon collapses!") would certainly stop it, but
>> it'd likely just start again when you restart the node.
>>
>>>
>>>- Is stopping the compaction dangerous?
>>>
>>>  Not if you're in a version that properly cleans up partial compactions,
>> which is most of them.
>>
>>>
>>>- Is killing the cassandra process dangerous while compacting(I did
>>>nodetool drain on one node)?
>>>
>>> No. But probably nodetool drain couldn't actually stop the in-progress
>> compaction either, FWIW.
>>
>>> This is output of nodetool compactionstats grepped for the keyspace that
>>> seems stuck.
>>>
>>> Do you have gigantic rows in that keyspace? What does cfstats say about
>> the largest row compaction has seen/do you have log messages about
>> compacting large rows?
>>
>
> I don't know about the gigantic rows. How can I check?
>
> I've checked the logs and found this:
>  INFO [Co

Re: How to organize a timeseries by device?

[Guillaume Charhon]

Thank you for all theses details. I will do a monthly bucket.

On Mon, Nov 9, 2015 at 7:58 PM, Kai Wang  wrote:

> bucket key is just like any column of the table, you can use any type as
> long as it's convenient for you to write the query.
>
> But I don't think you should use 5 minute as your bucket key since you
> only have 1 event every 5 minute. 5-minute bucket seems too small. The
> bucket key we mentioned is for you to break the (device_id, timestamp)
> partitions into ones with size between ~1MB to ~10MB.
>
> On Mon, Nov 9, 2015 at 11:50 AM, Guillaume Charhon <
> guilla...@databerries.com> wrote:
>
>> Is it usually recommended to use the bucket key (usually an 5 minutes
>> period in my case) for the table of the events_by_time using a timestamp or
>> a string?
>>
>> On Mon, Nov 9, 2015 at 5:05 PM, Kai Wang  wrote:
>>
>>> it depends on the size of each event. You want to bound each partition
>>> under ~10MB. In system.log look for entry like:
>>>
>>> WARN  [CompactionExecutor:39] 2015-11-07 17:32:00,019
>>> SSTableWriter.java:240 - Compacting large partition
>>> :9f80ce31-b7e7-40c7-b642-f5d03fc320aa (13443863224 bytes)
>>>
>>> This is the warning sign that you have large partitions. The threshold
>>> is defined by compaction_large_partition_warning_threshold_mb in
>>> cassandra.yaml. The default is 100MB.
>>>
>>> You can also use nodetool cfstats to check partition size.
>>>
>>> On Mon, Nov 9, 2015 at 10:53 AM, Guillaume Charhon <
>>> guilla...@databerries.com> wrote:
>>>
 For the first table: (device_id, timestamp), should I add a bucket
 even if I know I might have millions of events per device but never
 billions?

 On Mon, Nov 9, 2015 at 4:37 PM, Jack Krupansky <
 jack.krupan...@gmail.com> wrote:

> Cassandra is good at two kinds of queries: 1) access a specific row by
> a specific key, and 2) Access a slice or consecutive sequence of rows
> within a given partition.
>
> It is recommended to avoid ALLOW FILTERING. If it happens to work well
> for you, great, go for it, but if it doesn't then simply don't do it. Best
> to redesign your data model to play to Cassandra's strengths.
>
> If you bucket the time-based table, do a separate query for each time
> bucket.
>
> -- Jack Krupansky
>
> On Mon, Nov 9, 2015 at 10:16 AM, Guillaume Charhon <
> guilla...@databerries.com> wrote:
>
>> Kai, Jack,
>>
>> On 1., should the bucket be a STRING with a date format or do I have
>> a better option ? For (device_id, bucket, timestamp), did you mean
>> ((device_id, bucket), timestamp) ?
>>
>> On 2., what are the risks of timeout ? I currently have this warning:
>> "Cannot execute this query as it might involve data filtering and thus 
>> may
>> have unpredictable performance. If you want to execute this query despite
>> the performance unpredictability, use ALLOW FILTERING".
>>
>> On Mon, Nov 9, 2015 at 3:02 PM, Kai Wang  wrote:
>>
>>> 1. Don't make your partition unbound. It's tempting to just use
>>> (device_id, timestamp). But soon or later you will have problem when 
>>> time
>>> goes by. You can keep the partition bound by using (device_id, bucket,
>>> timestamp). Use hour, day, month or even year like Jack mentioned 
>>> depending
>>> on the size of data.
>>>
>>> 2. As to your specific query, for a given partition and a time
>>> range, C* doesn't need to load the whole partition then filter. It only
>>> retrieves the slice within the time range from disk because the data is
>>> clustered by timestamp.
>>>
>>> On Mon, Nov 9, 2015 at 8:13 AM, Jack Krupansky <
>>> jack.krupan...@gmail.com> wrote:
>>>
 The general rule in Cassandra data modeling is to look at all of
 your queries first and then to declare a table for each query, even if 
 that
 means storing multiple copies of the data. So, create a second table 
 with
 bucketed time as the partition key (hour, 15 minutes, or whatever time
 interval makes sense to give 1 to 10 megabytes per partition) and time 
 and
 device as the clustering keys.

 Or, consider DSE SEarch  and then you can do whatever ad hoc
 queries you want using Solr. Or Stratio or TupleJump Stargate for an 
 open
 source Lucene plugin.

 -- Jack Krupansky

 On Mon, Nov 9, 2015 at 8:05 AM, Guillaume Charhon <
 guilla...@databerries.com> wrote:

> Hello,
>
> We are currently storing geolocation events (about 1 per 5
> minutes) for each device we track. We currently have 2 TB of data. I 
> would
> like to store the device_id, the timestamp of the event, latitude and
> longitude. I though about using the device_id as the partition key and
> timestamp as the clus

Nested Collections in Cassandra

[Neha Dave]

How can we achieve Nested Collections in cassandra.

My requirement :
metadata map> ... Is it possible?

Eg. 'mime-type' : 'MIME'
  'Security'  : {'SOX','FOX'}

Query will be Give me all the ID's where 'Security'  : {'SOX'} OR contains
'SOX'

Is it Possible?
Can I use UDT to do it?

Eg CQL :

CREATE TYPE security (
  number text,
  tags set
  );


CREATE TYPE listdata (
  values set
  );

  CREATE TABLE test_path (
  path_id text PRIMARY KEY,
  metadata map>
  );

INSERT INTO test_path (path_id, metadata ) VALUES ( '2', { 'mime-type':
{values : {'Mime'}}
{'applicable-security-policy' : {'SOX','FOX'}} });


Query (which does not work) can be :
SELECT * from test_path where metadata CONTAINS {values: {'FOX'}, 'SOX'}} ;
OR
SELECT * from test_path where metadata CONTAINS {values: {'FOX'};


Thanks
Regards
Neha

cassandra 3.0 and datastax java driver 3.0.0 beta1: unresolved user type DoubleType

[Vova Shelgunov]

Hi All,

When I am trying to insert an object of the attached "table_1.png" class
I've got the error:

com.datastax.driver.core.exceptions.UnresolvedUserTypeException: Cannot
resolve user type keyspace1."org.apache.cassandra.db.marshal.DoubleType"

Could you please suggest the solution?
Thank you

Re: Cassandra compaction stuck? Should I disable?

[PenguinWhispererThe .]

Still stuck with this. However I enabled GC logging. This shows the
following:

[root@myhost cassandra]# tail -f gc-1447180680.log
2015-11-10T18:41:45.516+: 225.428: [GC 2721842K->2066508K(6209536K),
0.0199040 secs]
2015-11-10T18:41:45.977+: 225.889: [GC 2721868K->2066511K(6209536K),
0.0221910 secs]
2015-11-10T18:41:46.437+: 226.349: [GC 2721871K->2066524K(6209536K),
0.0222140 secs]
2015-11-10T18:41:46.897+: 226.809: [GC 2721884K->2066539K(6209536K),
0.0224140 secs]
2015-11-10T18:41:47.359+: 227.271: [GC 2721899K->2066538K(6209536K),
0.0302520 secs]
2015-11-10T18:41:47.821+: 227.733: [GC 2721898K->2066557K(6209536K),
0.0280530 secs]
2015-11-10T18:41:48.293+: 228.205: [GC 2721917K->2066571K(6209536K),
0.0218000 secs]
2015-11-10T18:41:48.790+: 228.702: [GC 2721931K->2066780K(6209536K),
0.0292470 secs]
2015-11-10T18:41:49.290+: 229.202: [GC 2722140K->2066843K(6209536K),
0.0288740 secs]
2015-11-10T18:41:49.756+: 229.668: [GC 2722203K->2066818K(6209536K),
0.0283380 secs]
2015-11-10T18:41:50.249+: 230.161: [GC 2722178K->2067158K(6209536K),
0.0218690 secs]
2015-11-10T18:41:50.713+: 230.625: [GC 2722518K->2067236K(6209536K),
0.0278810 secs]

This is a VM with 12GB of RAM. Highered the HEAP_SIZE to 6GB and
HEAP_NEWSIZE to 800MB.

Still the same result.

This looks very similar to following issue:
http://mail-archives.apache.org/mod_mbox/cassandra-user/201411.mbox/%3CCAJ=3xgRLsvpnZe0uXEYjG94rKhfXeU+jBR=q3a-_c3rsdd5...@mail.gmail.com%3E

Is the only possibility to upgrade memory? I mean, I can't believe it's
just loading all it's data in memory. That would require to keep scaling up
the node to keep it work?


2015-11-10 9:36 GMT+01:00 PenguinWhispererThe . <
th3penguinwhispe...@gmail.com>:

> Correction...
> I was grepping on Segmentation on the strace and it happens a lot.
>
> Do I need to run a scrub?
>
> 2015-11-10 9:30 GMT+01:00 PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com>:
>
>> Hi Rob,
>>
>> Thanks for your reply.
>>
>> 2015-11-09 23:17 GMT+01:00 Robert Coli :
>>
>>> On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
>>> th3penguinwhispe...@gmail.com> wrote:

 In Opscenter I see one of the nodes is orange. It seems like it's
 working on compaction. I used nodetool compactionstats and whenever I did
 this the Completed nad percentage stays the same (even with hours in
 between).

>>> Are you the same person from IRC, or a second report today of compaction
>>> hanging in this way?
>>>
>> Same person ;) Just didn't had things to work with from the chat there. I
>> want to understand the issue more, see what I can tune or fix. I want to do
>> nodetool repair before upgrading to 2.1.11 but the compaction is blocking
>> it.
>>
>>>
>>>
>>>
>> What version of Cassandra?
>>>
>> 2.0.9
>>
>>> I currently don't see cpu load from cassandra on that node. So it seems
 stuck (somewhere mid 60%). Also some other nodes have compaction on the
 same columnfamily. I don't see any progress.

  WARN [RMI TCP Connection(554)-192.168.0.68] 2015-11-09 17:18:13,677 
 ColumnFamilyStore.java (line 2101) Unable to cancel in-progress 
 compactions for usage_record_ptd.  Probably there is an unusually large 
 row in progress somewhere.  It is also possible that buggy code left some 
 sstables compacting after it was done with them


- How can I assure that nothing is happening?

 Find the thread that is doing compaction and strace it. Generally it is
>>> one of the threads with a lower thread priority.
>>>
>>
>> I have 141 threads. Not sure if that's normal.
>>
>> This seems to be the one:
>>  61404 cassandr  24   4 8948m 4.3g 820m R 90.2 36.8 292:54.47 java
>>
>> In the strace I see basically this part repeating (with once in a while
>> the "resource temporarily unavailable"):
>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
>> getpriority(PRIO_PROCESS, 61404)= 16
>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494045, NULL) = -1 EAGAIN (Resource
>> temporarily unavailable)
>> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494047, NULL) = 0
>> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
>> getpriority(PRIO_PROCESS, 61404)= 16
>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> f

Re: Cassandra compaction stuck? Should I disable?

[PenguinWhispererThe .]

I also have the following memory usage:
[root@US-BILLINGDSX4 cassandra]# free -m
 total   used   free sharedbuffers cached
Mem: 12024   9455   2569  0110   2163
-/+ buffers/cache:   7180   4844
Swap: 2047  0   2047

Still a lot free and a lot of free buffers/cache.

2015-11-10 19:45 GMT+01:00 PenguinWhispererThe . <
th3penguinwhispe...@gmail.com>:

> Still stuck with this. However I enabled GC logging. This shows the
> following:
>
> [root@myhost cassandra]# tail -f gc-1447180680.log
> 2015-11-10T18:41:45.516+: 225.428: [GC 2721842K->2066508K(6209536K),
> 0.0199040 secs]
> 2015-11-10T18:41:45.977+: 225.889: [GC 2721868K->2066511K(6209536K),
> 0.0221910 secs]
> 2015-11-10T18:41:46.437+: 226.349: [GC 2721871K->2066524K(6209536K),
> 0.0222140 secs]
> 2015-11-10T18:41:46.897+: 226.809: [GC 2721884K->2066539K(6209536K),
> 0.0224140 secs]
> 2015-11-10T18:41:47.359+: 227.271: [GC 2721899K->2066538K(6209536K),
> 0.0302520 secs]
> 2015-11-10T18:41:47.821+: 227.733: [GC 2721898K->2066557K(6209536K),
> 0.0280530 secs]
> 2015-11-10T18:41:48.293+: 228.205: [GC 2721917K->2066571K(6209536K),
> 0.0218000 secs]
> 2015-11-10T18:41:48.790+: 228.702: [GC 2721931K->2066780K(6209536K),
> 0.0292470 secs]
> 2015-11-10T18:41:49.290+: 229.202: [GC 2722140K->2066843K(6209536K),
> 0.0288740 secs]
> 2015-11-10T18:41:49.756+: 229.668: [GC 2722203K->2066818K(6209536K),
> 0.0283380 secs]
> 2015-11-10T18:41:50.249+: 230.161: [GC 2722178K->2067158K(6209536K),
> 0.0218690 secs]
> 2015-11-10T18:41:50.713+: 230.625: [GC 2722518K->2067236K(6209536K),
> 0.0278810 secs]
>
> This is a VM with 12GB of RAM. Highered the HEAP_SIZE to 6GB and
> HEAP_NEWSIZE to 800MB.
>
> Still the same result.
>
> This looks very similar to following issue:
>
> http://mail-archives.apache.org/mod_mbox/cassandra-user/201411.mbox/%3CCAJ=3xgRLsvpnZe0uXEYjG94rKhfXeU+jBR=q3a-_c3rsdd5...@mail.gmail.com%3E
>
> Is the only possibility to upgrade memory? I mean, I can't believe it's
> just loading all it's data in memory. That would require to keep scaling up
> the node to keep it work?
>
>
> 2015-11-10 9:36 GMT+01:00 PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com>:
>
>> Correction...
>> I was grepping on Segmentation on the strace and it happens a lot.
>>
>> Do I need to run a scrub?
>>
>> 2015-11-10 9:30 GMT+01:00 PenguinWhispererThe . <
>> th3penguinwhispe...@gmail.com>:
>>
>>> Hi Rob,
>>>
>>> Thanks for your reply.
>>>
>>> 2015-11-09 23:17 GMT+01:00 Robert Coli :
>>>
 On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
 th3penguinwhispe...@gmail.com> wrote:
>
> In Opscenter I see one of the nodes is orange. It seems like it's
> working on compaction. I used nodetool compactionstats and whenever I did
> this the Completed nad percentage stays the same (even with hours in
> between).
>
 Are you the same person from IRC, or a second report today of
 compaction hanging in this way?

>>> Same person ;) Just didn't had things to work with from the chat there.
>>> I want to understand the issue more, see what I can tune or fix. I want to
>>> do nodetool repair before upgrading to 2.1.11 but the compaction is
>>> blocking it.
>>>



>>> What version of Cassandra?

>>> 2.0.9
>>>
 I currently don't see cpu load from cassandra on that node. So it seems
> stuck (somewhere mid 60%). Also some other nodes have compaction on the
> same columnfamily. I don't see any progress.
>
>  WARN [RMI TCP Connection(554)-192.168.0.68] 2015-11-09 17:18:13,677 
> ColumnFamilyStore.java (line 2101) Unable to cancel in-progress 
> compactions for usage_record_ptd.  Probably there is an unusually large 
> row in progress somewhere.  It is also possible that buggy code left some 
> sstables compacting after it was done with them
>
>
>- How can I assure that nothing is happening?
>
> Find the thread that is doing compaction and strace it. Generally it
 is one of the threads with a lower thread priority.

>>>
>>> I have 141 threads. Not sure if that's normal.
>>>
>>> This seems to be the one:
>>>  61404 cassandr  24   4 8948m 4.3g 820m R 90.2 36.8 292:54.47 java
>>>
>>> In the strace I see basically this part repeating (with once in a while
>>> the "resource temporarily unavailable"):
>>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 1
>>> getpriority(PRIO_PROCESS, 61404)= 16
>>> futex(0x7f5c64145e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f5c64145e50,
>>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>>> futex(0x7f5c64145e28, FUTEX_WAKE_PRIVATE, 1) = 0
>>> futex(0x1233854, FUTEX_WAIT_PRIVATE, 494045, NULL) = -1 EAGAIN (Resource
>>> temporarily unavailable)
>>> futex(0x1233828, FUTEX_WAKE_PRIVATE, 1) = 0
>>> fut

Re: Cassandra compaction stuck? Should I disable?

[Sebastian Estevez]

Turn off Swap.

http://docs.datastax.com/en/cassandra/2.1/cassandra/install/installRecommendSettings.html?scroll=reference_ds_sxl_gf3_2k__disable-swap


All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Tue, Nov 10, 2015 at 1:48 PM, PenguinWhispererThe . <
th3penguinwhispe...@gmail.com> wrote:

> I also have the following memory usage:
> [root@US-BILLINGDSX4 cassandra]# free -m
>  total   used   free sharedbuffers cached
> Mem: 12024   9455   2569  0110   2163
> -/+ buffers/cache:   7180   4844
> Swap: 2047  0   2047
>
> Still a lot free and a lot of free buffers/cache.
>
> 2015-11-10 19:45 GMT+01:00 PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com>:
>
>> Still stuck with this. However I enabled GC logging. This shows the
>> following:
>>
>> [root@myhost cassandra]# tail -f gc-1447180680.log
>> 2015-11-10T18:41:45.516+: 225.428: [GC 2721842K->2066508K(6209536K),
>> 0.0199040 secs]
>> 2015-11-10T18:41:45.977+: 225.889: [GC 2721868K->2066511K(6209536K),
>> 0.0221910 secs]
>> 2015-11-10T18:41:46.437+: 226.349: [GC 2721871K->2066524K(6209536K),
>> 0.0222140 secs]
>> 2015-11-10T18:41:46.897+: 226.809: [GC 2721884K->2066539K(6209536K),
>> 0.0224140 secs]
>> 2015-11-10T18:41:47.359+: 227.271: [GC 2721899K->2066538K(6209536K),
>> 0.0302520 secs]
>> 2015-11-10T18:41:47.821+: 227.733: [GC 2721898K->2066557K(6209536K),
>> 0.0280530 secs]
>> 2015-11-10T18:41:48.293+: 228.205: [GC 2721917K->2066571K(6209536K),
>> 0.0218000 secs]
>> 2015-11-10T18:41:48.790+: 228.702: [GC 2721931K->2066780K(6209536K),
>> 0.0292470 secs]
>> 2015-11-10T18:41:49.290+: 229.202: [GC 2722140K->2066843K(6209536K),
>> 0.0288740 secs]
>> 2015-11-10T18:41:49.756+: 229.668: [GC 2722203K->2066818K(6209536K),
>> 0.0283380 secs]
>> 2015-11-10T18:41:50.249+: 230.161: [GC 2722178K->2067158K(6209536K),
>> 0.0218690 secs]
>> 2015-11-10T18:41:50.713+: 230.625: [GC 2722518K->2067236K(6209536K),
>> 0.0278810 secs]
>>
>> This is a VM with 12GB of RAM. Highered the HEAP_SIZE to 6GB and
>> HEAP_NEWSIZE to 800MB.
>>
>> Still the same result.
>>
>> This looks very similar to following issue:
>>
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201411.mbox/%3CCAJ=3xgRLsvpnZe0uXEYjG94rKhfXeU+jBR=q3a-_c3rsdd5...@mail.gmail.com%3E
>>
>> Is the only possibility to upgrade memory? I mean, I can't believe it's
>> just loading all it's data in memory. That would require to keep scaling up
>> the node to keep it work?
>>
>>
>> 2015-11-10 9:36 GMT+01:00 PenguinWhispererThe . <
>> th3penguinwhispe...@gmail.com>:
>>
>>> Correction...
>>> I was grepping on Segmentation on the strace and it happens a lot.
>>>
>>> Do I need to run a scrub?
>>>
>>> 2015-11-10 9:30 GMT+01:00 PenguinWhispererThe . <
>>> th3penguinwhispe...@gmail.com>:
>>>
 Hi Rob,

 Thanks for your reply.

 2015-11-09 23:17 GMT+01:00 Robert Coli :

> On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com> wrote:
>>
>> In Opscenter I see one of the nodes is orange. It seems like it's
>> working on compaction. I used nodetool compactionstats and whenever I did
>> this the Completed nad percentage stays the same (even with hours in
>> between).
>>
> Are you the same person from IRC, or a second report today of
> compaction hanging in this way?
>
 Same person ;) Just didn't had things to work with from the chat there.
 I want to understand the issue more, see what I can tune or fix. I want to
 do nodetool repair before upgrading to 2.1.11 but the compaction is
 blocking it.

>
>
>
 What version of Cassandra?
>
 2.0.9

> I currently don't see cpu load from cassandra on that node. So it
>> seems stuck (somewhere mid 60%). Also some other nodes have compaction on
>> the same columnfamily. I don't see any progress.
>>
>>  WARN [RMI TCP Connection(554)-192.168.0.68] 2015-11-09 17:18:13,677 
>> ColumnFamilyStore.java (li

Re: Cassandra compaction stuck? Should I disable?

[PenguinWhispererThe .]

Hi Sebastian,

Thanks for your response.

No swap is used. No offense, I just don't see a reason why having swap
would be the issue here. I put swapiness on 1. I also have jna installed.
That should prevent java being swapped out as wel AFAIK.


2015-11-10 19:50 GMT+01:00 Sebastian Estevez :

> Turn off Swap.
>
>
> http://docs.datastax.com/en/cassandra/2.1/cassandra/install/installRecommendSettings.html?scroll=reference_ds_sxl_gf3_2k__disable-swap
>
>
> All the best,
>
>
> [image: datastax_logo.png] 
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
>
> [image: linkedin.png]  [image:
> facebook.png]  [image: twitter.png]
>  [image: g+.png]
> 
> 
> 
>
>
> 
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Tue, Nov 10, 2015 at 1:48 PM, PenguinWhispererThe . <
> th3penguinwhispe...@gmail.com> wrote:
>
>> I also have the following memory usage:
>> [root@US-BILLINGDSX4 cassandra]# free -m
>>  total   used   free sharedbuffers cached
>> Mem: 12024   9455   2569  0110   2163
>> -/+ buffers/cache:   7180   4844
>> Swap: 2047  0   2047
>>
>> Still a lot free and a lot of free buffers/cache.
>>
>> 2015-11-10 19:45 GMT+01:00 PenguinWhispererThe . <
>> th3penguinwhispe...@gmail.com>:
>>
>>> Still stuck with this. However I enabled GC logging. This shows the
>>> following:
>>>
>>> [root@myhost cassandra]# tail -f gc-1447180680.log
>>> 2015-11-10T18:41:45.516+: 225.428: [GC 2721842K->2066508K(6209536K),
>>> 0.0199040 secs]
>>> 2015-11-10T18:41:45.977+: 225.889: [GC 2721868K->2066511K(6209536K),
>>> 0.0221910 secs]
>>> 2015-11-10T18:41:46.437+: 226.349: [GC 2721871K->2066524K(6209536K),
>>> 0.0222140 secs]
>>> 2015-11-10T18:41:46.897+: 226.809: [GC 2721884K->2066539K(6209536K),
>>> 0.0224140 secs]
>>> 2015-11-10T18:41:47.359+: 227.271: [GC 2721899K->2066538K(6209536K),
>>> 0.0302520 secs]
>>> 2015-11-10T18:41:47.821+: 227.733: [GC 2721898K->2066557K(6209536K),
>>> 0.0280530 secs]
>>> 2015-11-10T18:41:48.293+: 228.205: [GC 2721917K->2066571K(6209536K),
>>> 0.0218000 secs]
>>> 2015-11-10T18:41:48.790+: 228.702: [GC 2721931K->2066780K(6209536K),
>>> 0.0292470 secs]
>>> 2015-11-10T18:41:49.290+: 229.202: [GC 2722140K->2066843K(6209536K),
>>> 0.0288740 secs]
>>> 2015-11-10T18:41:49.756+: 229.668: [GC 2722203K->2066818K(6209536K),
>>> 0.0283380 secs]
>>> 2015-11-10T18:41:50.249+: 230.161: [GC 2722178K->2067158K(6209536K),
>>> 0.0218690 secs]
>>> 2015-11-10T18:41:50.713+: 230.625: [GC 2722518K->2067236K(6209536K),
>>> 0.0278810 secs]
>>>
>>> This is a VM with 12GB of RAM. Highered the HEAP_SIZE to 6GB and
>>> HEAP_NEWSIZE to 800MB.
>>>
>>> Still the same result.
>>>
>>> This looks very similar to following issue:
>>>
>>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201411.mbox/%3CCAJ=3xgRLsvpnZe0uXEYjG94rKhfXeU+jBR=q3a-_c3rsdd5...@mail.gmail.com%3E
>>>
>>> Is the only possibility to upgrade memory? I mean, I can't believe it's
>>> just loading all it's data in memory. That would require to keep scaling up
>>> the node to keep it work?
>>>
>>>
>>> 2015-11-10 9:36 GMT+01:00 PenguinWhispererThe . <
>>> th3penguinwhispe...@gmail.com>:
>>>
 Correction...
 I was grepping on Segmentation on the strace and it happens a lot.

 Do I need to run a scrub?

 2015-11-10 9:30 GMT+01:00 PenguinWhispererThe . <
 th3penguinwhispe...@gmail.com>:

> Hi Rob,
>
> Thanks for your reply.
>
> 2015-11-09 23:17 GMT+01:00 Robert Coli :
>
>> On Mon, Nov 9, 2015 at 1:29 PM, PenguinWhispererThe . <
>> th3penguinwhispe...@gmail.com> wrote:
>>>
>>> In Opscenter I see one of the nodes is orange. It seems like it's
>>> working on compaction. I used nodetool compactionstats and whenever I 
>>> did
>>> this the Completed nad percentage stays the same (even with hours in
>>> between).
>>>
>> Are you the same person from IRC, or a second report today of
>> compaction hanging in this way?
>>
> Same person ;) Just didn't had things to work with from the chat
> there. I want to understand the issue more, see what I can tune or fix. I
> want to do nodetool repair before upgrading to 2.1.

UnknownColumnFamily exception / schema inconsistencies

[Maciek Sakrejda]

Hello,

I've been having some strange issues with one of our test clusters
(4-day-old, 3-node, 2.1.10 cluster on AWS). I saw a number of messages like
the following:

[] 10 Nov 20:21:00.406 * pri=WARN  t=MessagingService-Incoming-/
192.168.168.202 at=IncomingTcpConnection.run UnknownColumnFamilyException
reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=3ecce750-84d3-11e5-bdd9-dd7717dcdbd5

A colleague suggested I run repair, but that failed with:

[2015-11-10 20:06:54,329] Nothing to repair for keyspace 'eventPipesState'
[2015-11-10 20:06:54,348] Starting repair command #1, repairing 768 ranges
for keyspace dbs8okvd7jcurj (parallelism=SEQUENTIAL, full=true)
[2015-11-10 20:06:55,599] Repair command #1 finished
[2015-11-10 20:06:55,610] Starting repair command #2, repairing 487 ranges
for keyspace context (parallelism=SEQUENTIAL, full=true)
[2015-11-10 20:11:21,213] Lost notification. You should check server log
for repair status of keyspace context
[2015-11-10 20:11:21,288] Lost notification. You should check server log
for repair status of keyspace context
Exception occurred during clean-up.
java.lang.reflect.UndeclaredThrowableException
error: JMX connection closed. You should check server log for repair status
of keyspace context(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: JMX connection closed. You should check server log for
repair status of keyspace context(Subsequent keyspaces are not going to be
repaired).

I searched for other cases of similar issues, and found some posts (e.g.,
http://stackoverflow.com/questions/22783577/org-apache-cassandra-db-unknowncolumnfamilyexception-couldnt-find-cfid
), but nothing that seemed directly relevant. Still, I tried `nodetool
describecluster` and all the nodes showed up as being on the same schema
version.

The server log did not include any more info. I asked about this on IRC and
got the suggestion to run `nodetool resetlocalschema`. I tried running
that, and it completed (and `nodetool describecluster` now shows this node
as having a different schema version from the other two nodes) but now I
still get the original error in the server logs but also

[] 10 Nov 22:51:10.466 * pri=ERROR t=Thrift:12
at=CustomTThreadPoolServer.run Error occurred during processing of message.
java.lang.IllegalArgumentException: Unknown keyspace/cf pair
(system_auth.credentials)

Further `nodetool repair`s on the same node do complete, but only seem to
process the `system` keyspace (and don't do anything with it):

[2015-11-10 22:38:07,415] Nothing to repair for keyspace 'system'

I also tried running `nodetool repair` from another node in the cluster,
but that just seems to hang:

[2015-11-10 22:53:11,830] Starting repair command #7, repairing 768 ranges
for keyspace dbs8okvd7jcurj (parallelism=SEQUENTIAL, full=true)
[2015-11-10 22:53:12,943] Repair command #7 finished
[2015-11-10 22:53:12,958] Starting repair command #8, repairing 534 ranges
for keyspace context (parallelism=SEQUENTIAL, full=true)

How can I restore this cluster? And ideally, how can I figure out what went
wrong here in the first place?

Re: UnknownColumnFamily exception / schema inconsistencies

[Maciek Sakrejda]

Oh and for what it's worth, I've also looked through the logs for this
node, and the oldest error in the logs seems to be:

[] 06 Nov 22:10:53.260 * pri=ERROR t=Thrift:16
at=CustomTThreadPoolServer.run Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ConfigurationException: Column family ID
mismatch (found 3ed23e80-84d3-11e5-83b3-0fc0205655f5; expected
3ecce750-84d3-11e5-bdd9-dd7717dcdbd5)

Then the logs show a compaction, and then the UnknownColumnFamilyException
starts occuring.
​

Re: UnknownColumnFamily exception / schema inconsistencies

[Sebastian Estevez]

#1 The cause of this problem is a CREATE TABLE statement collision. Do
not generate tables
dynamically from multiple clients, even with IF NOT EXISTS. First
thing you need to do is
fix your code so that this does not happen. Just create your tables
manually from cqlsh allowing
time for the schema to settle.

#2 Here's the fix:

1) Change your code to not automatically re-create tables (even with
IF NOT EXISTS).

2) Run a rolling restart to ensure schema matches across nodes. Run
nodetool describecluster

around your cluster. Check that there is only one schema version.

ON EACH NODE:

3) Check your filesystem and see if you have two directories for the table in

question in the data directory.

If THERE ARE TWO OR MORE DIRECTORIES:

4)Identify from schema_column_families which cf ID is the "new" one
(currently in use).

cqlsh -e "select * from system.schema_column_families"|grep 

5) Move the data from the "old" one to the "new" one and remove the
old directory.

6) If there are multiple "old" ones repeat 5 for every "old" directory.

7) run nodetool refresh

IF THERE IS ONLY ONE DIRECTORY:

No further action is needed.


All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]







DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Tue, Nov 10, 2015 at 6:09 PM, Maciek Sakrejda  wrote:

> Oh and for what it's worth, I've also looked through the logs for this
> node, and the oldest error in the logs seems to be:
>
> [] 06 Nov 22:10:53.260 * pri=ERROR t=Thrift:16
> at=CustomTThreadPoolServer.run Error occurred during processing of message.
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found 3ed23e80-84d3-11e5-83b3-0fc0205655f5; expected
> 3ecce750-84d3-11e5-bdd9-dd7717dcdbd5)
>
> Then the logs show a compaction, and then the UnknownColumnFamilyException
> starts occuring.
> ​
>

Multi-column slice restrictions not respected by the returned result

[Yuri Shkuro]

According to this blog:
http://www.datastax.com/dev/blog/a-deep-look-to-the-cql-where-clause

I should be able to do multi-column restrictions on clustering columns, as
in the blog example: WHERE (server, time) >= (‘196.8.0.0’, 12:00) AND
(server, time) <= (‘196.8.255.255’, 14:00)

However, I am getting data returned from such query that does not match the
restrictions. Tried on Cassandra 2.17 and 2.2.3. Here's an example:

CREATE TABLE IF NOT EXISTS dur (
s  text,
nd bigint,
ts bigint,
tidbigint,
PRIMARY KEY (s, nd, ts)
) WITH CLUSTERING ORDER BY (nd ASC, ts DESC);

insert INTO dur (s, nd, ts, tid) values ('x', 1, 10, 99);
insert INTO dur (s, nd, ts, tid) values ('x', 2, 11, 98) ;
insert INTO dur (s, nd, ts, tid) values ('x', 3, 10, 97) ;
insert INTO dur (s, nd, ts, tid) values ('x', 1, 11, 96) ;
insert INTO dur (s, nd, ts, tid) values ('x', 1, 12, 95) ;
insert INTO dur (s, nd, ts, tid) values ('x', 2, 10, 94) ;
insert INTO dur (s, nd, ts, tid) values ('x', 2, 12, 93) ;
insert INTO dur (s, nd, ts, tid) values ('x', 3, 11, 92) ;
insert INTO dur (s, nd, ts, tid) values ('x', 3, 12, 91) ;

select * from dur where s='x' and (nd,ts) > (2, 11);

 s | nd | ts | tid
---+++-
 x |  2 | 10 |  94
 x |  3 | 12 |  91
 x |  3 | 11 |  92
 x |  3 | 10 |  97
(4 rows)

The first row in the result does not satisfy the restriction (nd,ts) > (2,
11). Am I doing something incorrectly?

Thanks,
--Yuri