FW: Cassandra trigger to send notifications

[Oren Yekutieli]

Hello,

In a Cassandra cluster I want to push a notification to rabbitmq whenever a 
change (insert/update/delete) was made to some Cassandra tables, with the 
following requirements:
The notifications should:

1.  Be ordered in the same order the changes were stored.

2.  Be sent only if the change was (or going to be) committed successfully.
And in case of insert/update:

3.  Include all the row values.

4.  Indicate what values were changed.

I'm trying to figure out if Cassandra triggers are the correct way to go.
According to 
http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
 one of the potential use cases for triggers is:
*>> implementing alerts/notifications
I also played a bit with the example given here: 
http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
 and from what I read and learned so far I assume that:
Regarding #1, I doubt if the notification order can be kept since the trigger 
is invoked on the coordinator node while there are multiple clients that can 
connect to the various cluster nodes (coordinators).
Regarding #2, I'm not sure that notification will be sent only for changes that 
are going to be committed successfully since the trigger is invoked before the 
change is being applied.
Regarding #3, I seen that this requirement is fulfilled.
Regarding #4, I didn't find a way to figure out what values are actually 
changed, without re-reading the current (old) values from the table which of 
course impose a significant performance penalty.

Are my assumptions correct?
If yes then triggers are not the right approach. What other ways can be taken?

Regards,
Oren

Re: FW: Cassandra trigger to send notifications

[Matija Gobec]

Hi Oren,

I've spent a reasonable time working out triggers and I would say that your
best bet is doing this in the app.
Just publish a rabbitmq message from the app when you execute a statement.
If your goal is to have an audit then try batch writing data to the tables
and delta to their audit counterparts.

Matija

On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli 
wrote:

> Hello,
>
>
>
> In a Cassandra cluster I want to push a notification to rabbitmq whenever
> a change (insert/update/delete) was made to some Cassandra tables, with the
> following requirements:
>
> The notifications should:
>
> 1.  Be ordered in the same order the changes were stored.
>
> 2.  Be sent only if the change was (or going to be) committed
> successfully.
>
> And in case of insert/update:
>
> 3.  Include all the row values.
>
> 4.  Indicate what values were changed.
>
>
>
> I’m trying to figure out if Cassandra triggers are the correct way to go.
>
> According to http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-
> 0-prototype-triggers-support, one of the potential use cases for triggers
> is:
>
> ·>> implementing alerts/notifications
>
> I also played a bit with the example given here: http://stackoverflow.com/
> questions/35161747/example-of-creating-triggers-in-
> cassandra-and-does-this-support-only-for-java and from what I read and
> learned so far I assume that:
>
> Regarding #1, I doubt if the notification order can be kept since the
> trigger is invoked on the coordinator node while there are multiple clients
> that can connect to the various cluster nodes (coordinators).
>
> Regarding #2, I’m not sure that notification will be sent only for changes
> that are going to be committed successfully since the trigger is invoked
> before the change is being applied.
>
> Regarding #3, I seen that this requirement is fulfilled.
>
> Regarding #4, I didn’t find a way to figure out what values are actually
> changed, without re-reading the current (old) values from the table which
> of course impose a significant performance penalty.
>
>
>
> Are my assumptions correct?
>
> If yes then triggers are not the right approach. What other ways can be
> taken?
>
>
>
> Regards,
>
> Oren
>
>
>

Choosing a compaction strategy (TWCS)

[Voytek Jarnot]

Scenario:
Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra
tables, basically time-series - think log or auditing.  Retention is 10
years, but greater than 95% of reads will occur on data written within the
last year. 7 day TTL used on a small percentage of the records, majority do
not use TTL. Other than the aforementioned TTL, and the 10-year purge, no
updates or deletes are done.

Seems like TWCS is the right choice, but I have a few questions/concerns:

1) I'll be bulk loading a few years of existing data upon deployment - any
issues with that?  I assume using "with timestamp" when inserting this data
will be mandatory if I choose TWCS?

2) I read here (https://github.com/jeffjirsa/twcs/) that "You should target
fewer than 50 buckets per table based on your TTL." That's going to be a
tough goal with a 10 year retention ... can anyone speak to how important
this target really is?

3) If I'm bucketing my data with week/year (i.e., partition on year, week -
so today would be in 2016, 50), it seems like a natural fit for
compaction_window_size would be 7 days, but I'm thinking my calendar-based
weeks will never align with TWCS 7-day-period weeks anyway - am I missing
something there?

I'd appreciate any other thoughts on compaction and/or twcs.

Thanks

Re: Choosing a compaction strategy (TWCS)

[Jeff Jirsa]

With a 10 year retention, just ignore the target sstable count (I should remove 
that guidance, to be honest), and go for a 1 week window to match your 
partition size. 520 sstables on disk isn’t going to hurt you as long as you’re 
not reading from all of them, and with a partition-per-week the bloom filter is 
going to make things nice and easy for you.

 

-  Jeff

 

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 10:37 AM
To: "user@cassandra.apache.org" 
Subject: Choosing a compaction strategy (TWCS)

 

Scenario: 

Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra 
tables, basically time-series - think log or auditing.  Retention is 10 years, 
but greater than 95% of reads will occur on data written within the last year. 
7 day TTL used on a small percentage of the records, majority do not use TTL. 
Other than the aforementioned TTL, and the 10-year purge, no updates or deletes 
are done.

 

Seems like TWCS is the right choice, but I have a few questions/concerns:

 

1) I'll be bulk loading a few years of existing data upon deployment - any 
issues with that?  I assume using "with timestamp" when inserting this data 
will be mandatory if I choose TWCS?

 

2) I read here (https://github.com/jeffjirsa/twcs/) that "You should target 
fewer than 50 buckets per table based on your TTL." That's going to be a tough 
goal with a 10 year retention ... can anyone speak to how important this target 
really is?

 

3) If I'm bucketing my data with week/year (i.e., partition on year, week - so 
today would be in 2016, 50), it seems like a natural fit for 
compaction_window_size would be 7 days, but I'm thinking my calendar-based 
weeks will never align with TWCS 7-day-period weeks anyway - am I missing 
something there?

 

I'd appreciate any other thoughts on compaction and/or twcs.

 

Thanks



smime.p7s
Description: S/MIME cryptographic signature

Re: Choosing a compaction strategy (TWCS)

[Jeff Jirsa]

I skipped over the more important question  - loading data in. Two options:

1)   Load data in order through the normal writepath and use “USING 
TIMESTAMP” to set the timestamp, or

2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables, then 
sstableloader them into the cluster.

 

Either way, try not to mix writes of old data and new data in the “normal” 
write path  at the same time, even if you write “USING TIMESTAMP”, because 
it’ll get mixed in the memTable, and flushed into the same sstable – it won’t 
kill you, but if you can avoid it, avoid it.

 

-  Jeff

 

 

From: Jeff Jirsa 
Date: Friday, December 16, 2016 at 10:47 AM
To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

With a 10 year retention, just ignore the target sstable count (I should remove 
that guidance, to be honest), and go for a 1 week window to match your 
partition size. 520 sstables on disk isn’t going to hurt you as long as you’re 
not reading from all of them, and with a partition-per-week the bloom filter is 
going to make things nice and easy for you.

 

-  Jeff

 

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 10:37 AM
To: "user@cassandra.apache.org" 
Subject: Choosing a compaction strategy (TWCS)

 

Scenario: 

Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra 
tables, basically time-series - think log or auditing.  Retention is 10 years, 
but greater than 95% of reads will occur on data written within the last year. 
7 day TTL used on a small percentage of the records, majority do not use TTL. 
Other than the aforementioned TTL, and the 10-year purge, no updates or deletes 
are done.

 

Seems like TWCS is the right choice, but I have a few questions/concerns:

 

1) I'll be bulk loading a few years of existing data upon deployment - any 
issues with that?  I assume using "with timestamp" when inserting this data 
will be mandatory if I choose TWCS?

 

2) I read here (https://github.com/jeffjirsa/twcs/) that "You should target 
fewer than 50 buckets per table based on your TTL." That's going to be a tough 
goal with a 10 year retention ... can anyone speak to how important this target 
really is?

 

3) If I'm bucketing my data with week/year (i.e., partition on year, week - so 
today would be in 2016, 50), it seems like a natural fit for 
compaction_window_size would be 7 days, but I'm thinking my calendar-based 
weeks will never align with TWCS 7-day-period weeks anyway - am I missing 
something there?

 

I'd appreciate any other thoughts on compaction and/or twcs.

 

Thanks



smime.p7s
Description: S/MIME cryptographic signature

Re: Choosing a compaction strategy (TWCS)

[Voytek Jarnot]

Thank you Jeff - always nice to hear straight from the source.

Any issues you can see with 3 (my calendar-week bucket not aligning with
the arbitrary 7-day window)? Or am I confused (I'd put money on this
option, but I've been wrong once or twice before)?

On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa 
wrote:

> I skipped over the more important question  - loading data in. Two options:
>
> 1)   Load data in order through the normal writepath and use “USING
> TIMESTAMP” to set the timestamp, or
>
> 2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables,
> then sstableloader them into the cluster.
>
>
>
> Either way, try not to mix writes of old data and new data in the “normal”
> write path  at the same time, even if you write “USING TIMESTAMP”, because
> it’ll get mixed in the memTable, and flushed into the same sstable – it
> won’t kill you, but if you can avoid it, avoid it.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Jeff Jirsa 
> *Date: *Friday, December 16, 2016 at 10:47 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> With a 10 year retention, just ignore the target sstable count (I should
> remove that guidance, to be honest), and go for a 1 week window to match
> your partition size. 520 sstables on disk isn’t going to hurt you as long
> as you’re not reading from all of them, and with a partition-per-week the
> bloom filter is going to make things nice and easy for you.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 10:37 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Choosing a compaction strategy (TWCS)
>
>
>
> Scenario:
>
> Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra
> tables, basically time-series - think log or auditing.  Retention is 10
> years, but greater than 95% of reads will occur on data written within the
> last year. 7 day TTL used on a small percentage of the records, majority do
> not use TTL. Other than the aforementioned TTL, and the 10-year purge, no
> updates or deletes are done.
>
>
>
> Seems like TWCS is the right choice, but I have a few questions/concerns:
>
>
>
> 1) I'll be bulk loading a few years of existing data upon deployment - any
> issues with that?  I assume using "with timestamp" when inserting this data
> will be mandatory if I choose TWCS?
>
>
>
> 2) I read here (https://github.com/jeffjirsa/twcs/
> )
> that "You should target fewer than 50 buckets per table based on your TTL."
> That's going to be a tough goal with a 10 year retention ... can anyone
> speak to how important this target really is?
>
>
>
> 3) If I'm bucketing my data with week/year (i.e., partition on year, week
> - so today would be in 2016, 50), it seems like a natural fit for
> compaction_window_size would be 7 days, but I'm thinking my calendar-based
> weeks will never align with TWCS 7-day-period weeks anyway - am I missing
> something there?
>
>
>
> I'd appreciate any other thoughts on compaction and/or twcs.
>
>
>
> Thanks
>

Re: Choosing a compaction strategy (TWCS)

[Jeff Jirsa]

The issue is that your partitions will likely be in 2 sstables instead of 
“theoretically” 1. In practice, they’re probably going to bleed into 2 anyway 
(memTable flush to sstable isn’t going to happen exactly when the window 
expires, so it’ll bleed a bit anyway), so I bet no meaningful impact.

 

-  Jeff

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 11:12 AM
To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

Thank you Jeff - always nice to hear straight from the source. 

 

Any issues you can see with 3 (my calendar-week bucket not aligning with the 
arbitrary 7-day window)? Or am I confused (I'd put money on this option, but 
I've been wrong once or twice before)?

 

On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa  wrote:

I skipped over the more important question  - loading data in. Two options:

1)   Load data in order through the normal writepath and use “USING 
TIMESTAMP” to set the timestamp, or

2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables, then 
sstableloader them into the cluster.

 

Either way, try not to mix writes of old data and new data in the “normal” 
write path  at the same time, even if you write “USING TIMESTAMP”, because 
it’ll get mixed in the memTable, and flushed into the same sstable – it won’t 
kill you, but if you can avoid it, avoid it.

 

-  Jeff

 

 

From: Jeff Jirsa 
Date: Friday, December 16, 2016 at 10:47 AM
To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

With a 10 year retention, just ignore the target sstable count (I should remove 
that guidance, to be honest), and go for a 1 week window to match your 
partition size. 520 sstables on disk isn’t going to hurt you as long as you’re 
not reading from all of them, and with a partition-per-week the bloom filter is 
going to make things nice and easy for you.

 

-  Jeff

 

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 10:37 AM
To: "user@cassandra.apache.org" 
Subject: Choosing a compaction strategy (TWCS)

 

Scenario: 

Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra 
tables, basically time-series - think log or auditing.  Retention is 10 years, 
but greater than 95% of reads will occur on data written within the last year. 
7 day TTL used on a small percentage of the records, majority do not use TTL. 
Other than the aforementioned TTL, and the 10-year purge, no updates or deletes 
are done.

 

Seems like TWCS is the right choice, but I have a few questions/concerns:

 

1) I'll be bulk loading a few years of existing data upon deployment - any 
issues with that?  I assume using "with timestamp" when inserting this data 
will be mandatory if I choose TWCS?

 

2) I read here (https://github.com/jeffjirsa/twcs/) that "You should target 
fewer than 50 buckets per table based on your TTL." That's going to be a tough 
goal with a 10 year retention ... can anyone speak to how important this target 
really is?

 

3) If I'm bucketing my data with week/year (i.e., partition on year, week - so 
today would be in 2016, 50), it seems like a natural fit for 
compaction_window_size would be 7 days, but I'm thinking my calendar-based 
weeks will never align with TWCS 7-day-period weeks anyway - am I missing 
something there?

 

I'd appreciate any other thoughts on compaction and/or twcs.

 

Thanks

 



smime.p7s
Description: S/MIME cryptographic signature

Incremental repair for the first time

[Kathiresan S]

Hi,

We have a brand new Cassandra cluster (version 3.0.4) and we set up
nodetool repair scheduled for every day (without any options for repair).
As per documentation, incremental repair is the default in this case.
Should we do a full repair for the very first time on each node once and
then leave it to do incremental repair afterwards?

*Problem we are facing:*

On a random node, the repair process throws validation failed error,
pointing to some other node

For Eg. Node A, where the repair is run (without any option), throws below
error

*Validation failed in /Node B*

In Node B when we check the logs, below exception is seen at the same exact
time...

*java.lang.RuntimeException: Cannot start multiple repair sessions over the
same sstables*
*at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1087)
~[apache-cassandra-3.0.4.jar:3.0.4]*
*at
org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:80)
~[apache-cassandra-3.0.4.jar:3.0.4]*
*at
org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:700)
~[apache-cassandra-3.0.4.jar:3.0.4]*
*at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_73]*
*at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_73]*

Can you please help on how this can be fixed?

Thanks,
Kathir

Re: Incremental repair for the first time

[Jake Luciani]

This was fixed post 3.0.4 please upgrade to latest 3.0 release

On Fri, Dec 16, 2016 at 4:49 PM, Kathiresan S 
wrote:

> Hi,
>
> We have a brand new Cassandra cluster (version 3.0.4) and we set up
> nodetool repair scheduled for every day (without any options for repair).
> As per documentation, incremental repair is the default in this case.
> Should we do a full repair for the very first time on each node once and
> then leave it to do incremental repair afterwards?
>
> *Problem we are facing:*
>
> On a random node, the repair process throws validation failed error,
> pointing to some other node
>
> For Eg. Node A, where the repair is run (without any option), throws below
> error
>
> *Validation failed in /Node B*
>
> In Node B when we check the logs, below exception is seen at the same
> exact time...
>
> *java.lang.RuntimeException: Cannot start multiple repair sessions over
> the same sstables*
> *at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1087)
> ~[apache-cassandra-3.0.4.jar:3.0.4]*
> *at
> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:80)
> ~[apache-cassandra-3.0.4.jar:3.0.4]*
> *at
> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:700)
> ~[apache-cassandra-3.0.4.jar:3.0.4]*
> *at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_73]*
> *at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_73]*
>
> Can you please help on how this can be fixed?
>
> Thanks,
> Kathir
>



-- 
http://twitter.com/tjake

Re: Incremental repair for the first time

[Kathiresan S]

Thank you!

Is any work around available for this version?

Thanks,
Kathir

On Friday, December 16, 2016, Jake Luciani  wrote:

> This was fixed post 3.0.4 please upgrade to latest 3.0 release
>
> On Fri, Dec 16, 2016 at 4:49 PM, Kathiresan S <
> kathiresanselva...@gmail.com
> > wrote:
>
>> Hi,
>>
>> We have a brand new Cassandra cluster (version 3.0.4) and we set up
>> nodetool repair scheduled for every day (without any options for repair).
>> As per documentation, incremental repair is the default in this case.
>> Should we do a full repair for the very first time on each node once and
>> then leave it to do incremental repair afterwards?
>>
>> *Problem we are facing:*
>>
>> On a random node, the repair process throws validation failed error,
>> pointing to some other node
>>
>> For Eg. Node A, where the repair is run (without any option), throws
>> below error
>>
>> *Validation failed in /Node B*
>>
>> In Node B when we check the logs, below exception is seen at the same
>> exact time...
>>
>> *java.lang.RuntimeException: Cannot start multiple repair sessions over
>> the same sstables*
>> *at
>> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1087)
>> ~[apache-cassandra-3.0.4.jar:3.0.4]*
>> *at
>> org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:80)
>> ~[apache-cassandra-3.0.4.jar:3.0.4]*
>> *at
>> org.apache.cassandra.db.compaction.CompactionManager$10.call(CompactionManager.java:700)
>> ~[apache-cassandra-3.0.4.jar:3.0.4]*
>> *at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_73]*
>> *at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_73]*
>>
>> Can you please help on how this can be fixed?
>>
>> Thanks,
>> Kathir
>>
>
>
>
> --
> http://twitter.com/tjake
>

Re: FW: Cassandra trigger to send notifications

[Eric Stevens]

You probably want to look at change data capture rather than triggers:
http://cassandra.apache.org/doc/latest/operating/cdc.html

Be aware that one of your criteria regarding operation order is going to be
very difficult to guarantee due to eventual consistency.

On Fri, Dec 16, 2016, 2:43 AM Matija Gobec  wrote:

> Hi Oren,
>
> I've spent a reasonable time working out triggers and I would say that
> your best bet is doing this in the app.
> Just publish a rabbitmq message from the app when you execute a statement.
> If your goal is to have an audit then try batch writing data to the tables
> and delta to their audit counterparts.
>
> Matija
>
> On Fri, Dec 16, 2016 at 9:14 AM, Oren Yekutieli <
> oren.yekuti...@ecitele.com> wrote:
>
> Hello,
>
>
>
> In a Cassandra cluster I want to push a notification to rabbitmq whenever
> a change (insert/update/delete) was made to some Cassandra tables, with the
> following requirements:
>
> The notifications should:
>
> 1.  Be ordered in the same order the changes were stored.
>
> 2.  Be sent only if the change was (or going to be) committed
> successfully.
>
> And in case of insert/update:
>
> 3.  Include all the row values.
>
> 4.  Indicate what values were changed.
>
>
>
> I’m trying to figure out if Cassandra triggers are the correct way to go.
>
> According to
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-0-prototype-triggers-support,
> one of the potential use cases for triggers is:
>
> ·>> implementing alerts/notifications
>
> I also played a bit with the example given here:
> http://stackoverflow.com/questions/35161747/example-of-creating-triggers-in-cassandra-and-does-this-support-only-for-java
> and from what I read and learned so far I assume that:
>
> Regarding #1, I doubt if the notification order can be kept since the
> trigger is invoked on the coordinator node while there are multiple clients
> that can connect to the various cluster nodes (coordinators).
>
> Regarding #2, I’m not sure that notification will be sent only for changes
> that are going to be committed successfully since the trigger is invoked
> before the change is being applied.
>
> Regarding #3, I seen that this requirement is fulfilled.
>
> Regarding #4, I didn’t find a way to figure out what values are actually
> changed, without re-reading the current (old) values from the table which
> of course impose a significant performance penalty.
>
>
>
> Are my assumptions correct?
>
> If yes then triggers are not the right approach. What other ways can be
> taken?
>
>
>
> Regards,
>
> Oren
>
>
>
>
>

High CPU on nodes

[Anubhav Kale]

Hello,

I am trying to fight a high CPU problem on some of our nodes. Thread dumps show 
that it's not GC threads (we have 30GB heap), iostat %iowait confirms it's not 
disk (ranges between 0.3 - 0.9%). One of the ways in which the problem 
manifests is that the nodes can't compact SSTables and it happens randomly. We 
run Cassandra 2.1.13 on Azure Premium Storage (network attached SSDs).

One of the sample threads that was taking high CPU shows :

"pool-13-thread-1" #3352 prio=5 
os_prio=0 tid=0x7f2275340bb0 nid=0x1b0b runnable [0x7f33ffaae000]
java.lang.Thread.State: RUNNABLE
at java.util.TimSort.gallopRight(TimSort.java:632)
at java.util.TimSort.mergeLo(TimSort.java:739)
at java.util.TimSort.mergeAt(TimSort.java:514)
at java.util.TimSort.mergeCollapse(TimSort.java:441)
at java.util.TimSort.sort(TimSort.java:245)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:163)
at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:200)
at 
org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:152)
at 
org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1581)
at 
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:1739)

Looking at code, I can't figure out why things like this would require a high 
CPU and I don't find any JIRAs relating this as well. So, what can I do next to 
troubleshoot this ?

Thanks !

Re: Choosing a compaction strategy (TWCS)

[Voytek Jarnot]

Thanks again, Jeff.

Thinking about this some more, I'm wondering if I'm overthinking or if
there's a potential issue:

If my compaction_window_size is 7 (DAYS), and I've got TTLs of 7 days on
some (relatively small percentage) of my records - am I going to be leaving
tombstones around all over the place?  My noob-read on this is that TWCS
will not compact tables comprised of records older than 7 days (
https://docs.datastax.com/en/cassandra/3.x/cassandra/dml/dmlHowDataMaintain.html#dmlHowDataMaintain__twcs),
but Cassandra will not evict my tombstones until 7 days + consideration for
gc_grace_seconds have passed ... resulting in no tombstone removal (?).



On Fri, Dec 16, 2016 at 1:17 PM, Jeff Jirsa 
wrote:

> The issue is that your partitions will likely be in 2 sstables instead of
> “theoretically” 1. In practice, they’re probably going to bleed into 2
> anyway (memTable flush to sstable isn’t going to happen exactly when the
> window expires, so it’ll bleed a bit anyway), so I bet no meaningful impact.
>
>
>
> -  Jeff
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 11:12 AM
>
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> Thank you Jeff - always nice to hear straight from the source.
>
>
>
> Any issues you can see with 3 (my calendar-week bucket not aligning with
> the arbitrary 7-day window)? Or am I confused (I'd put money on this
> option, but I've been wrong once or twice before)?
>
>
>
> On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa 
> wrote:
>
> I skipped over the more important question  - loading data in. Two options:
>
> 1)   Load data in order through the normal writepath and use “USING
> TIMESTAMP” to set the timestamp, or
>
> 2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables,
> then sstableloader them into the cluster.
>
>
>
> Either way, try not to mix writes of old data and new data in the “normal”
> write path  at the same time, even if you write “USING TIMESTAMP”, because
> it’ll get mixed in the memTable, and flushed into the same sstable – it
> won’t kill you, but if you can avoid it, avoid it.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Jeff Jirsa 
> *Date: *Friday, December 16, 2016 at 10:47 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> With a 10 year retention, just ignore the target sstable count (I should
> remove that guidance, to be honest), and go for a 1 week window to match
> your partition size. 520 sstables on disk isn’t going to hurt you as long
> as you’re not reading from all of them, and with a partition-per-week the
> bloom filter is going to make things nice and easy for you.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 10:37 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Choosing a compaction strategy (TWCS)
>
>
>
> Scenario:
>
> Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra
> tables, basically time-series - think log or auditing.  Retention is 10
> years, but greater than 95% of reads will occur on data written within the
> last year. 7 day TTL used on a small percentage of the records, majority do
> not use TTL. Other than the aforementioned TTL, and the 10-year purge, no
> updates or deletes are done.
>
>
>
> Seems like TWCS is the right choice, but I have a few questions/concerns:
>
>
>
> 1) I'll be bulk loading a few years of existing data upon deployment - any
> issues with that?  I assume using "with timestamp" when inserting this data
> will be mandatory if I choose TWCS?
>
>
>
> 2) I read here (https://github.com/jeffjirsa/twcs/
> )
> that "You should target fewer than 50 buckets per table based on your TTL."
> That's going to be a tough goal with a 10 year retention ... can anyone
> speak to how important this target really is?
>
>
>
> 3) If I'm bucketing my data with week/year (i.e., partition on year, week
> - so today would be in 2016, 50), it seems like a natural fit for
> compaction_window_size would be 7 days, but I'm thinking my calendar-based
> weeks will never align with TWCS 7-day-period weeks anyway - am I missing
> something there?
>
>
>
> I'd appreciate any other thoughts on compaction and/or twcs.
>
>
>
> Thanks
>
>
>

Re: Choosing a compaction strategy (TWCS)

[Jeff Jirsa]

Tombstone compaction subproperties can handle tombstone removal for you (you’ll 
set a ratio of tombstones worth compacting away – for example, 80%, and set an 
interval to prevent continuous compaction – for example, 24 hours, and then 
anytime there’s no other work to do, if there’s an sstable over 24 hours old 
that’s at least 80% tombstones, it’ll compact it in a single sstable 
compaction).

 

-  Jeff

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 7:34 PM
To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

Thanks again, Jeff. 

 

Thinking about this some more, I'm wondering if I'm overthinking or if there's 
a potential issue:

 

If my compaction_window_size is 7 (DAYS), and I've got TTLs of 7 days on some 
(relatively small percentage) of my records - am I going to be leaving 
tombstones around all over the place?  My noob-read on this is that TWCS will 
not compact tables comprised of records older than 7 days 
(https://docs.datastax.com/en/cassandra/3.x/cassandra/dml/dmlHowDataMaintain.html#dmlHowDataMaintain__twcs),
 but Cassandra will not evict my tombstones until 7 days + consideration for 
gc_grace_seconds have passed ... resulting in no tombstone removal (?).

 

 

 

On Fri, Dec 16, 2016 at 1:17 PM, Jeff Jirsa  wrote:

The issue is that your partitions will likely be in 2 sstables instead of 
“theoretically” 1. In practice, they’re probably going to bleed into 2 anyway 
(memTable flush to sstable isn’t going to happen exactly when the window 
expires, so it’ll bleed a bit anyway), so I bet no meaningful impact.

 

-  Jeff

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 11:12 AM


To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

Thank you Jeff - always nice to hear straight from the source. 

 

Any issues you can see with 3 (my calendar-week bucket not aligning with the 
arbitrary 7-day window)? Or am I confused (I'd put money on this option, but 
I've been wrong once or twice before)?

 

On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa  wrote:

I skipped over the more important question  - loading data in. Two options:

1)   Load data in order through the normal writepath and use “USING 
TIMESTAMP” to set the timestamp, or

2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables, then 
sstableloader them into the cluster.

 

Either way, try not to mix writes of old data and new data in the “normal” 
write path  at the same time, even if you write “USING TIMESTAMP”, because 
it’ll get mixed in the memTable, and flushed into the same sstable – it won’t 
kill you, but if you can avoid it, avoid it.

 

-  Jeff

 

 

From: Jeff Jirsa 
Date: Friday, December 16, 2016 at 10:47 AM
To: "user@cassandra.apache.org" 
Subject: Re: Choosing a compaction strategy (TWCS)

 

With a 10 year retention, just ignore the target sstable count (I should remove 
that guidance, to be honest), and go for a 1 week window to match your 
partition size. 520 sstables on disk isn’t going to hurt you as long as you’re 
not reading from all of them, and with a partition-per-week the bloom filter is 
going to make things nice and easy for you.

 

-  Jeff

 

 

From: Voytek Jarnot 
Reply-To: "user@cassandra.apache.org" 
Date: Friday, December 16, 2016 at 10:37 AM
To: "user@cassandra.apache.org" 
Subject: Choosing a compaction strategy (TWCS)

 

Scenario: 

Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra 
tables, basically time-series - think log or auditing.  Retention is 10 years, 
but greater than 95% of reads will occur on data written within the last year. 
7 day TTL used on a small percentage of the records, majority do not use TTL. 
Other than the aforementioned TTL, and the 10-year purge, no updates or deletes 
are done.

 

Seems like TWCS is the right choice, but I have a few questions/concerns:

 

1) I'll be bulk loading a few years of existing data upon deployment - any 
issues with that?  I assume using "with timestamp" when inserting this data 
will be mandatory if I choose TWCS?

 

2) I read here (https://github.com/jeffjirsa/twcs/) that "You should target 
fewer than 50 buckets per table based on your TTL." That's going to be a tough 
goal with a 10 year retention ... can anyone speak to how important this target 
really is?

 

3) If I'm bucketing my data with week/year (i.e., partition on year, week - so 
today would be in 2016, 50), it seems like a natural fit for 
compaction_window_size would be 7 days, but I'm thinking my calendar-based 
weeks will never align with TWCS 7-day-period weeks anyway - am I missing 
something there?

 

I'd appreciate any other thoughts on compaction and/or twcs.

 

Thanks

 

 



smime.p7s
Description: S/MIME cryptographic signature

Re: Choosing a compaction strategy (TWCS)

[Voytek Jarnot]

Gotcha.  "never compacted" has an implicit asterisk referencing
tombstone_compaction_interval and tombstone_threshold, sounds like.  More
of a "never compacted" via strategy selection, but eligible for
tombstone-triggered compaction.

On Fri, Dec 16, 2016 at 10:07 PM, Jeff Jirsa 
wrote:

> Tombstone compaction subproperties can handle tombstone removal for you
> (you’ll set a ratio of tombstones worth compacting away – for example, 80%,
> and set an interval to prevent continuous compaction – for example, 24
> hours, and then anytime there’s no other work to do, if there’s an sstable
> over 24 hours old that’s at least 80% tombstones, it’ll compact it in a
> single sstable compaction).
>
>
>
> -  Jeff
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 7:34 PM
>
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> Thanks again, Jeff.
>
>
>
> Thinking about this some more, I'm wondering if I'm overthinking or if
> there's a potential issue:
>
>
>
> If my compaction_window_size is 7 (DAYS), and I've got TTLs of 7 days on
> some (relatively small percentage) of my records - am I going to be leaving
> tombstones around all over the place?  My noob-read on this is that TWCS
> will not compact tables comprised of records older than 7 days (
> https://docs.datastax.com/en/cassandra/3.x/cassandra/dml/
> dmlHowDataMaintain.html#dmlHowDataMaintain__twcs
> ),
> but Cassandra will not evict my tombstones until 7 days + consideration for
> gc_grace_seconds have passed ... resulting in no tombstone removal (?).
>
>
>
>
>
>
>
> On Fri, Dec 16, 2016 at 1:17 PM, Jeff Jirsa 
> wrote:
>
> The issue is that your partitions will likely be in 2 sstables instead of
> “theoretically” 1. In practice, they’re probably going to bleed into 2
> anyway (memTable flush to sstable isn’t going to happen exactly when the
> window expires, so it’ll bleed a bit anyway), so I bet no meaningful impact.
>
>
>
> -  Jeff
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 11:12 AM
>
>
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> Thank you Jeff - always nice to hear straight from the source.
>
>
>
> Any issues you can see with 3 (my calendar-week bucket not aligning with
> the arbitrary 7-day window)? Or am I confused (I'd put money on this
> option, but I've been wrong once or twice before)?
>
>
>
> On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa 
> wrote:
>
> I skipped over the more important question  - loading data in. Two options:
>
> 1)   Load data in order through the normal writepath and use “USING
> TIMESTAMP” to set the timestamp, or
>
> 2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables,
> then sstableloader them into the cluster.
>
>
>
> Either way, try not to mix writes of old data and new data in the “normal”
> write path  at the same time, even if you write “USING TIMESTAMP”, because
> it’ll get mixed in the memTable, and flushed into the same sstable – it
> won’t kill you, but if you can avoid it, avoid it.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Jeff Jirsa 
> *Date: *Friday, December 16, 2016 at 10:47 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: Choosing a compaction strategy (TWCS)
>
>
>
> With a 10 year retention, just ignore the target sstable count (I should
> remove that guidance, to be honest), and go for a 1 week window to match
> your partition size. 520 sstables on disk isn’t going to hurt you as long
> as you’re not reading from all of them, and with a partition-per-week the
> bloom filter is going to make things nice and easy for you.
>
>
>
> -  Jeff
>
>
>
>
>
> *From: *Voytek Jarnot 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Friday, December 16, 2016 at 10:37 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Choosing a compaction strategy (TWCS)
>
>
>
> Scenario:
>
> Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra
> tables, basically time-series - think log or auditing.  Retention is 10
> years, but greater than 95% of reads will occur on data written within the
> last year. 7 day TTL used on a small percentage of the records, majority do
> not use TTL. Other than the aforementioned TTL, and the 10-year purge, no
> updates or deletes are done.
>
>
>
> Seems like TWCS is the right choice, but I have a few questions/concerns:
>
>
>
> 1) I'll be bulk loading a few years of existing data upon deployment - any
> issues with that?  I assume using "

Re: Choosing a compaction strategy (TWCS)

[Jeff Jirsa]

With the caveat that tombstone compactions are disabled by default in TWCS (and 
DTCS)

-- 
Jeff Jirsa


> On Dec 16, 2016, at 8:34 PM, Voytek Jarnot  wrote:
> 
> Gotcha.  "never compacted" has an implicit asterisk referencing 
> tombstone_compaction_interval and tombstone_threshold, sounds like.  More of 
> a "never compacted" via strategy selection, but eligible for 
> tombstone-triggered compaction.
> 
>> On Fri, Dec 16, 2016 at 10:07 PM, Jeff Jirsa  
>> wrote:
>> Tombstone compaction subproperties can handle tombstone removal for you 
>> (you’ll set a ratio of tombstones worth compacting away – for example, 80%, 
>> and set an interval to prevent continuous compaction – for example, 24 
>> hours, and then anytime there’s no other work to do, if there’s an sstable 
>> over 24 hours old that’s at least 80% tombstones, it’ll compact it in a 
>> single sstable compaction).
>> 
>>  
>> 
>> -  Jeff
>> 
>>  
>> 
>> From: Voytek Jarnot 
>> Reply-To: "user@cassandra.apache.org" 
>> Date: Friday, December 16, 2016 at 7:34 PM
>> 
>> 
>> To: "user@cassandra.apache.org" 
>> Subject: Re: Choosing a compaction strategy (TWCS)
>>  
>> 
>> Thanks again, Jeff.
>> 
>>  
>> 
>> Thinking about this some more, I'm wondering if I'm overthinking or if 
>> there's a potential issue:
>> 
>>  
>> 
>> If my compaction_window_size is 7 (DAYS), and I've got TTLs of 7 days on 
>> some (relatively small percentage) of my records - am I going to be leaving 
>> tombstones around all over the place?  My noob-read on this is that TWCS 
>> will not compact tables comprised of records older than 7 days 
>> (https://docs.datastax.com/en/cassandra/3.x/cassandra/dml/dmlHowDataMaintain.html#dmlHowDataMaintain__twcs),
>>  but Cassandra will not evict my tombstones until 7 days + consideration for 
>> gc_grace_seconds have passed ... resulting in no tombstone removal (?).
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Fri, Dec 16, 2016 at 1:17 PM, Jeff Jirsa  
>> wrote:
>> 
>> The issue is that your partitions will likely be in 2 sstables instead of 
>> “theoretically” 1. In practice, they’re probably going to bleed into 2 
>> anyway (memTable flush to sstable isn’t going to happen exactly when the 
>> window expires, so it’ll bleed a bit anyway), so I bet no meaningful impact.
>> 
>>  
>> 
>> -  Jeff
>> 
>>  
>> 
>> From: Voytek Jarnot 
>> Reply-To: "user@cassandra.apache.org" 
>> Date: Friday, December 16, 2016 at 11:12 AM
>> 
>> 
>> To: "user@cassandra.apache.org" 
>> Subject: Re: Choosing a compaction strategy (TWCS)
>> 
>>  
>> 
>> Thank you Jeff - always nice to hear straight from the source.
>> 
>>  
>> 
>> Any issues you can see with 3 (my calendar-week bucket not aligning with the 
>> arbitrary 7-day window)? Or am I confused (I'd put money on this option, but 
>> I've been wrong once or twice before)?
>> 
>>  
>> 
>> On Fri, Dec 16, 2016 at 12:50 PM, Jeff Jirsa  
>> wrote:
>> 
>> I skipped over the more important question  - loading data in. Two options:
>> 
>> 1)   Load data in order through the normal writepath and use “USING 
>> TIMESTAMP” to set the timestamp, or
>> 
>> 2)   Use CQLSSTableWriter and “USING TIMESTAMP” to create sstables, then 
>> sstableloader them into the cluster.
>> 
>>  
>> 
>> Either way, try not to mix writes of old data and new data in the “normal” 
>> write path  at the same time, even if you write “USING TIMESTAMP”, because 
>> it’ll get mixed in the memTable, and flushed into the same sstable – it 
>> won’t kill you, but if you can avoid it, avoid it.
>> 
>>  
>> 
>> -  Jeff
>> 
>>  
>> 
>>  
>> 
>> From: Jeff Jirsa 
>> Date: Friday, December 16, 2016 at 10:47 AM
>> To: "user@cassandra.apache.org" 
>> Subject: Re: Choosing a compaction strategy (TWCS)
>> 
>>  
>> 
>> With a 10 year retention, just ignore the target sstable count (I should 
>> remove that guidance, to be honest), and go for a 1 week window to match 
>> your partition size. 520 sstables on disk isn’t going to hurt you as long as 
>> you’re not reading from all of them, and with a partition-per-week the bloom 
>> filter is going to make things nice and easy for you.
>> 
>>  
>> 
>> -  Jeff
>> 
>>  
>> 
>>  
>> 
>> From: Voytek Jarnot 
>> Reply-To: "user@cassandra.apache.org" 
>> Date: Friday, December 16, 2016 at 10:37 AM
>> To: "user@cassandra.apache.org" 
>> Subject: Choosing a compaction strategy (TWCS)
>> 
>>  
>> 
>> Scenario:
>> 
>> Converting an Oracle table to Cassandra, one Oracle table to 4 Cassandra 
>> tables, basically time-series - think log or auditing.  Retention is 10 
>> years, but greater than 95% of reads will occur on data written within the 
>> last year. 7 day TTL used on a small percentage of the records, majority do 
>> not use TTL. Other than the aforementioned TTL, and the 10-year purge, no 
>> updates or deletes are done.
>> 
>>  
>> 
>> Seems like TWCS is the right choice, but I have a few questions/concerns:
>> 
>>  
>> 
>> 1) I'll be bulk loading a few years of existing