Re: User Defined Compaction Issue

[shalom sagges]

Thanks Jeff!

I'll try that.
I'm not sure I understand how the tombstones are covering data in another
file. Do you have a small example perhaps?

Thanks again!

On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:

> The problem is likely that your sstables overlap - your 91% droppable
> tombstones are probably covering data in another file. Until that data is
> removed, those tombstones cant be dropped.
>
> This is why a full major compaction often works better for simply
> reclaiming disk space (though it takes a lot more disk as it runs and has
> other drawbacks) - it joins the shadowed data with the tombstones into a
> single file. You can sorta fake that by using user-defined compaction with
> multiple sstables to try to compact away the shadowed data - pick 2 or more
> -Data files and compact them together. Works best if you can be sure which
> data is overlapping, but short of that you'll probably want to pick data
> with approximately the same (or older) calendar timestamps.
>
>
>
> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges 
> wrote:
>
>> Hi Everyone,
>>
>> I'm running into an issue I can't seem to Solve.
>>
>> I execute force compaction in order to reclaim back storage.
>> Everything was working fine for a time, but after a while I found that
>> tombstones aren't being removed any longer.
>>
>> For example, I've compacted the following SSTable:
>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>> Estimated droppable tombstones:
>>
>> *0.9115601492568154*
>> I ran it with jmxterm and the compaction started and completed
>> successfully.
>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>> forceUserDefinedCompaction file-jb-69103-Data.db
>>  #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>  #operation returns:
>> null
>>
>>
>> A new SStable was created, but no tombstones were removed.
>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>  Estimated droppable tombstones:
>>
>> * 0.9115601492568154*
>> I've tried running it from jconsole as well, but the outcome was the
>> same.
>>
>> Is anyone familiar with this issue?
>> Btw, this cluster is on version 2.0.14 .
>>
>>
>> Thanks!
>> Shalom
>>
>>
>

Datastax Driver Mapper & Secondary Indexes

[Daniel Hölbling-Inzko]

Hi,
I am currently moving an application from SQL to Cassandra using Java. I
successfully got the DataStax driver and the mapper up and running, but
can't seem to figure out how to set secondary indexes through the mapper.
I also can't seem to find anything related to indexes in the mapper sources
- am I missing something or is this missing from the client library?

greetings Daniel

Re: Datastax Driver Mapper & Secondary Indexes

[kurt greaves]

If you've created a secondary index you simply query it by specifying it as
part of the where clause. Note that you should really understand the
drawbacks of secondary indexes before using them, as they might not be
incredibly efficient depending on what you need them for.
http://www.wentnet.com/blog/?p=77 and
https://pantheon.io/blog/cassandra-scale-problem-secondary-indexes might
help.

On 26 September 2017 at 07:17, Daniel Hölbling-Inzko <
daniel.hoelbling-in...@bitmovin.com> wrote:

> Hi,
> I am currently moving an application from SQL to Cassandra using Java. I
> successfully got the DataStax driver and the mapper up and running, but
> can't seem to figure out how to set secondary indexes through the mapper.
> I also can't seem to find anything related to indexes in the mapper
> sources - am I missing something or is this missing from the client library?
>
> greetings Daniel
>

Compaction through put and compaction tasks

[Anshu Vajpayee]

Hello -

Ihave very generic question regarding  compaction. How does cassandra
internally generate the  number of comapction tasks?  How does it get
affected with compaction throughput ?

If we increase the  number of compaction  throughput, will the per second
compaction  task increase for same kind of write load ?

Finally whats your  recommendation for  compaction throughput for SSDs?

-- 
*​C*heers,*
*Anshu *

Re: Interrogation about expected performance

[kurt greaves]

Sounds reasonable. How big are your writes? Also are you seeing a
bottleneck? If so, what are the details? If everything is running fine with
200k writes/sec (compactions aren't backing up, not a lot of disk IO) then
that's good. However you will also need to compare what you can achieve
when you actually want to read the data, as reads are much more expensive
than writes and it may greatly impact the number of w/s you can achieve.

You can tune Cassandra a lot to squeeze out as much performance as
possible, but really depends what you're doing with it. Worth getting a
benchmark of the load you expect to be on the cluster (both reads and
writes) before going down a tuning path.
​

Re: Compaction through put and compaction tasks

[kurt greaves]

Number of active tasks is controlled by concurrent_compactors yaml setting.
Recommendation is set to number of cpu cores you have.

Number of pending tasks is an estimate generated by Cassandra to achieve a
completely compacted state (i.e, there are no more possible compactions).
How this is estimated depends on the compaction strategies in use.

If you increase the compaction throughput it simply increases the rate at
which C* will compact, allocating more resources to reading/writing from
disk.

On 26 September 2017 at 08:20, Anshu Vajpayee 
wrote:

> Hello -
>
> Ihave very generic question regarding  compaction. How does cassandra
> internally generate the  number of comapction tasks?  How does it get
> affected with compaction throughput ?
>
> If we increase the  number of compaction  throughput, will the per second
> compaction  task increase for same kind of write load ?
>
> Finally whats your  recommendation for  compaction throughput for SSDs?
>
> --
> *​C*heers,*
> *Anshu *
>
>
>

Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

[Matope Ono]

Hi. We met similar situation after upgrading from 2.1.14 to 3.11 in our
production.

Have you already tried G1GC instead of CMS? Our timeouts were mitigated
after replacing CMS with G1GC.

Thanks.

2017-09-25 20:01 GMT+09:00 Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com>:

> Hello,
>
>
>
> I have now some concrete numbers from our 9 node loadtest cluster with
> constant load, same infrastructure after upgrading to 3.0.14 from 2.1.18.
>
>
>
> We see doubled GC suspension time + correlating CPU increase. In short,
> 3.0.14 is not able to handle the same load.
>
>
>
> I have created https://issues.apache.org/jira/browse/CASSANDRA-13900.
> Feel free to request any further additional information on the ticket.
>
>
>
> Unfortunately this is a real show-stopper for us upgrading to 3.0.
>
>
>
> Thanks for your attention.
>
>
>
> Thomas
>
>
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
> *Sent:* Freitag, 15. September 2017 13:51
> *To:* user@cassandra.apache.org
> *Subject:* RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)
>
>
>
> Hi Jeff,
>
>
>
> we are using native (CQL3) via Java DataStax driver (3.1). We also have
> OpsCenter running (to be removed soon) via Thrift, if I remember correctly.
>
>
>
> As said, the write request latency for our keyspace hasn’t really changed,
> so perhaps another one (system related, OpsCenter …?) is affected or
> perhaps the JMX metric is reporting something differently now. J So not a
> real issue for now hopefully, just popping up in our monitoring, wondering
> what this may be.
>
>
>
> Regarding compression metadata memory usage drop. Right, storage engine
> re-write could be a reason. Thanks.
>
>
>
> Still wondering about the GC/CPU increase.
>
>
>
> Thanks!
>
>
>
> Thomas
>
>
>
>
>
>
>
> *From:* Jeff Jirsa [mailto:jji...@gmail.com ]
> *Sent:* Freitag, 15. September 2017 13:14
> *To:* user@cassandra.apache.org
> *Subject:* Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)
>
>
>
> Most people find 3.0 slightly slower than 2.1. The only thing that really
> stands out in your email is the huge change in 95% latency - that's
> atypical. Are you using thrift or native 9042)?  The decrease in
> compression metadata offheap usage is likely due to the increased storage
> efficiency of the storage engine (see Cassandra-8099).
>
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Sep 15, 2017, at 2:37 AM, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hello,
>
>
>
> we have a test (regression) environment hosted in AWS, which is used for
> auto deploying our software on a daily basis and attach constant load
> across all deployments. Basically to allow us to detect any regressions in
> our software on a daily basis.
>
>
>
> On the Cassandra-side, this is single-node in AWS, m4.xlarge, EBS gp2, 8G
> heap, CMS. The environment has also been upgraded from Cassandra 2.1.18 to
> 3.0.14 at a certain point in time. Without running upgradesstables so far.
> We have not made any additional JVM/GC configuration change when going from
> 2.1.18 to 3.0.14 on our own, thus, any self-made configuration changes
> (e.g. new gen heap size) for 2.1.18 are also in place with 3.0.14.
>
>
>
> What we see after a time-frame of ~ 7 days (so, e.g. should not be caused
> by some sort of spiky compaction pattern) is an AVG increase in GC/CPU
> (most likely correlating):
>
> · CPU: ~ 12% => ~ 17%
>
> · GC Suspension: ~ 1,7% => 3,29%
>
>
>
> In this environment not a big deal, but relatively we have a CPU increase
> of ~ 50% (with increased GC most likely contributing). Something we have
> deal with when going into production (going into larger, multi-node
> loadtest environments first though).
>
>
>
> Beside the CPU/GC shift, we also monitor the following noticeable changes
> (don’t know if they somehow correlate with the CPU/GC shift above):
>
> · Increased AVG Write Client Requests Latency (95th Percentile),
> org.apache.cassandra.metrics.ClientRequest.Latency.Write: 6,05ms =>
> 29,2ms, but almost constant (no change in) write client request latency for
> our particular keyspace only, org.apache.cassandra.metrics.
> Keyspace.ruxitdb.WriteLatency
>
> · Compression metadata memory usage drop,
> org.apache.cassandra.metrics.Keyspace.XXX. 
> CompressionMetadataOffHeapMemoryUsed:
> ~218MB => ~105MB => Good or bad? Known?
>
>
>
> I know, looks all a bit vague, but perhaps someone else has seen something
> similar when upgrading to 3.0.14 and can share their thoughts/ideas.
> Especially the (relative) CPU/GC increase is something we are curious about.
>
>
>
> Thanks a lot.
>
>
>
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrac

Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

[Alexander Dejanovski]

Hi Thomas,

I wouldn't move to G1GC with small heaps (<24GB) but just looking at your
ticket I think that your new gen is way too small.
I get that it worked better in 2.1 in your case though, which would suggest
that the memory footprint is different between 2.1 and 3.0. It looks like
you're using batches extensively.
Hopefully you're aware that multi partition batches are discouraged because
they indeed create heap pressure and high coordination costs (on top of
batchlog writes/deletions), leading to more GC pauses.
With a 400MB new gen, you're very likely to have a lot of premature
promotions (especially with the default max tenuring threshold), which will
fill the old gen faster than necessary and is likely to trigger major GCs.

I'd suggest you re-run those tests with a 2GB new gen and compare results.
Know that with Cassandra you can easily go up to 40%-50% of your heap for
the new gen.

Cheers,


On Tue, Sep 26, 2017 at 10:58 AM Matope Ono  wrote:

> Hi. We met similar situation after upgrading from 2.1.14 to 3.11 in our
> production.
>
> Have you already tried G1GC instead of CMS? Our timeouts were mitigated
> after replacing CMS with G1GC.
>
> Thanks.
>
> 2017-09-25 20:01 GMT+09:00 Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com>:
>
>> Hello,
>>
>>
>>
>> I have now some concrete numbers from our 9 node loadtest cluster with
>> constant load, same infrastructure after upgrading to 3.0.14 from 2.1.18.
>>
>>
>>
>> We see doubled GC suspension time + correlating CPU increase. In short,
>> 3.0.14 is not able to handle the same load.
>>
>>
>>
>> I have created https://issues.apache.org/jira/browse/CASSANDRA-13900.
>> Feel free to request any further additional information on the ticket.
>>
>>
>>
>> Unfortunately this is a real show-stopper for us upgrading to 3.0.
>>
>>
>>
>> Thanks for your attention.
>>
>>
>>
>> Thomas
>>
>>
>>
>> *From:* Steinmaurer, Thomas [mailto:thomas.steinmau...@dynatrace.com]
>> *Sent:* Freitag, 15. September 2017 13:51
>> *To:* user@cassandra.apache.org
>> *Subject:* RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)
>>
>>
>>
>> Hi Jeff,
>>
>>
>>
>> we are using native (CQL3) via Java DataStax driver (3.1). We also have
>> OpsCenter running (to be removed soon) via Thrift, if I remember correctly.
>>
>>
>>
>> As said, the write request latency for our keyspace hasn’t really
>> changed, so perhaps another one (system related, OpsCenter …?) is affected
>> or perhaps the JMX metric is reporting something differently now. J So
>> not a real issue for now hopefully, just popping up in our monitoring,
>> wondering what this may be.
>>
>>
>>
>> Regarding compression metadata memory usage drop. Right, storage engine
>> re-write could be a reason. Thanks.
>>
>>
>>
>> Still wondering about the GC/CPU increase.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> Thomas
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jeff Jirsa [mailto:jji...@gmail.com ]
>> *Sent:* Freitag, 15. September 2017 13:14
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)
>>
>>
>>
>> Most people find 3.0 slightly slower than 2.1. The only thing that really
>> stands out in your email is the huge change in 95% latency - that's
>> atypical. Are you using thrift or native 9042)?  The decrease in
>> compression metadata offheap usage is likely due to the increased storage
>> efficiency of the storage engine (see Cassandra-8099).
>>
>>
>>
>>
>> --
>>
>> Jeff Jirsa
>>
>>
>>
>>
>> On Sep 15, 2017, at 2:37 AM, Steinmaurer, Thomas <
>> thomas.steinmau...@dynatrace.com> wrote:
>>
>> Hello,
>>
>>
>>
>> we have a test (regression) environment hosted in AWS, which is used for
>> auto deploying our software on a daily basis and attach constant load
>> across all deployments. Basically to allow us to detect any regressions in
>> our software on a daily basis.
>>
>>
>>
>> On the Cassandra-side, this is single-node in AWS, m4.xlarge, EBS gp2, 8G
>> heap, CMS. The environment has also been upgraded from Cassandra 2.1.18 to
>> 3.0.14 at a certain point in time. Without running upgradesstables so far.
>> We have not made any additional JVM/GC configuration change when going from
>> 2.1.18 to 3.0.14 on our own, thus, any self-made configuration changes
>> (e.g. new gen heap size) for 2.1.18 are also in place with 3.0.14.
>>
>>
>>
>> What we see after a time-frame of ~ 7 days (so, e.g. should not be caused
>> by some sort of spiky compaction pattern) is an AVG increase in GC/CPU
>> (most likely correlating):
>>
>> · CPU: ~ 12% => ~ 17%
>>
>> · GC Suspension: ~ 1,7% => 3,29%
>>
>>
>>
>> In this environment not a big deal, but relatively we have a CPU increase
>> of ~ 50% (with increased GC most likely contributing). Something we have
>> deal with when going into production (going into larger, multi-node
>> loadtest environments first though).
>>
>>
>>
>> Beside the CPU/GC shift, we also monitor the following noticeable changes
>> (don’t know if they somehow correlate with the CPU

RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

[Steinmaurer, Thomas]

Hi,

in our experience CMS is doing much better with smaller heaps.
Regards,
Thomas


From: Matope Ono [mailto:matope@gmail.com]
Sent: Dienstag, 26. September 2017 10:58
To: user@cassandra.apache.org
Subject: Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Hi. We met similar situation after upgrading from 2.1.14 to 3.11 in our 
production.

Have you already tried G1GC instead of CMS? Our timeouts were mitigated after 
replacing CMS with G1GC.

Thanks.

2017-09-25 20:01 GMT+09:00 Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>>:
Hello,

I have now some concrete numbers from our 9 node loadtest cluster with constant 
load, same infrastructure after upgrading to 3.0.14 from 2.1.18.

We see doubled GC suspension time + correlating CPU increase. In short, 3.0.14 
is not able to handle the same load.

I have created https://issues.apache.org/jira/browse/CASSANDRA-13900. Feel free 
to request any further additional information on the ticket.

Unfortunately this is a real show-stopper for us upgrading to 3.0.

Thanks for your attention.

Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com]
Sent: Freitag, 15. September 2017 13:51
To: user@cassandra.apache.org
Subject: RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Hi Jeff,

we are using native (CQL3) via Java DataStax driver (3.1). We also have 
OpsCenter running (to be removed soon) via Thrift, if I remember correctly.

As said, the write request latency for our keyspace hasn’t really changed, so 
perhaps another one (system related, OpsCenter …?) is affected or perhaps the 
JMX metric is reporting something differently now. ☺ So not a real issue for 
now hopefully, just popping up in our monitoring, wondering what this may be.

Regarding compression metadata memory usage drop. Right, storage engine 
re-write could be a reason. Thanks.

Still wondering about the GC/CPU increase.

Thanks!

Thomas



From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Freitag, 15. September 2017 13:14
To: user@cassandra.apache.org
Subject: Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Most people find 3.0 slightly slower than 2.1. The only thing that really 
stands out in your email is the huge change in 95% latency - that's atypical. 
Are you using thrift or native 9042)?  The decrease in compression metadata 
offheap usage is likely due to the increased storage efficiency of the storage 
engine (see Cassandra-8099).


--
Jeff Jirsa


On Sep 15, 2017, at 2:37 AM, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

we have a test (regression) environment hosted in AWS, which is used for auto 
deploying our software on a daily basis and attach constant load across all 
deployments. Basically to allow us to detect any regressions in our software on 
a daily basis.

On the Cassandra-side, this is single-node in AWS, m4.xlarge, EBS gp2, 8G heap, 
CMS. The environment has also been upgraded from Cassandra 2.1.18 to 3.0.14 at 
a certain point in time. Without running upgradesstables so far. We have not 
made any additional JVM/GC configuration change when going from 2.1.18 to 
3.0.14 on our own, thus, any self-made configuration changes (e.g. new gen heap 
size) for 2.1.18 are also in place with 3.0.14.

What we see after a time-frame of ~ 7 days (so, e.g. should not be caused by 
some sort of spiky compaction pattern) is an AVG increase in GC/CPU (most 
likely correlating):

• CPU: ~ 12% => ~ 17%

• GC Suspension: ~ 1,7% => 3,29%

In this environment not a big deal, but relatively we have a CPU increase of ~ 
50% (with increased GC most likely contributing). Something we have deal with 
when going into production (going into larger, multi-node loadtest environments 
first though).

Beside the CPU/GC shift, we also monitor the following noticeable changes 
(don’t know if they somehow correlate with the CPU/GC shift above):

• Increased AVG Write Client Requests Latency (95th Percentile), 
org.apache.cassandra.metrics.ClientRequest.Latency.Write: 6,05ms => 29,2ms, but 
almost constant (no change in) write client request latency for our particular 
keyspace only, org.apache.cassandra.metrics.Keyspace.ruxitdb.WriteLatency

• Compression metadata memory usage drop, 
org.apache.cassandra.metrics.Keyspace.XXX. 
CompressionMetadataOffHeapMemoryUsed: ~218MB => ~105MB => Good or bad? Known?

I know, looks all a bit vague, but perhaps someone else has seen something 
similar when upgrading to 3.0.14 and can share their thoughts/ideas. Especially 
the (relative) CPU/GC increase is something we are curious about.

Thanks a lot.

Thomas
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose

RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

[Steinmaurer, Thomas]

Hi Alex,

we tested with larger new gen sizes up to ¼ of max heap, but m4.xlarge look 
like being to weak to deal with larger new gen. The result was that we then got 
much more GCInspector related logs, but perhaps we need to re-test.

Right, we are using batches extensively. Unlogged/non-atomic. We are aware of 
avoiding multi partition batches, if possible. For test purposes we built 
something into our application to switch a flag to move from multi partition 
batches to strictly single partition per batch. We have not seen any measurable 
high-level improvement (e.g. decreased CPU, GC suspension …) on the 
Cassandra-side with single partition batches. Naturally, this resulted in much 
more requests executed by our application against the Cassandra cluster, with 
the affect in our application/server, that we saw a significant GC/CPU increase 
on our server, caused by the DataStax driver due to executing now more requests 
by a factor of X. So, with no visible gain on the Cassandra-side, but impacting 
our application/server negatively, we don’t strictly execute single partition 
batches.

As said on the ticket (https://issues.apache.org/jira/browse/CASSANDRA-13900), 
anything except Cassandra binaries have been unchanged in our loadtest 
environment.


Thanks,
Thomas



From: Alexander Dejanovski [mailto:a...@thelastpickle.com]
Sent: Dienstag, 26. September 2017 11:14
To: user@cassandra.apache.org
Subject: Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Hi Thomas,

I wouldn't move to G1GC with small heaps (<24GB) but just looking at your 
ticket I think that your new gen is way too small.
I get that it worked better in 2.1 in your case though, which would suggest 
that the memory footprint is different between 2.1 and 3.0. It looks like 
you're using batches extensively.
Hopefully you're aware that multi partition batches are discouraged because 
they indeed create heap pressure and high coordination costs (on top of 
batchlog writes/deletions), leading to more GC pauses.
With a 400MB new gen, you're very likely to have a lot of premature promotions 
(especially with the default max tenuring threshold), which will fill the old 
gen faster than necessary and is likely to trigger major GCs.

I'd suggest you re-run those tests with a 2GB new gen and compare results. Know 
that with Cassandra you can easily go up to 40%-50% of your heap for the new 
gen.

Cheers,


On Tue, Sep 26, 2017 at 10:58 AM Matope Ono 
mailto:matope@gmail.com>> wrote:
Hi. We met similar situation after upgrading from 2.1.14 to 3.11 in our 
production.

Have you already tried G1GC instead of CMS? Our timeouts were mitigated after 
replacing CMS with G1GC.

Thanks.

2017-09-25 20:01 GMT+09:00 Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>>:
Hello,

I have now some concrete numbers from our 9 node loadtest cluster with constant 
load, same infrastructure after upgrading to 3.0.14 from 2.1.18.

We see doubled GC suspension time + correlating CPU increase. In short, 3.0.14 
is not able to handle the same load.

I have created https://issues.apache.org/jira/browse/CASSANDRA-13900. Feel free 
to request any further additional information on the ticket.

Unfortunately this is a real show-stopper for us upgrading to 3.0.

Thanks for your attention.

Thomas

From: Steinmaurer, Thomas 
[mailto:thomas.steinmau...@dynatrace.com]
Sent: Freitag, 15. September 2017 13:51
To: user@cassandra.apache.org
Subject: RE: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Hi Jeff,

we are using native (CQL3) via Java DataStax driver (3.1). We also have 
OpsCenter running (to be removed soon) via Thrift, if I remember correctly.

As said, the write request latency for our keyspace hasn’t really changed, so 
perhaps another one (system related, OpsCenter …?) is affected or perhaps the 
JMX metric is reporting something differently now. ☺ So not a real issue for 
now hopefully, just popping up in our monitoring, wondering what this may be.

Regarding compression metadata memory usage drop. Right, storage engine 
re-write could be a reason. Thanks.

Still wondering about the GC/CPU increase.

Thanks!

Thomas



From: Jeff Jirsa [mailto:jji...@gmail.com]
Sent: Freitag, 15. September 2017 13:14
To: user@cassandra.apache.org
Subject: Re: GC/CPU increase after upgrading to 3.0.14 (from 2.1.18)

Most people find 3.0 slightly slower than 2.1. The only thing that really 
stands out in your email is the huge change in 95% latency - that's atypical. 
Are you using thrift or native 9042)?  The decrease in compression metadata 
offheap usage is likely due to the increased storage efficiency of the storage 
engine (see Cassandra-8099).


--
Jeff Jirsa


On Sep 15, 2017, at 2:37 AM, Steinmaurer, Thomas 
mailto:thomas.steinmau...@dynatrace.com>> 
wrote:
Hello,

we have a test (regression) environment hosted in AWS, which is used f

Re: User Defined Compaction Issue

[Jeff Jirsa]

Write row A, flush into sstable 1
Delete row A, flush the tombstone into sstable 2

The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets 
removed. If you just keep recompacting sstable 2 by itself, the row in sstable 
A remains on disk.



-- 
Jeff Jirsa


> On Sep 26, 2017, at 2:01 AM, shalom sagges  wrote:
> 
> Thanks Jeff!
> 
> I'll try that. 
> I'm not sure I understand how the tombstones are covering data in another 
> file. Do you have a small example perhaps?
> 
> Thanks again!
> 
>> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:
>> The problem is likely that your sstables overlap - your 91% droppable 
>> tombstones are probably covering data in another file. Until that data is 
>> removed, those tombstones cant be dropped.
>> 
>> This is why a full major compaction often works better for simply reclaiming 
>> disk space (though it takes a lot more disk as it runs and has other 
>> drawbacks) - it joins the shadowed data with the tombstones into a single 
>> file. You can sorta fake that by using user-defined compaction with multiple 
>> sstables to try to compact away the shadowed data - pick 2 or more -Data 
>> files and compact them together. Works best if you can be sure which data is 
>> overlapping, but short of that you'll probably want to pick data with 
>> approximately the same (or older) calendar timestamps.
>> 
>> 
>> 
>>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges  
>>> wrote:
>>> Hi Everyone, 
>>> 
>>> I'm running into an issue I can't seem to Solve. 
>>> 
>>> I execute force compaction in order to reclaim back storage. 
>>> Everything was working fine for a time, but after a while I found that 
>>> tombstones aren't being removed any longer. 
>>> 
>>> For example, I've compacted the following SSTable:
>>> 21G Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I ran it with jmxterm and the compaction started and completed 
>>> successfully. 
>>>  $>run -b org.apache.cassandra.db:type=CompactionManager 
>>> forceUserDefinedCompaction file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean 
>>> org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns: 
>>> null
>>> 
>>> 
>>> A new SStable was created, but no tombstones were removed. 
>>>  21G Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones: 0.9115601492568154
>>> 
>>> I've tried running it from jconsole as well, but the outcome was the same. 
>>> 
>>> Is anyone familiar with this issue? 
>>> Btw, this cluster is on version 2.0.14 . 
>>> 
>>> 
>>> Thanks!
>>> Shalom
>>> 
>> 
> 

Re: Datastax Driver Mapper & Secondary Indexes

[Daniel Hölbling-Inzko]

Hi, I also just figured out that there is no schema generation off the
mapper.
Thanks for pointing me to the secondary index info. I'll have a look.

greetings Daniel

On Tue, 26 Sep 2017 at 09:42 kurt greaves  wrote:

> If you've created a secondary index you simply query it by specifying it
> as part of the where clause. Note that you should really understand the
> drawbacks of secondary indexes before using them, as they might not be
> incredibly efficient depending on what you need them for.
> http://www.wentnet.com/blog/?p=77 and
> https://pantheon.io/blog/cassandra-scale-problem-secondary-indexes might
> help.
>
> On 26 September 2017 at 07:17, Daniel Hölbling-Inzko <
> daniel.hoelbling-in...@bitmovin.com> wrote:
>
>> Hi,
>> I am currently moving an application from SQL to Cassandra using Java. I
>> successfully got the DataStax driver and the mapper up and running, but
>> can't seem to figure out how to set secondary indexes through the mapper.
>> I also can't seem to find anything related to indexes in the mapper
>> sources - am I missing something or is this missing from the client library?
>>
>> greetings Daniel
>>
>
>

Re: User Defined Compaction Issue

[shalom sagges]

Awesome explanation :-)

Thanks a lot!

On Tue, Sep 26, 2017 at 3:40 PM, Jeff Jirsa  wrote:

> Write row A, flush into sstable 1
> Delete row A, flush the tombstone into sstable 2
>
> The tombstone in sstable 2 can’t be removed until row A in sstable 1 gets
> removed. If you just keep recompacting sstable 2 by itself, the row in
> sstable A remains on disk.
>
>
>
> --
> Jeff Jirsa
>
>
> On Sep 26, 2017, at 2:01 AM, shalom sagges  wrote:
>
> Thanks Jeff!
>
> I'll try that.
> I'm not sure I understand how the tombstones are covering data in another
> file. Do you have a small example perhaps?
>
> Thanks again!
>
> On Tue, Sep 26, 2017 at 1:38 AM, Jeff Jirsa  wrote:
>
>> The problem is likely that your sstables overlap - your 91% droppable
>> tombstones are probably covering data in another file. Until that data is
>> removed, those tombstones cant be dropped.
>>
>> This is why a full major compaction often works better for simply
>> reclaiming disk space (though it takes a lot more disk as it runs and has
>> other drawbacks) - it joins the shadowed data with the tombstones into a
>> single file. You can sorta fake that by using user-defined compaction with
>> multiple sstables to try to compact away the shadowed data - pick 2 or more
>> -Data files and compact them together. Works best if you can be sure which
>> data is overlapping, but short of that you'll probably want to pick data
>> with approximately the same (or older) calendar timestamps.
>>
>>
>>
>> On Mon, Sep 25, 2017 at 11:10 AM, shalom sagges 
>> wrote:
>>
>>> Hi Everyone,
>>>
>>> I'm running into an issue I can't seem to Solve.
>>>
>>> I execute force compaction in order to reclaim back storage.
>>> Everything was working fine for a time, but after a while I found that
>>> tombstones aren't being removed any longer.
>>>
>>> For example, I've compacted the following SSTable:
>>> *21G *Sep 19 10:18 file-jb-69103-Data.db
>>> Estimated droppable tombstones:
>>>
>>> *0.9115601492568154*
>>> I ran it with jmxterm and the compaction started and completed
>>> successfully.
>>>  $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction file-jb-69103-Data.db
>>>  #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>>  #operation returns:
>>> null
>>>
>>>
>>> A new SStable was created, but no tombstones were removed.
>>>  *21G *Sep 25 12:16 file-jb-71319-Data.db
>>>  Estimated droppable tombstones:
>>>
>>> * 0.9115601492568154*
>>> I've tried running it from jconsole as well, but the outcome was the
>>> same.
>>>
>>> Is anyone familiar with this issue?
>>> Btw, this cluster is on version 2.0.14 .
>>>
>>>
>>> Thanks!
>>> Shalom
>>>
>>>
>>
>

Re: Datastax Driver Mapper & Secondary Indexes

[DuyHai Doan]

If you're looking for schema generation from Bean annotations:
https://github.com/doanduyhai/Achilles/wiki/DDL-Scripts-Generation

On Tue, Sep 26, 2017 at 2:50 PM, Daniel Hölbling-Inzko <
daniel.hoelbling-in...@bitmovin.com> wrote:

> Hi, I also just figured out that there is no schema generation off the
> mapper.
> Thanks for pointing me to the secondary index info. I'll have a look.
>
> greetings Daniel
>
> On Tue, 26 Sep 2017 at 09:42 kurt greaves  wrote:
>
>> If you've created a secondary index you simply query it by specifying it
>> as part of the where clause. Note that you should really understand the
>> drawbacks of secondary indexes before using them, as they might not be
>> incredibly efficient depending on what you need them for.
>> http://www.wentnet.com/blog/?p=77 and https://pantheon.io/blog/
>> cassandra-scale-problem-secondary-indexes might help.
>>
>> On 26 September 2017 at 07:17, Daniel Hölbling-Inzko <
>> daniel.hoelbling-in...@bitmovin.com> wrote:
>>
>>> Hi,
>>> I am currently moving an application from SQL to Cassandra using Java. I
>>> successfully got the DataStax driver and the mapper up and running, but
>>> can't seem to figure out how to set secondary indexes through the mapper.
>>> I also can't seem to find anything related to indexes in the mapper
>>> sources - am I missing something or is this missing from the client library?
>>>
>>> greetings Daniel
>>>
>>
>>

Poor Performance of Cassandra UDF/UDA

[Xin Jin]

Hi All,

I am new to the Cassandra community and  thank you in advance for your kindly 
comments on an issue we met recently.

We have found that running query with direct UDF execution is ten time more 
faster than the async UDF execution. The in-line comment: "Using async UDF 
execution is expensive (adds about 100us overhead per invocation on a Core-i7 
MBPr)” 
https://insight.io/github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/UDFunction.java?line=293
 show that this is a known behavior.  My questions are as below:

1. What are the main pros and cons of these two methods? Can I find any 
documents that discuss this?

2. Are there any plans to improve the performance of using async UDF? A simple 
way come to my mind is to use some sort of batch method, e.g., replace current 
row by row method with some rows by some rows. Are there any concerns on this?

3. How people solve this performance issue in general? It seems this 
performance issue is not an urgent or an important issue to solve because it is 
known and it is still there. Therefore people must have some sort of good 
solution solving this issue.

4. Can anyone share your experience on a solution for this?

I really appreciate your comments in advance.

Best regards,

Xin

nodetool cleanup in parallel

[Peng Xiao]

hi,


nodetool cleanup will only remove those keys which no longer belong to those 
nodes,than theoretically we can run nodetool cleanup in parallel,right?the 
document suggests us to run this one by one,but it's too slow.


Thanks,
Peng Xiao

Re: nodetool cleanup in parallel

[kurt greaves]

correct. you can run it in parallel across many nodes if you have capacity.
generally see about a 10% CPU increase from cleanups which isn't a big deal
if you have the capacity to handle it + the io.

on that note on later versions you can specify -j  to run
multiple cleanup compactions at the same time on a single node, and also
increase compaction throughput to speed the process up.

On 27 Sep. 2017 13:20, "Peng Xiao" <2535...@qq.com> wrote:

hi,

nodetool cleanup will only remove those keys which no longer belong to
those nodes,than theoretically we can run nodetool cleanup in
parallel,right?the document suggests us to run this one by one,but it's too
slow.

Thanks,
Peng Xiao

?????? nodetool cleanup in parallel

[Peng Xiao]

Thanks Kurt.




--  --
??: "kurt";;
: 2017??9??27??(??) 11:57
??: "User";

: Re: nodetool cleanup in parallel



correct. you can run it in parallel across many nodes if you have capacity. 
generally see about a 10% CPU increase from cleanups which isn't a big deal if 
you have the capacity to handle it + the io.

on that note on later versions you can specify -j  to run multiple 
cleanup compactions at the same time on a single node, and also increase 
compaction throughput to speed the process up.


On 27 Sep. 2017 13:20, "Peng Xiao" <2535...@qq.com> wrote:
hi,


nodetool cleanup will only remove those keys which no longer belong to those 
nodes,than theoretically we can run nodetool cleanup in parallel,right?the 
document suggests us to run this one by one,but it's too slow.


Thanks,
Peng Xiao

RE: 回复： nodetool cleanup in parallel

[Steinmaurer, Thomas]

Side-note: At least with 2.1 (or even later), be aware that you might run into 
the following issue:
https://issues.apache.org/jira/browse/CASSANDRA-11155

We are doing cron―job based hourly snapshots in production and have tried to 
also run cleanup after extending a cluster from 6 to 9 nodes. This resulted in 
snapshot creation getting stuck, so we gave up running cleanup (and reclaiming 
disk space) in favor of still having actual snapshots in place.

We might find a time window where we disable snapshots, but cleanup may take 
while depending on the data volume, thus this would mean possibly disabling 
snapshots for many hours.

Regards,
Thomas


From: Peng Xiao [mailto:2535...@qq.com]
Sent: Mittwoch, 27. September 2017 06:25
To: user 
Subject: 回复： nodetool cleanup in parallel

Thanks Kurt.


-- 原始邮件 --
发件人: "kurt";mailto:k...@instaclustr.com>>;
发送时间: 2017年9月27日(星期三) 中午11:57
收件人: "User"mailto:user@cassandra.apache.org>>;
主题: Re: nodetool cleanup in parallel

correct. you can run it in parallel across many nodes if you have capacity. 
generally see about a 10% CPU increase from cleanups which isn't a big deal if 
you have the capacity to handle it + the io.

on that note on later versions you can specify -j  to run multiple 
cleanup compactions at the same time on a single node, and also increase 
compaction throughput to speed the process up.

On 27 Sep. 2017 13:20, "Peng Xiao" <2535...@qq.com> 
wrote:
hi,

nodetool cleanup will only remove those keys which no longer belong to those 
nodes,than theoretically we can run nodetool cleanup in parallel,right?the 
document suggests us to run this one by one,but it's too slow.

Thanks,
Peng Xiao

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freist?dterstra?e 313

Re: Datastax Driver Mapper & Secondary Indexes

[Nicolas Henneaux]

If you just use the Cassandra default mapper, you can also use the class 
defined in the Stack Overflow answer 
https://stackoverflow.com/a/45073242/1630604 to generate the DDL from your 
entities.

Nicolas HENNEAUX


From: DuyHai Doan 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, 26 September 2017 at 15:17
To: user 
Subject: Re: Datastax Driver Mapper & Secondary Indexes

If you're looking for schema generation from Bean annotations: 
https://github.com/doanduyhai/Achilles/wiki/DDL-Scripts-Generation

On Tue, Sep 26, 2017 at 2:50 PM, Daniel Hölbling-Inzko 
mailto:daniel.hoelbling-in...@bitmovin.com>>
 wrote:
Hi, I also just figured out that there is no schema generation off the mapper.
Thanks for pointing me to the secondary index info. I'll have a look.

greetings Daniel

On Tue, 26 Sep 2017 at 09:42 kurt greaves 
mailto:k...@instaclustr.com>> wrote:
If you've created a secondary index you simply query it by specifying it as 
part of the where clause. Note that you should really understand the drawbacks 
of secondary indexes before using them, as they might not be incredibly 
efficient depending on what you need them for.
http://www.wentnet.com/blog/?p=77 and 
https://pantheon.io/blog/cassandra-scale-problem-secondary-indexes might help.

On 26 September 2017 at 07:17, Daniel Hölbling-Inzko 
mailto:daniel.hoelbling-in...@bitmovin.com>>
 wrote:
Hi,
I am currently moving an application from SQL to Cassandra using Java. I 
successfully got the DataStax driver and the mapper up and running, but can't 
seem to figure out how to set secondary indexes through the mapper.
I also can't seem to find anything related to indexes in the mapper sources - 
am I missing something or is this missing from the client library?

greetings Daniel