Cassandra Poor Read Performance Response Time

[_ _]

Hi

Currently i am running a cassandra cluster of 3 nodes (with it replicating to 
both nodes) and am experiencing poor performance, usually getting second 
response times when running queries when i am expecting/needing millisecond 
response times. Currently i have a table which looks like:

CREATE TABLE tracker.all_ad_impressions_counter_1d (
time_bucket bigint,
ad_id text,
uc text,
count counter,
PRIMARY KEY ((time_bucket, ad_id), uc)
) WITH CLUSTERING ORDER BY (uc ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'base_time_seconds': '3600', 'class': 
'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy', 
'max_sstable_age_days': '30', 'max_threshold': '32', 'min_threshold': '4', 
'timestamp_resolution': 'MILLISECONDS'}
AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';


and queries which look like:

SELECT
time_bucket,
uc,
count
FROM
all_ad_impressions_counter_1d

WHERE ad_id = ?
AND time_bucket = ?

the cluster is running on servers with 16 GB RAM, and 4 CPU cores and 3 100GB 
datastores, the storage is not local and these VMs are being managed through 
openstack. There are roughly 200 million records being written per day (1 
time_bucket) and maybe a few thousand records per partition (time_bucket, 
ad_id) at most. The amount of writes is not having a significant effect on our 
read performance as when writes are stopped, the read response time does not 
improve noticeably. I have attached a trace of one query i ran which took 
around 3 seconds which i would expect to take well below a second. I have also 
included the cassandra.yaml file and jvm options file. We do intend to change 
the storage to local storage and expect this will have a significant impact but 
i was wondering if there's anything else which could be changed which will also 
have a significant impact on read performance?

Thanks
Ian



cassandra.yaml
Description: cassandra.yaml


jvm_options.sh
Description: jvm_options.sh


Query 2.xlsx
Description: Query 2.xlsx

Re: Does securing C*'s CQL native interface (running on port 9042) automatically secure its Thrift API interface (running on port 9160)?

[Sam Tunnicliffe]

It does, yes. Clients will be required to call the thrift login method with
a valid set of credentials before performing any other RPC calls.
btw, in versions of C* >= 2.2 the Thrift server is not enabled by default
(CASSANDRA-9319).

On Mon, Oct 31, 2016 at 4:50 PM, Li, Guangxing 
wrote:

> Hi,
>
> I secured my C* cluster by having "authenticator:
> org.apache.cassandra.auth.PasswordAuthenticator" in cassandra.yaml. I
> know it secures the CQL native interface running on port 9042 because my
> code uses such interface. Does this also secure the Thrift API interface
> running on port 9160? I searched around the web for answers but could not
> find any. I supposed I can write a sample application using Thrift API
> interface to confirm it, but wondering if I can get a quick answer from you
> experts.
>
> Thanks.
>
> George.
>

Re: Cassandra Poor Read Performance Response Time

[Kant Kodali]

+1

On Tue, Nov 1, 2016 at 2:10 AM, _ _  wrote:

> Hi
>
> Currently i am running a cassandra cluster of 3 nodes (with it replicating
> to both nodes) and am experiencing poor performance, usually getting second
> response times when running queries when i am expecting/needing millisecond
> response times. Currently i have a table which looks like:
>
> CREATE TABLE tracker.all_ad_impressions_counter_1d (
> time_bucket bigint,
> ad_id text,
> uc text,
> count counter,
> PRIMARY KEY ((time_bucket, ad_id), uc)
> ) WITH CLUSTERING ORDER BY (uc ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'base_time_seconds': '3600', 'class':
> 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy',
> 'max_sstable_age_days': '30', 'max_threshold': '32', 'min_threshold': '4',
> 'timestamp_resolution': 'MILLISECONDS'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
>
> and queries which look like:
>
> SELECT
> time_bucket,
> uc,
> count
> FROM
> all_ad_impressions_counter_1d
>
> WHERE ad_id = ?
> AND time_bucket = ?
>
> the cluster is running on servers with 16 GB RAM, and 4 CPU cores and 3
> 100GB datastores, the storage is not local and these VMs are being managed
> through openstack. There are roughly 200 million records being written per
> day (1 time_bucket) and maybe a few thousand records per partition
> (time_bucket, ad_id) at most. The amount of writes is not having a
> significant effect on our read performance as when writes are stopped, the
> read response time does not improve noticeably. I have attached a trace of
> one query i ran which took around 3 seconds which i would expect to take
> well below a second. I have also included the cassandra.yaml file and jvm
> options file. We do intend to change the storage to local storage and
> expect this will have a significant impact but i was wondering if there's
> anything else which could be changed which will also have a significant
> impact on read performance?
>
> Thanks
> Ian
>
>

Re: Incremental repairs leading to unrepaired data

[Stefano Ortolani]

That is not happening anymore since I am repairing a keyspace with
much less data (the other one is still there in write-only mode).
The command I am using is the most boring (even shed the -pr option so
to keep anticompactions to a minimum): nodetool -h localhost repair

It's executed sequentially on each node (no overlapping, next node
waits for the previous to complete).

Regards,
Stefano Ortolani

On Mon, Oct 31, 2016 at 11:18 PM, kurt Greaves  wrote:
> Blowing out to 1k SSTables seems a bit full on. What args are you passing to
> repair?
>
> Kurt Greaves
> k...@instaclustr.com
> www.instaclustr.com
>
> On 31 October 2016 at 09:49, Stefano Ortolani  wrote:
>>
>> I've collected some more data-points, and I still see dropped
>> mutations with compaction_throughput_mb_per_sec set to 8.
>> The only notable thing regarding the current setup is that I have
>> another keyspace (not being repaired though) with really wide rows
>> (100MB per partition), but that shouldn't have any impact in theory.
>> Nodes do not seem that overloaded either and don't see any GC spikes
>> while those mutations are dropped :/
>>
>> Hitting a dead end here, any further idea where to look for further ideas?
>>
>> Regards,
>> Stefano
>>
>> On Wed, Aug 10, 2016 at 12:41 PM, Stefano Ortolani 
>> wrote:
>> > That's what I was thinking. Maybe GC pressure?
>> > Some more details: during anticompaction I have some CFs exploding to 1K
>> > SStables (to be back to ~200 upon completion).
>> > HW specs should be quite good (12 cores/32 GB ram) but, I admit, still
>> > relying on spinning disks, with ~150GB per node.
>> > Current version is 3.0.8.
>> >
>> >
>> > On Wed, Aug 10, 2016 at 12:36 PM, Paulo Motta 
>> > wrote:
>> >>
>> >> That's pretty low already, but perhaps you should lower to see if it
>> >> will
>> >> improve the dropped mutations during anti-compaction (even if it
>> >> increases
>> >> repair time), otherwise the problem might be somewhere else. Generally
>> >> dropped mutations is a signal of cluster overload, so if there's
>> >> nothing
>> >> else wrong perhaps you need to increase your capacity. What version are
>> >> you
>> >> in?
>> >>
>> >> 2016-08-10 8:21 GMT-03:00 Stefano Ortolani :
>> >>>
>> >>> Not yet. Right now I have it set at 16.
>> >>> Would halving it more or less double the repair time?
>> >>>
>> >>> On Tue, Aug 9, 2016 at 7:58 PM, Paulo Motta 
>> >>> wrote:
>> 
>>  Anticompaction throttling can be done by setting the usual
>>  compaction_throughput_mb_per_sec knob on cassandra.yaml or via
>>  nodetool
>>  setcompactionthroughput. Did you try lowering that  and checking if
>>  that
>>  improves the dropped mutations?
>> 
>>  2016-08-09 13:32 GMT-03:00 Stefano Ortolani :
>> >
>> > Hi all,
>> >
>> > I am running incremental repaird on a weekly basis (can't do it
>> > every
>> > day as one single run takes 36 hours), and every time, I have at
>> > least one
>> > node dropping mutations as part of the process (this almost always
>> > during
>> > the anticompaction phase). Ironically this leads to a system where
>> > repairing
>> > makes data consistent at the cost of making some other data not
>> > consistent.
>> >
>> > Does anybody know why this is happening?
>> >
>> > My feeling is that this might be caused by anticompacting column
>> > families with really wide rows and with many SStables. If that is
>> > the case,
>> > any way I can throttle that?
>> >
>> > Thanks!
>> > Stefano
>> 
>> 
>> >>>
>> >>
>> >
>
>

Specifying multiple conditions for lightweight conditions?

[Ali Akhtar]

In the following query:

UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200'
WHERE id = '1'
IF last_due_at < '2013-01-01 00:00:00+0200';

The intent is to change the value of 'last_due_at' as long as 'last_due_at'
isn't already set to a later date than the one I've supplied.

The problem is, last_due_at starts off with an initial value of null, so
the above query fails.

If I try the following:


UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200'
WHERE id = '1'
IF last_due_at < '2013-01-01 00:00:00+0200' OR last_due_at = null;

That fails due to a syntax error.

Is there any other way to achieve this?

Cannot mix counter and non counter columns in the same table

[Ali Akhtar]

In the documentation for counters:

https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html

The example table is created via:

CREATE TABLE counterks.page_view_counts
  (counter_value counter,
  url_name varchar,
  page_name varchar,
  PRIMARY KEY (url_name, page_name)
);

Yet if I try to create a table with a mixture of texts, ints, timestamps,
and counters, i get the error ' Cannot mix counter and non counter columns
in the same table'

Is that supposed to be allowed or not allowed, given that the official
example contains a mix of counters and non-counters?

Re: Cannot mix counter and non counter columns in the same table

[Cody Yancey]

For counter tables, non-counter types are of course allowed in the primary
key. Counters would be meaningless otherwise.

Thanks,
Cody

On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:

> In the documentation for counters:
>
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>
> The example table is created via:
>
> CREATE TABLE counterks.page_view_counts
>   (counter_value counter,
>   url_name varchar,
>   page_name varchar,
>   PRIMARY KEY (url_name, page_name)
> );
>
> Yet if I try to create a table with a mixture of texts, ints, timestamps,
> and counters, i get the error ' Cannot mix counter and non counter columns
> in the same table'
>
> Is that supposed to be allowed or not allowed, given that the official
> example contains a mix of counters and non-counters?
>

Re: Cannot mix counter and non counter columns in the same table

[Ali Akhtar]

I'm not referring to the primary key, just to other columns.

My primary key is a text, and my table contains a mix of texts, ints, and
timestamps.

If I try to change one of the ints to a counter and run the create table
query, I get the error ' Cannot mix counter and non counter columns in the
same table'


On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:

> For counter tables, non-counter types are of course allowed in the primary
> key. Counters would be meaningless otherwise.
>
> Thanks,
> Cody
>
> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>
>> In the documentation for counters:
>>
>> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>>
>> The example table is created via:
>>
>> CREATE TABLE counterks.page_view_counts
>>   (counter_value counter,
>>   url_name varchar,
>>   page_name varchar,
>>   PRIMARY KEY (url_name, page_name)
>> );
>>
>> Yet if I try to create a table with a mixture of texts, ints, timestamps,
>> and counters, i get the error ' Cannot mix counter and non counter columns
>> in the same table'
>>
>> Is that supposed to be allowed or not allowed, given that the official
>> example contains a mix of counters and non-counters?
>>
>

Re: Cannot mix counter and non counter columns in the same table

[Cody Yancey]

In your table schema, you have KEYS and you have VALUES. Your KEYS are
text, but they could be any non-counter type or compound thereof. KEYS
obviously cannot ever be counters.

Your VALUES, however, must be either all counters or all non-counters. The
official example you posted conforms to this limitation.

Thanks,
Cody

On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:

> I'm not referring to the primary key, just to other columns.
>
> My primary key is a text, and my table contains a mix of texts, ints, and
> timestamps.
>
> If I try to change one of the ints to a counter and run the create table
> query, I get the error ' Cannot mix counter and non counter columns in
> the same table'
>
>
> On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:
>
>> For counter tables, non-counter types are of course allowed in the
>> primary key. Counters would be meaningless otherwise.
>>
>> Thanks,
>> Cody
>>
>> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>>
>>> In the documentation for counters:
>>>
>>> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>>>
>>> The example table is created via:
>>>
>>> CREATE TABLE counterks.page_view_counts
>>>   (counter_value counter,
>>>   url_name varchar,
>>>   page_name varchar,
>>>   PRIMARY KEY (url_name, page_name)
>>> );
>>>
>>> Yet if I try to create a table with a mixture of texts, ints,
>>> timestamps, and counters, i get the error ' Cannot mix counter and non
>>> counter columns in the same table'
>>>
>>> Is that supposed to be allowed or not allowed, given that the official
>>> example contains a mix of counters and non-counters?
>>>
>>
>

Re: Cannot mix counter and non counter columns in the same table

[Ali Akhtar]

That's a terrible gotcha rule.

On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:

> In your table schema, you have KEYS and you have VALUES. Your KEYS are
> text, but they could be any non-counter type or compound thereof. KEYS
> obviously cannot ever be counters.
>
> Your VALUES, however, must be either all counters or all non-counters. The
> official example you posted conforms to this limitation.
>
> Thanks,
> Cody
>
> On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:
>
>> I'm not referring to the primary key, just to other columns.
>>
>> My primary key is a text, and my table contains a mix of texts, ints, and
>> timestamps.
>>
>> If I try to change one of the ints to a counter and run the create table
>> query, I get the error ' Cannot mix counter and non counter columns in
>> the same table'
>>
>>
>> On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:
>>
>>> For counter tables, non-counter types are of course allowed in the
>>> primary key. Counters would be meaningless otherwise.
>>>
>>> Thanks,
>>> Cody
>>>
>>> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>>>
 In the documentation for counters:

 https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html

 The example table is created via:

 CREATE TABLE counterks.page_view_counts
   (counter_value counter,
   url_name varchar,
   page_name varchar,
   PRIMARY KEY (url_name, page_name)
 );

 Yet if I try to create a table with a mixture of texts, ints,
 timestamps, and counters, i get the error ' Cannot mix counter and non
 counter columns in the same table'

 Is that supposed to be allowed or not allowed, given that the official
 example contains a mix of counters and non-counters?

>>>
>>

Re: Cassandra reaper

[Alexander Dejanovski]

Hi,

The first step in using reaper is to add a cluster to it, as it is a tool
that can manage multiple clusters and does not need to be executed on a
Cassandra node (you can run in on any edge node you want).

You should run : ./bin/spreaper add-cluster 127.0.0.1
Where you'll replace 127.0.0.1 by the address of one of the nodes of your
cluster.

Then you can run : ./bin/spreaper cluster_name keyspace_name
to start repairing a keyspace.

You might want to drop in the UI made by Stefan Podkowinski which might
ease things up for you, at least at the beginning :
https://github.com/spodkowinski/cassandra-reaper-ui

Worth mentioning that at The Last Pickle we maintain a fork of Reaper that
handles incremental repair, works with C* 2.x and 3.0, and bundles the UI :
https://github.com/thelastpickle/cassandra-reaper
We have a branch that allows using Cassandra as a storage backend instead
of Postgres :
https://github.com/thelastpickle/cassandra-reaper/tree/add-cassandra-storage
It should be merged to master really soon and should be ready to use.

Cheers,


On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> Has anyone played around with the cassandra reaper (
> https://github.com/spotify/cassandra-reaper)?
>
> if so can some please help me with the set-up, I can't get it working. I
> used the below steps:
>
> 1. create jar file using maven
> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
> cassandra-reaper.yaml
> 3. ./bin/spreaper repair production users
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Specifying multiple conditions for lightweight conditions?

[Vladimir Yudovin]

Hi,



Unfortunately CQL syntax doesn't allow use of OR operator in condition list: 



UPDATE [ keyspace_name. ] table_name [ USING TTL time_value | USING TIMESTAMP 
timestamp_value ] SET assignment [ , assignment ] . . . WHERE row_specification 
[ IF EXISTS | IF NOT EXISTS | IF condition [ AND condition ] . . . ] ;



The simplest solution to make two different request, first with null check:



UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200' WHERE id = '1' IF 
last_due_at = null;

then test result, if applied = False make second request:



UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200' WHERE id = '1' IF 
last_due_at < '2013-01-01 00:00:00+0200'



Sure, it's less effective then OR condition. Probably you can use IF NOT EXISTS 
in first request (depending on your application logic), may be it will be 
slightly faster (not sure).





Best regards, Vladimir Yudovin, 

Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.





 On Tue, 01 Nov 2016 08:22:31 -0400Ali Akhtar  
wrote 




In the following query:



UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200'

WHERE id = '1' 

IF last_due_at < '2013-01-01 00:00:00+0200';




The intent is to change the value of 'last_due_at' as long as 'last_due_at' 
isn't already set to a later date than the one I've supplied.



The problem is, last_due_at starts off with an initial value of null, so the 
above query fails.



If I try the following:





UPDATE project SET last_due_at = '2013-01-01 00:00:00+0200'


WHERE id = '1' 

IF last_due_at < '2013-01-01 00:00:00+0200' OR last_due_at = null;




That fails due to a syntax error.



Is there any other way to achieve this?

Re: Unsubscribe

[SmartCat - Scott Hirleman]

That doesn't work. You need to send an email to user-unsubscribe@cassandra.
apache.org

On Sat, Oct 29, 2016 at 1:53 AM, dhanesh malviya  wrote:

>
>
> --
> Regards,
> Dhanesh Malviya..
>



-- 
*Scott Hirleman*
*Head of US Marketing and Sales*
www.smartcat.io
https://github.com/smartcat-labs 

RE: An extremely fast cassandra table full scan utility

[SEAN_R_DURITY]

In general, I recommend that full table scans are a bad use case for Cassandra. 
Another technology might be a better choice.


Sean Durity

From: Edward Capriolo [mailto:edlinuxg...@gmail.com]
Sent: Monday, October 03, 2016 4:38 PM
To: user@cassandra.apache.org
Subject: Re: An extremely fast cassandra table full scan utility

I undertook a similar effort a while ago.

https://issues.apache.org/jira/browse/CASSANDRA-7014

Other than the fact that it was closed with no comments, I can tell you that 
other efforts I had to embed things in Cassandra did not go swimmingly. 
Although at the time ideas were rejected like groovy udfs

On Mon, Oct 3, 2016 at 4:22 PM, Bhuvan Rawal 
mailto:bhu1ra...@gmail.com>> wrote:
Hi Jonathan,

If full scan is a regular requirement then setting up a spark cluster in 
locality with Cassandra nodes makes perfect sense. But supposing that it is a 
one off requirement, say a weekly or a fortnightly task, a spark cluster could 
be an added overhead with additional capacity, resource planning as far as 
operations / maintenance is concerned.

So this could be thought a simple substitute for a single threaded scan without 
additional efforts to setup and maintain another technology.

Regards,
Bhuvan

On Tue, Oct 4, 2016 at 1:37 AM, siddharth verma 
mailto:sidd.verma29.l...@gmail.com>> wrote:
Hi Jon,
It wan't allowed.
Moreover, if someone who isn't familiar with spark, and might be new to map 
filter reduce etc. operations, could also use the utility for some simple 
operations assuming a sequential scan of the cassandra table.

Regards
Siddharth Verma

On Tue, Oct 4, 2016 at 1:32 AM, Jonathan Haddad 
mailto:j...@jonhaddad.com>> wrote:
Couldn't set up as couldn't get it working, or its not allowed?

On Mon, Oct 3, 2016 at 3:23 PM Siddharth Verma 
mailto:verma.siddha...@snapdeal.com>> wrote:
Hi Jon,
We couldn't setup a spark cluster.

For some use case, a spark cluster was required, but for some reason we 
couldn't create spark cluster. Hence, one may use this utility to iterate 
through the entire table at very high speed.
Had to find a work around, that would be faster than paging on result set.
Regards




Siddharth Verma
Software Engineer I - CaMS

[http://i.sdlcdn.com/img/marketing-mailers/mailer/2015/signature_24mar/images/dilkideal_logo.png]


M: +91 9013689856, T: 011 22791596 EXT: 14697
CA2125, 2nd Floor, ASF Centre-A, Jwala Mill Road,
Udyog Vihar Phase - IV, Gurgaon-122016, INDIA


[http://i3.sdlcdn.com/img/homepage/03/sirertPlaceWrk2.png]

Download Our App

[A]

[A]

[W]







On Tue, Oct 4, 2016 at 12:41 AM, Jonathan Haddad 
mailto:j...@jonhaddad.com>> wrote:
It almost sounds like you're duplicating all the work of both spark and the 
connector. May I ask why you decided to not use the existing tools?

On Mon, Oct 3, 2016 at 2:21 PM siddharth verma 
mailto:sidd.verma29.l...@gmail.com>> wrote:
Hi DuyHai,
Thanks for your reply.
A few more features planned in the next one(if there is one) like,
custom policy keeping in mind the replication of token range on specific nodes,
fine graining the token range(for more speedup),
and a few more.

I think, as fine graining a token range,
If one token range is split further in say, 2-3 parts, divided among threads, 
this would exploit the possible parallelism on a large scaled out cluster.

And, as you mentioned the JIRA, streaming of request, that would of huge help 
with further splitting the range.

Thanks once again for your valuable comments. :-)

Regards,
Siddharth Verma







The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in a

Re: Cassandra reaper

[Jai Bheemsen Rao Dhanwada]

Thanks Alex,

Forgot to mention but I did add the cluster. See the status below. It says
the status is running but I don't see any repair happening. this is in the
same state from past 1 days.
b/w there not much of data in cluster.

[root@machine cassandra-reaper]#  ./bin/spreaper status-repair 3
# Report improvements/bugs at
https://github.com/spotify/cassandra-reaper/issues
#
--
# Repair run with id '3':
{
  "cause": "manual spreaper run",
  "cluster_name": "production",
  "column_families": [],
  "creation_time": "2016-11-01T00:39:15Z",
  "duration": null,
  "end_time": null,
  "estimated_time_of_arrival": null,
  "id": 3,
  "intensity": 0.900,
  "keyspace_name": "users",
*  "last_event": "no events",*
  "owner": "root",
  "pause_time": null,
  "repair_parallelism": "DATACENTER_AWARE",
  "segments_repaired": 0,
  "start_time": "2016-11-01T00:39:15Z",
*  "state": "RUNNING",*
  "total_segments": 301
}
[root@ machine cassandra-reaper]#

On Tue, Nov 1, 2016 at 9:24 AM, Alexander Dejanovski  wrote:

> Hi,
>
> The first step in using reaper is to add a cluster to it, as it is a tool
> that can manage multiple clusters and does not need to be executed on a
> Cassandra node (you can run in on any edge node you want).
>
> You should run : ./bin/spreaper add-cluster 127.0.0.1
> Where you'll replace 127.0.0.1 by the address of one of the nodes of your
> cluster.
>
> Then you can run : ./bin/spreaper cluster_name keyspace_name
> to start repairing a keyspace.
>
> You might want to drop in the UI made by Stefan Podkowinski which might
> ease things up for you, at least at the beginning : https://github.com/
> spodkowinski/cassandra-reaper-ui
>
> Worth mentioning that at The Last Pickle we maintain a fork of Reaper that
> handles incremental repair, works with C* 2.x and 3.0, and bundles the UI :
> https://github.com/thelastpickle/cassandra-reaper
> We have a branch that allows using Cassandra as a storage backend instead
> of Postgres : https://github.com/thelastpickle/cassandra-
> reaper/tree/add-cassandra-storage
> It should be merged to master really soon and should be ready to use.
>
> Cheers,
>
>
> On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello,
>>
>> Has anyone played around with the cassandra reaper (
>> https://github.com/spotify/cassandra-reaper)?
>>
>> if so can some please help me with the set-up, I can't get it working. I
>> used the below steps:
>>
>> 1. create jar file using maven
>> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
>> cassandra-reaper.yaml
>> 3. ./bin/spreaper repair production users
>>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Cassandra reaper

[Alexander Dejanovski]

Do you have anything in the reaper logs that would show a failure of some
sort ?
Also, can you tell me which version of Cassandra you're using ?

Thanks

On Tue, Nov 1, 2016 at 6:15 PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Alex,
>
> Forgot to mention but I did add the cluster. See the status below. It says
> the status is running but I don't see any repair happening. this is in the
> same state from past 1 days.
> b/w there not much of data in cluster.
>
> [root@machine cassandra-reaper]#  ./bin/spreaper status-repair 3
> # Report improvements/bugs at
> https://github.com/spotify/cassandra-reaper/issues
> #
> --
> # Repair run with id '3':
> {
>   "cause": "manual spreaper run",
>   "cluster_name": "production",
>   "column_families": [],
>   "creation_time": "2016-11-01T00:39:15Z",
>   "duration": null,
>   "end_time": null,
>   "estimated_time_of_arrival": null,
>   "id": 3,
>   "intensity": 0.900,
>   "keyspace_name": "users",
> *  "last_event": "no events",*
>   "owner": "root",
>   "pause_time": null,
>   "repair_parallelism": "DATACENTER_AWARE",
>   "segments_repaired": 0,
>   "start_time": "2016-11-01T00:39:15Z",
> *  "state": "RUNNING",*
>   "total_segments": 301
> }
> [root@ machine cassandra-reaper]#
>
> On Tue, Nov 1, 2016 at 9:24 AM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
>
> Hi,
>
> The first step in using reaper is to add a cluster to it, as it is a tool
> that can manage multiple clusters and does not need to be executed on a
> Cassandra node (you can run in on any edge node you want).
>
> You should run : ./bin/spreaper add-cluster 127.0.0.1
> Where you'll replace 127.0.0.1 by the address of one of the nodes of your
> cluster.
>
> Then you can run : ./bin/spreaper cluster_name keyspace_name
> to start repairing a keyspace.
>
> You might want to drop in the UI made by Stefan Podkowinski which might
> ease things up for you, at least at the beginning :
> https://github.com/spodkowinski/cassandra-reaper-ui
>
> Worth mentioning that at The Last Pickle we maintain a fork of Reaper that
> handles incremental repair, works with C* 2.x and 3.0, and bundles the UI :
> https://github.com/thelastpickle/cassandra-reaper
> We have a branch that allows using Cassandra as a storage backend instead
> of Postgres :
> https://github.com/thelastpickle/cassandra-reaper/tree/add-cassandra-storage
> It should be merged to master really soon and should be ready to use.
>
> Cheers,
>
>
> On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Hello,
>
> Has anyone played around with the cassandra reaper (
> https://github.com/spotify/cassandra-reaper)?
>
> if so can some please help me with the set-up, I can't get it working. I
> used the below steps:
>
> 1. create jar file using maven
> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
> cassandra-reaper.yaml
> 3. ./bin/spreaper repair production users
>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Cassandra reaper

[Jai Bheemsen Rao Dhanwada]

Cassandra version is 2.1.16

In my setup I don't see it is writting to any logs

On Tue, Nov 1, 2016 at 10:25 AM, Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> Do you have anything in the reaper logs that would show a failure of some
> sort ?
> Also, can you tell me which version of Cassandra you're using ?
>
> Thanks
>
> On Tue, Nov 1, 2016 at 6:15 PM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thanks Alex,
>>
>> Forgot to mention but I did add the cluster. See the status below. It
>> says the status is running but I don't see any repair happening. this is in
>> the same state from past 1 days.
>> b/w there not much of data in cluster.
>>
>> [root@machine cassandra-reaper]#  ./bin/spreaper status-repair 3
>> # Report improvements/bugs at https://github.com/spotify/
>> cassandra-reaper/issues
>> # 
>> --
>> # Repair run with id '3':
>> {
>>   "cause": "manual spreaper run",
>>   "cluster_name": "production",
>>   "column_families": [],
>>   "creation_time": "2016-11-01T00:39:15Z",
>>   "duration": null,
>>   "end_time": null,
>>   "estimated_time_of_arrival": null,
>>   "id": 3,
>>   "intensity": 0.900,
>>   "keyspace_name": "users",
>> *  "last_event": "no events",*
>>   "owner": "root",
>>   "pause_time": null,
>>   "repair_parallelism": "DATACENTER_AWARE",
>>   "segments_repaired": 0,
>>   "start_time": "2016-11-01T00:39:15Z",
>> *  "state": "RUNNING",*
>>   "total_segments": 301
>> }
>> [root@ machine cassandra-reaper]#
>>
>> On Tue, Nov 1, 2016 at 9:24 AM, Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>
>> Hi,
>>
>> The first step in using reaper is to add a cluster to it, as it is a tool
>> that can manage multiple clusters and does not need to be executed on a
>> Cassandra node (you can run in on any edge node you want).
>>
>> You should run : ./bin/spreaper add-cluster 127.0.0.1
>> Where you'll replace 127.0.0.1 by the address of one of the nodes of your
>> cluster.
>>
>> Then you can run : ./bin/spreaper cluster_name keyspace_name
>> to start repairing a keyspace.
>>
>> You might want to drop in the UI made by Stefan Podkowinski which might
>> ease things up for you, at least at the beginning : https://github.com/
>> spodkowinski/cassandra-reaper-ui
>>
>> Worth mentioning that at The Last Pickle we maintain a fork of Reaper
>> that handles incremental repair, works with C* 2.x and 3.0, and bundles the
>> UI : https://github.com/thelastpickle/cassandra-reaper
>> We have a branch that allows using Cassandra as a storage backend instead
>> of Postgres : https://github.com/thelastpickle/cassandra-
>> reaper/tree/add-cassandra-storage
>> It should be merged to master really soon and should be ready to use.
>>
>> Cheers,
>>
>>
>> On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>> Hello,
>>
>> Has anyone played around with the cassandra reaper (
>> https://github.com/spotify/cassandra-reaper)?
>>
>> if so can some please help me with the set-up, I can't get it working. I
>> used the below steps:
>>
>> 1. create jar file using maven
>> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
>> cassandra-reaper.yaml
>> 3. ./bin/spreaper repair production users
>>
>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Cannot mix counter and non counter columns in the same table

[Robert Wille]

I used to think it was terrible as well. But it really isn’t. Just put your 
non-counter columns in a separate table with the same primary key. If you want 
to query both counter and non-counter columns at the same time, just query both 
tables at the same time with asynchronous queries.

On Nov 1, 2016, at 7:29 AM, Ali Akhtar 
mailto:ali.rac...@gmail.com>> wrote:

That's a terrible gotcha rule.

On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey 
mailto:yan...@uber.com>> wrote:

In your table schema, you have KEYS and you have VALUES. Your KEYS are text, 
but they could be any non-counter type or compound thereof. KEYS obviously 
cannot ever be counters.

Your VALUES, however, must be either all counters or all non-counters. The 
official example you posted conforms to this limitation.

Thanks,
Cody

On Nov 1, 2016 7:16 AM, "Ali Akhtar" 
mailto:ali.rac...@gmail.com>> wrote:
I'm not referring to the primary key, just to other columns.

My primary key is a text, and my table contains a mix of texts, ints, and 
timestamps.

If I try to change one of the ints to a counter and run the create table query, 
I get the error ' Cannot mix counter and non counter columns in the same table'


On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey 
mailto:yan...@uber.com>> wrote:

For counter tables, non-counter types are of course allowed in the primary key. 
Counters would be meaningless otherwise.

Thanks,
Cody

On Nov 1, 2016 7:00 AM, "Ali Akhtar" 
mailto:ali.rac...@gmail.com>> wrote:
In the documentation for counters:

https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html

The example table is created via:

CREATE TABLE counterks.page_view_counts
  (counter_value counter,
  url_name varchar,
  page_name varchar,
  PRIMARY KEY (url_name, page_name)
);

Yet if I try to create a table with a mixture of texts, ints, timestamps, and 
counters, i get the error ' Cannot mix counter and non counter columns in the 
same table'

Is that supposed to be allowed or not allowed, given that the official example 
contains a mix of counters and non-counters?

Re: Cassandra reaper

[Alexander Dejanovski]

Running reaper with INFO level logging (that can be configured in the yaml
file), you should have a console output telling you what's going on.

If you started reaper with memory back end, restarting it will reset it and
you'll have to register your cluster again, but if you used postgres it
will resume tasks where they were left off.

Please restart Reaper to at least have an output we can get information
from, otherwise we're blind.

Since you're using Cassandra 2.1, I'd advise switching to our fork since
the original one is compiled against Cassandra 2.0 libraries. If you switch
and use postgres, make sure you update the schema accordingly as we added
fields for incremental repair support.

Cheers,

Le mar. 1 nov. 2016 18:31, Jai Bheemsen Rao Dhanwada 
a écrit :

> Cassandra version is 2.1.16
>
> In my setup I don't see it is writting to any logs
>
> On Tue, Nov 1, 2016 at 10:25 AM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
>
> Do you have anything in the reaper logs that would show a failure of some
> sort ?
> Also, can you tell me which version of Cassandra you're using ?
>
> Thanks
>
> On Tue, Nov 1, 2016 at 6:15 PM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Thanks Alex,
>
> Forgot to mention but I did add the cluster. See the status below. It says
> the status is running but I don't see any repair happening. this is in the
> same state from past 1 days.
> b/w there not much of data in cluster.
>
> [root@machine cassandra-reaper]#  ./bin/spreaper status-repair 3
> # Report improvements/bugs at
> https://github.com/spotify/cassandra-reaper/issues
> #
> --
> # Repair run with id '3':
> {
>   "cause": "manual spreaper run",
>   "cluster_name": "production",
>   "column_families": [],
>   "creation_time": "2016-11-01T00:39:15Z",
>   "duration": null,
>   "end_time": null,
>   "estimated_time_of_arrival": null,
>   "id": 3,
>   "intensity": 0.900,
>   "keyspace_name": "users",
> *  "last_event": "no events",*
>   "owner": "root",
>   "pause_time": null,
>   "repair_parallelism": "DATACENTER_AWARE",
>   "segments_repaired": 0,
>   "start_time": "2016-11-01T00:39:15Z",
> *  "state": "RUNNING",*
>   "total_segments": 301
> }
> [root@ machine cassandra-reaper]#
>
> On Tue, Nov 1, 2016 at 9:24 AM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
>
> Hi,
>
> The first step in using reaper is to add a cluster to it, as it is a tool
> that can manage multiple clusters and does not need to be executed on a
> Cassandra node (you can run in on any edge node you want).
>
> You should run : ./bin/spreaper add-cluster 127.0.0.1
> Where you'll replace 127.0.0.1 by the address of one of the nodes of your
> cluster.
>
> Then you can run : ./bin/spreaper cluster_name keyspace_name
> to start repairing a keyspace.
>
> You might want to drop in the UI made by Stefan Podkowinski which might
> ease things up for you, at least at the beginning :
> https://github.com/spodkowinski/cassandra-reaper-ui
>
> Worth mentioning that at The Last Pickle we maintain a fork of Reaper that
> handles incremental repair, works with C* 2.x and 3.0, and bundles the UI :
> https://github.com/thelastpickle/cassandra-reaper
> We have a branch that allows using Cassandra as a storage backend instead
> of Postgres :
> https://github.com/thelastpickle/cassandra-reaper/tree/add-cassandra-storage
> It should be merged to master really soon and should be ready to use.
>
> Cheers,
>
>
> On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Hello,
>
> Has anyone played around with the cassandra reaper (
> https://github.com/spotify/cassandra-reaper)?
>
> if so can some please help me with the set-up, I can't get it working. I
> used the below steps:
>
> 1. create jar file using maven
> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
> cassandra-reaper.yaml
> 3. ./bin/spreaper repair production users
>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Cannot mix counter and non counter columns in the same table

[Ali Akhtar]

^ Stockholm syndrome :)

On Tue, Nov 1, 2016 at 10:54 PM, Robert Wille  wrote:

> I used to think it was terrible as well. But it really isn’t. Just put
> your non-counter columns in a separate table with the same primary key. If
> you want to query both counter and non-counter columns at the same time,
> just query both tables at the same time with asynchronous queries.
>
> On Nov 1, 2016, at 7:29 AM, Ali Akhtar  wrote:
>
> That's a terrible gotcha rule.
>
> On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:
>
>> In your table schema, you have KEYS and you have VALUES. Your KEYS are
>> text, but they could be any non-counter type or compound thereof. KEYS
>> obviously cannot ever be counters.
>>
>> Your VALUES, however, must be either all counters or all non-counters.
>> The official example you posted conforms to this limitation.
>>
>> Thanks,
>> Cody
>>
>> On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:
>>
>>> I'm not referring to the primary key, just to other columns.
>>>
>>> My primary key is a text, and my table contains a mix of texts, ints,
>>> and timestamps.
>>>
>>> If I try to change one of the ints to a counter and run the create table
>>> query, I get the error ' Cannot mix counter and non counter columns in
>>> the same table'
>>>
>>>
>>> On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:
>>>
 For counter tables, non-counter types are of course allowed in the
 primary key. Counters would be meaningless otherwise.

 Thanks,
 Cody

 On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:

> In the documentation for counters:
>
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>
> The example table is created via:
>
> CREATE TABLE counterks.page_view_counts
>   (counter_value counter,
>   url_name varchar,
>   page_name varchar,
>   PRIMARY KEY (url_name, page_name)
> );
>
> Yet if I try to create a table with a mixture of texts, ints,
> timestamps, and counters, i get the error ' Cannot mix counter and non
> counter columns in the same table'
>
> Is that supposed to be allowed or not allowed, given that the official
> example contains a mix of counters and non-counters?
>

>>>
>
>

Re: Cannot mix counter and non counter columns in the same table

[Benjamin Roth]

Big Fat lol!!!

Am 01.11.2016 19:02 schrieb "Ali Akhtar" :

> ^ Stockholm syndrome :)
>
> On Tue, Nov 1, 2016 at 10:54 PM, Robert Wille  wrote:
>
>> I used to think it was terrible as well. But it really isn’t. Just put
>> your non-counter columns in a separate table with the same primary key. If
>> you want to query both counter and non-counter columns at the same time,
>> just query both tables at the same time with asynchronous queries.
>>
>> On Nov 1, 2016, at 7:29 AM, Ali Akhtar  wrote:
>>
>> That's a terrible gotcha rule.
>>
>> On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:
>>
>>> In your table schema, you have KEYS and you have VALUES. Your KEYS are
>>> text, but they could be any non-counter type or compound thereof. KEYS
>>> obviously cannot ever be counters.
>>>
>>> Your VALUES, however, must be either all counters or all non-counters.
>>> The official example you posted conforms to this limitation.
>>>
>>> Thanks,
>>> Cody
>>>
>>> On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:
>>>
 I'm not referring to the primary key, just to other columns.

 My primary key is a text, and my table contains a mix of texts, ints,
 and timestamps.

 If I try to change one of the ints to a counter and run the create
 table query, I get the error ' Cannot mix counter and non counter
 columns in the same table'


 On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:

> For counter tables, non-counter types are of course allowed in the
> primary key. Counters would be meaningless otherwise.
>
> Thanks,
> Cody
>
> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>
>> In the documentation for counters:
>>
>> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>>
>> The example table is created via:
>>
>> CREATE TABLE counterks.page_view_counts
>>   (counter_value counter,
>>   url_name varchar,
>>   page_name varchar,
>>   PRIMARY KEY (url_name, page_name)
>> );
>>
>> Yet if I try to create a table with a mixture of texts, ints,
>> timestamps, and counters, i get the error ' Cannot mix counter and non
>> counter columns in the same table'
>>
>> Is that supposed to be allowed or not allowed, given that the
>> official example contains a mix of counters and non-counters?
>>
>

>>
>>
>

commit log on NFS volume

[John Sanda]

I know that using NFS is discouraged, particularly for the commit log. Can
anyone shed some light into what kinds of problems I might encounter aside
from performance? The reason for my inquiry is because I have some
deployments with Cassandra 2.2.1 that use NFS and are experiencing some
problems like reoccurring corrupted commit log segments on start up:

ERROR 19:38:42 Exiting due to error while processing commit log
during initialization.

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
[apache-cassandra-2.2.1.jar]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
[apache-cassandra-2.2.1.jar]


In one deployment after removing all of corrupted commit log segments I got
a different error:

Exception (java.lang.RuntimeException) encountered during startup:
java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
java.lang.RuntimeException: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
at 
org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
at 
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
at 
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
Caused by: java.nio.file.NoSuchFileException:
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at 
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
... 8 more


The latter error looks like it involves compaction and might be unrelated.
I don't know if it matters, but I have commit log compression enabled in
these environments.

-- 

- John

Re: commit log on NFS volume

[Vladimir Yudovin]

Hi,



it's not only performance issue. In case of network problem writer tread can be 
blocked, also in case of failure loss of data can occur.



Best regards, Vladimir Yudovin, 

Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.





 On Tue, 01 Nov 2016 14:10:10 -0400John Sanda  
wrote 




I know that using NFS is discouraged, particularly for the commit log. Can 
anyone shed some light into what kinds of problems I might encounter aside from 
performance? The reason for my inquiry is because I have some deployments with 
Cassandra 2.2.1 that use NFS and are experiencing some problems like 
reoccurring corrupted commit log segments on start up:



 ERROR 19:38:42 Exiting due to error while processing commit log during 
initialization. 
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
[apache-cassandra-2.2.1.jar]



In one deployment after removing all of corrupted commit log segments I got a 
different error:



Exception (java.lang.RuntimeException) encountered during startup: 
java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) 
at org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66) 
at 
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
 at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
 at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
 at 
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
 at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216) at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) 
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
Caused by: java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at 
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
 at java.nio.channels.FileChannel.open(FileChannel.java:287) at 
java.nio.channels.FileChannel.open(FileChannel.java:335) at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ... 
8 more



The latter error looks like it involves compaction and might be unrelated. I 
don't know if it matters, but I have commit log compression enabled in these 
environments.



-- 



- John

Re: commit log on NFS volume

[Benjamin Roth]

Using nfs for a distribited System like Cassandra is like putting a Ferrari
on a Truck and going for a Race with the Truck. It is simply nonsense.

Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :

> Hi,
>
> it's not only performance issue. In case of network problem writer tread
> can be blocked, also in case of failure loss of data can occur.
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone  - Hosted Cloud
> CassandraLaunch your cluster in minutes.*
>
>
>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda  >* wrote 
>
> I know that using NFS is discouraged, particularly for the commit log. Can
> anyone shed some light into what kinds of problems I might encounter aside
> from performance? The reason for my inquiry is because I have some
> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
> problems like reoccurring corrupted commit log segments on start up:
>
> ERROR 19:38:42 Exiting due to error while processing commit log during 
> initialization.
> 
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
> [apache-cassandra-2.2.1.jar]
>
>
> In one deployment after removing all of corrupted commit log segments I
> got a different error:
>
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>   at 
> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>   at 
> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>   at 
> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
> Caused by: java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>   at java.nio.channels.FileChannel.open(FileChannel.java:287)
>   at java.nio.channels.FileChannel.open(FileChannel.java:335)
>   at 
> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
>   ... 8 more
>
>
> The latter error looks like it involves compaction and might be unrelated.
> I don't know if it matters, but I have commit log compression enabled in
> these environments.
>
> --
>
> - John
>
>
>

native_transport_max_frame_size_in_mb vs max_value_size_in_mb

[Lahiru Gamathige]

Hi Users,

I see that C* introduced max_value_size_in_mb and if a SSTable is larger
than this it will be a currupted SSTable. In our current cluster I see
tables with very large SSTables, and if we are migrating to new version
should I increase this number ?

But increasing max_value_size_in_mb to a higher value looks risky but the
documentation says these should match.

Any thoughts ?

Re: native_transport_max_frame_size_in_mb vs max_value_size_in_mb

[Lahiru Gamathige]

Since this is already commented this size check is disabled or its default
set to 256MB and if we have higher sized SSTables those are going to mark
currupted ?

On Tue, Nov 1, 2016 at 11:47 AM, Lahiru Gamathige 
wrote:

> Hi Users,
>
> I see that C* introduced max_value_size_in_mb and if a SSTable is larger
> than this it will be a currupted SSTable. In our current cluster I see
> tables with very large SSTables, and if we are migrating to new version
> should I increase this number ?
>
> But increasing max_value_size_in_mb to a higher value looks risky but the
> documentation says these should match.
>
> Any thoughts ?
>

Re: commit log on NFS volume

[John Sanda]

>
> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.


As I mentioned in my original post, I am aware that using NFS is considered
bad and even documented as an anti-pattern. Your analogy, interesting as it
may be, is not helpful. It simply restating what has already been said. I
don't even know that NFS is to blame for the CommitLogReplayException that
I cited.

On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth 
wrote:

> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.
>
> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :
>
>> Hi,
>>
>> it's not only performance issue. In case of network problem writer tread
>> can be blocked, also in case of failure loss of data can occur.
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone  - Hosted Cloud
>> CassandraLaunch your cluster in minutes.*
>>
>>
>>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda > >* wrote 
>>
>> I know that using NFS is discouraged, particularly for the commit log.
>> Can anyone shed some light into what kinds of problems I might encounter
>> aside from performance? The reason for my inquiry is because I have some
>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>> problems like reoccurring corrupted commit log segments on start up:
>>
>> ERROR 19:38:42 Exiting due to error while processing commit log during 
>> initialization.
>> 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
>>  Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
>> [apache-cassandra-2.2.1.jar]
>>
>>
>> In one deployment after removing all of corrupted commit log segments I
>> got a different error:
>>
>> Exception (java.lang.RuntimeException) encountered during startup: 
>> java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>>  at 
>> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>  at 
>> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>> Caused by: java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>  at 
>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>>  at 
>> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>>  at java.nio.channels.FileChannel.open(FileChannel.java:287)
>>  at java.nio.channels

Re: commit log on NFS volume

[Benjamin Roth]

The Second exception states there is a File of an sstable Missing. Is it
possible you didnt only delete commit logs or nfs Mount is stale or Not
mounted?

Am 01.11.2016 19:52 schrieb "John Sanda" :

> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>
>
> As I mentioned in my original post, I am aware that using NFS is
> considered bad and even documented as an anti-pattern. Your analogy,
> interesting as it may be, is not helpful. It simply restating what has
> already been said. I don't even know that NFS is to blame for the
> CommitLogReplayException that I cited.
>
> On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth 
> wrote:
>
>> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>>
>> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :
>>
>>> Hi,
>>>
>>> it's not only performance issue. In case of network problem writer tread
>>> can be blocked, also in case of failure loss of data can occur.
>>>
>>> Best regards, Vladimir Yudovin,
>>>
>>> *Winguzone  - Hosted Cloud
>>> CassandraLaunch your cluster in minutes.*
>>>
>>>
>>>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda
>>> >* wrote 
>>>
>>> I know that using NFS is discouraged, particularly for the commit log.
>>> Can anyone shed some light into what kinds of problems I might encounter
>>> aside from performance? The reason for my inquiry is because I have some
>>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>>> problems like reoccurring corrupted commit log segments on start up:
>>>
>>> ERROR 19:38:42 Exiting due to error while processing commit log during 
>>> initialization.
>>> 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
>>>  Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
>>> [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
>>> [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
>>> [apache-cassandra-2.2.1.jar]
>>>
>>>
>>> In one deployment after removing all of corrupted commit log segments I
>>> got a different error:
>>>
>>> Exception (java.lang.RuntimeException) encountered during startup: 
>>> java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> at 
>>> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>> at 
>>> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>>> at 
>>> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>> at 
>>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>> at 
>>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>> at 
>>> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>>> Caused by: java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> at 
>>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>>> at sun.nio.fs.UnixException.rethrowAsIOException(

Re: Cassandra reaper

[Jai Bheemsen Rao Dhanwada]

ok thank you,
I will try and update you.

On Tue, Nov 1, 2016 at 10:57 AM, Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> Running reaper with INFO level logging (that can be configured in the yaml
> file), you should have a console output telling you what's going on.
>
> If you started reaper with memory back end, restarting it will reset it
> and you'll have to register your cluster again, but if you used postgres it
> will resume tasks where they were left off.
>
> Please restart Reaper to at least have an output we can get information
> from, otherwise we're blind.
>
> Since you're using Cassandra 2.1, I'd advise switching to our fork since
> the original one is compiled against Cassandra 2.0 libraries. If you switch
> and use postgres, make sure you update the schema accordingly as we added
> fields for incremental repair support.
>
> Cheers,
>
> Le mar. 1 nov. 2016 18:31, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> a écrit :
>
>> Cassandra version is 2.1.16
>>
>> In my setup I don't see it is writting to any logs
>>
>> On Tue, Nov 1, 2016 at 10:25 AM, Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>
>> Do you have anything in the reaper logs that would show a failure of some
>> sort ?
>> Also, can you tell me which version of Cassandra you're using ?
>>
>> Thanks
>>
>> On Tue, Nov 1, 2016 at 6:15 PM Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>> Thanks Alex,
>>
>> Forgot to mention but I did add the cluster. See the status below. It
>> says the status is running but I don't see any repair happening. this is in
>> the same state from past 1 days.
>> b/w there not much of data in cluster.
>>
>> [root@machine cassandra-reaper]#  ./bin/spreaper status-repair 3
>> # Report improvements/bugs at https://github.com/spotify/
>> cassandra-reaper/issues
>> # 
>> --
>> # Repair run with id '3':
>> {
>>   "cause": "manual spreaper run",
>>   "cluster_name": "production",
>>   "column_families": [],
>>   "creation_time": "2016-11-01T00:39:15Z",
>>   "duration": null,
>>   "end_time": null,
>>   "estimated_time_of_arrival": null,
>>   "id": 3,
>>   "intensity": 0.900,
>>   "keyspace_name": "users",
>> *  "last_event": "no events",*
>>   "owner": "root",
>>   "pause_time": null,
>>   "repair_parallelism": "DATACENTER_AWARE",
>>   "segments_repaired": 0,
>>   "start_time": "2016-11-01T00:39:15Z",
>> *  "state": "RUNNING",*
>>   "total_segments": 301
>> }
>> [root@ machine cassandra-reaper]#
>>
>> On Tue, Nov 1, 2016 at 9:24 AM, Alexander Dejanovski <
>> a...@thelastpickle.com> wrote:
>>
>> Hi,
>>
>> The first step in using reaper is to add a cluster to it, as it is a tool
>> that can manage multiple clusters and does not need to be executed on a
>> Cassandra node (you can run in on any edge node you want).
>>
>> You should run : ./bin/spreaper add-cluster 127.0.0.1
>> Where you'll replace 127.0.0.1 by the address of one of the nodes of your
>> cluster.
>>
>> Then you can run : ./bin/spreaper cluster_name keyspace_name
>> to start repairing a keyspace.
>>
>> You might want to drop in the UI made by Stefan Podkowinski which might
>> ease things up for you, at least at the beginning : https://github.com/
>> spodkowinski/cassandra-reaper-ui
>>
>> Worth mentioning that at The Last Pickle we maintain a fork of Reaper
>> that handles incremental repair, works with C* 2.x and 3.0, and bundles the
>> UI : https://github.com/thelastpickle/cassandra-reaper
>> We have a branch that allows using Cassandra as a storage backend instead
>> of Postgres : https://github.com/thelastpickle/cassandra-
>> reaper/tree/add-cassandra-storage
>> It should be merged to master really soon and should be ready to use.
>>
>> Cheers,
>>
>>
>> On Tue, Nov 1, 2016 at 1:45 AM Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>> Hello,
>>
>> Has anyone played around with the cassandra reaper (
>> https://github.com/spotify/cassandra-reaper)?
>>
>> if so can some please help me with the set-up, I can't get it working. I
>> used the below steps:
>>
>> 1. create jar file using maven
>> 2. java -jar cassandra-reaper-0.2.3-SNAPSHOT.jar server
>> cassandra-reaper.yaml
>> 3. ./bin/spreaper repair production users
>>
>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>>
>> --
>> -
>> Alexander Dejanovski
>> France
>> @alexanderdeja
>>
>> Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>>
>> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: Cannot mix counter and non counter columns in the same table

[Edward Capriolo]

Here is a solution that I have leverage. Ignore the count of the value and
use a multi-part column name as it's value.

For example:

create column family stuff (
rowkey string,
column string,
value string.
counter_to_ignore long,
primary key( rowkey, column, value));



On Tue, Nov 1, 2016 at 9:29 AM, Ali Akhtar  wrote:

> That's a terrible gotcha rule.
>
> On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:
>
>> In your table schema, you have KEYS and you have VALUES. Your KEYS are
>> text, but they could be any non-counter type or compound thereof. KEYS
>> obviously cannot ever be counters.
>>
>> Your VALUES, however, must be either all counters or all non-counters.
>> The official example you posted conforms to this limitation.
>>
>> Thanks,
>> Cody
>>
>> On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:
>>
>>> I'm not referring to the primary key, just to other columns.
>>>
>>> My primary key is a text, and my table contains a mix of texts, ints,
>>> and timestamps.
>>>
>>> If I try to change one of the ints to a counter and run the create table
>>> query, I get the error ' Cannot mix counter and non counter columns in
>>> the same table'
>>>
>>>
>>> On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:
>>>
 For counter tables, non-counter types are of course allowed in the
 primary key. Counters would be meaningless otherwise.

 Thanks,
 Cody

 On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:

> In the documentation for counters:
>
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>
> The example table is created via:
>
> CREATE TABLE counterks.page_view_counts
>   (counter_value counter,
>   url_name varchar,
>   page_name varchar,
>   PRIMARY KEY (url_name, page_name)
> );
>
> Yet if I try to create a table with a mixture of texts, ints,
> timestamps, and counters, i get the error ' Cannot mix counter and non
> counter columns in the same table'
>
> Is that supposed to be allowed or not allowed, given that the official
> example contains a mix of counters and non-counters?
>

>>>
>

Schema not translated completely from Thrift protocol to CQL protocol

[Nitin Pasari]

Hi,

I am trying to move from using pycassa to native protocol in my project
(which will let us upgrade the version of cassandra). My schema was defined
using pycassa so it created a Column family using compact storage and it
has 3 columns which are not part of the composite primary key (I know this
is not allowed by CQL protocol.)

The version of my current Cassandra cluster is 2.0.17. The schema when I do
"show schema" using the thrift protocol comes to be:

create column family store
  with column_type = 'Standard'
  and comparator =
'CompositeType(org.apache.cassandra.db.marshal.ReversedType(org.apache.cassandra.db.marshal.LongType),org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.AsciiType)'
  and default_validation_class = 'DoubleType'
  and key_validation_class = 'AsciiType'
  and column_metadata = [
{column_name : 'something1',
validation_class : AsciiType},
{column_name : 'something2',
validation_class : AsciiType}]


But when I check the schema on the native protocol, it is missing "column3"
and "value" columns. It comes out as follows:

CREATE TABLE store (
  key ascii,
  column1 bigint,
  column2 ascii,
  something1 ascii,
  something2 ascii,
  PRIMARY KEY ((key), column1, column2)) WITH COMPACT STORAGE AND
  CLUSTERING ORDER BY (column1 DESC, column2 ASC)


Now, because of this discrepancy, I cannot transition from pycassa to the
native protocol on the client side. I haven't been able to find anything to
overcome this problem and make sure that the native protocol sees the right
schema. Is there anything you could suggest me to fix this? Any help is
appreciated!

Thanks,
Nitin

Re: Cannot mix counter and non counter columns in the same table

[Cody Yancey]

I agree it makes code messier, but you aren't really losing anything by
separating them out into separate tables and then doing parallel queries.

Counter tables already don't support atomic batch operations (all batch
operations are unlogged), CAS operations (LWTs not supported) and have a
whole host of other gotchas that don't apply as much to Cassandra but have
more to do with the mathematical underpinnings of non-idempotent operations
in a world where the Two Generals problem is unsolved.

If Cassandra WAS to allow mixing of storage paradigms into one "logical"
table it would probably be just two separate tables under the hood anyway
since the write path is so different.

This isn't Stockholm Syndrome for Cassandra as much as it is Stockholm
Syndrome for databases. I've never used a database that could count very
well, even non-distributed databases like postgres or mysql. Cassandra's
implementation is at least fast and scalable.

Thanks,
Cody

On Tue, Nov 1, 2016 at 2:13 PM, Edward Capriolo 
wrote:

> Here is a solution that I have leverage. Ignore the count of the value and
> use a multi-part column name as it's value.
>
> For example:
>
> create column family stuff (
> rowkey string,
> column string,
> value string.
> counter_to_ignore long,
> primary key( rowkey, column, value));
>
>
>
> On Tue, Nov 1, 2016 at 9:29 AM, Ali Akhtar  wrote:
>
>> That's a terrible gotcha rule.
>>
>> On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:
>>
>>> In your table schema, you have KEYS and you have VALUES. Your KEYS are
>>> text, but they could be any non-counter type or compound thereof. KEYS
>>> obviously cannot ever be counters.
>>>
>>> Your VALUES, however, must be either all counters or all non-counters.
>>> The official example you posted conforms to this limitation.
>>>
>>> Thanks,
>>> Cody
>>>
>>> On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:
>>>
 I'm not referring to the primary key, just to other columns.

 My primary key is a text, and my table contains a mix of texts, ints,
 and timestamps.

 If I try to change one of the ints to a counter and run the create
 table query, I get the error ' Cannot mix counter and non counter
 columns in the same table'


 On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:

> For counter tables, non-counter types are of course allowed in the
> primary key. Counters would be meaningless otherwise.
>
> Thanks,
> Cody
>
> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>
>> In the documentation for counters:
>>
>> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_counter_t.html
>>
>> The example table is created via:
>>
>> CREATE TABLE counterks.page_view_counts
>>   (counter_value counter,
>>   url_name varchar,
>>   page_name varchar,
>>   PRIMARY KEY (url_name, page_name)
>> );
>>
>> Yet if I try to create a table with a mixture of texts, ints,
>> timestamps, and counters, i get the error ' Cannot mix counter and non
>> counter columns in the same table'
>>
>> Is that supposed to be allowed or not allowed, given that the
>> official example contains a mix of counters and non-counters?
>>
>

>>
>

Re: Incremental repairs leading to unrepaired data

[kurt Greaves]

Can't say I have too many ideas. If load is low during the repair it
shouldn't be happening. Your disks aren't overutilised correct? No other
processes writing loads of data to them?

Re: Cannot mix counter and non counter columns in the same table

[Ali Akhtar]

> I agree it makes code messier

That's not a trivial point.

It isn't just about doing two queries to read. Its also having to do two
queries to write and remembering to do that in all the places.

On Wed, Nov 2, 2016 at 1:56 AM, Cody Yancey  wrote:

> I agree it makes code messier, but you aren't really losing anything by
> separating them out into separate tables and then doing parallel queries.
>
> Counter tables already don't support atomic batch operations (all batch
> operations are unlogged), CAS operations (LWTs not supported) and have a
> whole host of other gotchas that don't apply as much to Cassandra but have
> more to do with the mathematical underpinnings of non-idempotent operations
> in a world where the Two Generals problem is unsolved.
>
> If Cassandra WAS to allow mixing of storage paradigms into one "logical"
> table it would probably be just two separate tables under the hood anyway
> since the write path is so different.
>
> This isn't Stockholm Syndrome for Cassandra as much as it is Stockholm
> Syndrome for databases. I've never used a database that could count very
> well, even non-distributed databases like postgres or mysql. Cassandra's
> implementation is at least fast and scalable.
>
> Thanks,
> Cody
>
> On Tue, Nov 1, 2016 at 2:13 PM, Edward Capriolo 
> wrote:
>
>> Here is a solution that I have leverage. Ignore the count of the value
>> and use a multi-part column name as it's value.
>>
>> For example:
>>
>> create column family stuff (
>> rowkey string,
>> column string,
>> value string.
>> counter_to_ignore long,
>> primary key( rowkey, column, value));
>>
>>
>>
>> On Tue, Nov 1, 2016 at 9:29 AM, Ali Akhtar  wrote:
>>
>>> That's a terrible gotcha rule.
>>>
>>> On Tue, Nov 1, 2016 at 6:27 PM, Cody Yancey  wrote:
>>>
 In your table schema, you have KEYS and you have VALUES. Your KEYS are
 text, but they could be any non-counter type or compound thereof. KEYS
 obviously cannot ever be counters.

 Your VALUES, however, must be either all counters or all non-counters.
 The official example you posted conforms to this limitation.

 Thanks,
 Cody

 On Nov 1, 2016 7:16 AM, "Ali Akhtar"  wrote:

> I'm not referring to the primary key, just to other columns.
>
> My primary key is a text, and my table contains a mix of texts, ints,
> and timestamps.
>
> If I try to change one of the ints to a counter and run the create
> table query, I get the error ' Cannot mix counter and non counter
> columns in the same table'
>
>
> On Tue, Nov 1, 2016 at 6:11 PM, Cody Yancey  wrote:
>
>> For counter tables, non-counter types are of course allowed in the
>> primary key. Counters would be meaningless otherwise.
>>
>> Thanks,
>> Cody
>>
>> On Nov 1, 2016 7:00 AM, "Ali Akhtar"  wrote:
>>
>>> In the documentation for counters:
>>>
>>> https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_count
>>> er_t.html
>>>
>>> The example table is created via:
>>>
>>> CREATE TABLE counterks.page_view_counts
>>>   (counter_value counter,
>>>   url_name varchar,
>>>   page_name varchar,
>>>   PRIMARY KEY (url_name, page_name)
>>> );
>>>
>>> Yet if I try to create a table with a mixture of texts, ints,
>>> timestamps, and counters, i get the error ' Cannot mix counter and non
>>> counter columns in the same table'
>>>
>>> Is that supposed to be allowed or not allowed, given that the
>>> official example contains a mix of counters and non-counters?
>>>
>>
>
>>>
>>
>