Data Model - Consistency question

[Roshni Rajagopal]

Hi Folks,
In the relational world, if I needed to model students, courses relationship, I 
may have donea students -master tablea course - master tablea bridge table 
students-course which gives me the ids to students and the courses they are 
taking. This can answer both 'which students take course A', as well as 'which 
courses are taken by student B'
In the cassandra world, I may design it like thisa static student column 
familya static course column familya student-course column family with student 
id as key and dynamic list of course - ids to answer 'which courses are taken 
by student B'a course-student column family with course id as key and dynamic 
list of student ids  'which students take course A'
A screen which displays some student entity details as well as all the courses 
she is taking will need to refer to 2 column families
Suppose an application inserts a new row in student column family, and a new 
row in student-course column family, as transactions or consistency across 
column families is not guaranteed, there is a chance that the client receives 
information that a student is attending a course from student-course column 
family, but does not exist in the student column family. 
If we use Strong consistency from the  reads + writes combination - will this 
scenario not occur ?And if we dont, can this scenario occur?  Regards,Roshni





Regards,Roshni

Solr Use Cases

[Roshni Rajagopal]

Hi,
Im new to Solr, and I hear that Solr is a great tool for improving search 
performanceIm unsure whether Solr or DSE Search is a must for all cassandra 
deployments
1. For performance - I thought cassandra had great read & write performance. 
When should solr be used ?Taking the following use cases for cassandra from the 
datastax FAQ page, in which cases would Solr be useful, and whether for 
all?Time series data managementHigh-velocity device data ingestion and 
analysisMedia streaming (e.g., music, movies)Social media input and 
analysisOnline web retail (e.g., shopping carts, user transactions)Web log 
management / analysisWeb click-stream analysisReal-time data analyticsOnline 
gaming (e.g., real-time messaging)Write-intensive transaction systemsBuyer 
event analyticsRisk analysis and management
2. what changes to cassandra data modeling does Solr bring? We have some 
guidelines & best practices around cassandra data modeling.Is Solr so powerful, 
that it does not matter how data is modelled in cassandra? Are there different 
best practices for cassandra data modeling when Solr is in the picture?Is this 
something we should keep in mind while modeling for cassandra today- that it 
should be  good to be used via Solr in future?
3. Does Solr come with any drawbacks like its not real time ? 
I can & should read the manual, but it will be great if someone can explain at 
a high level. 
Thank you!

Regards,Roshni

Re: Data Model - Consistency question

[Hiller, Dean]

Yes, this scenario can occur(even with quorum writes/reads as you are dealing 
with different rows) as one write may be complete and the other not while 
someone else is reading from the cluster.  Generally though, you can do read 
repair when you read it in ;).  Ie. See if things are inconsistent when reading 
it and either 1. Wait and read again or 2. Figure out a way to display the 
results correctly based on merging data.  In general #1 on a lot of systems is 
not a bad approach when you can't merge the data yourself because the conflicts 
are not happening to often(maybe < 1% of the time in a lot of cases)

With playOrm if you have partitions, you can still have that relational model 
you described if you can figure out a partition strategy AND you can then query 
on it with joins and such.

Later,
Dean


From: Roshni Rajagopal 
mailto:roshni_rajago...@hotmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, September 19, 2012 6:16 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Data Model - Consistency question

Hi Folks,

In the relational world, if I needed to model students, courses relationship, I 
may have done
a students -master table
a course - master table
a bridge table students-course which gives me the ids to students and the 
courses they are taking. This can answer both 'which students take course A', 
as well as 'which courses are taken by student B'

In the cassandra world, I may design it like this
a static student column family
a static course column family
a student-course column family with student id as key and dynamic list of 
course - ids to answer 'which courses are taken by student B'
a course-student column family with course id as key and dynamic list of 
student ids  'which students take course A'

A screen which displays some student entity details as well as all the courses 
she is taking will need to refer to 2 column families

Suppose an application inserts a new row in student column family, and a new 
row in student-course column family, as transactions or consistency across 
column families is not guaranteed, there is a chance that the client receives 
information that a student is attending a course from student-course column 
family, but does not exist in the student column family.

If we use Strong consistency from the  reads + writes combination - will this 
scenario not occur ?
And if we dont, can this scenario occur?

Regards,
Roshni






Regards,
Roshni

Re: Data Model - Consistency question

[Hiller, Dean]

Thinking a little more on your issue, you can also do that in playroom as
OneToMany is represented with a few columns in the owning table/entity
unlike JPA and RDBMS.

Ie.

Student.java {
 List - These course primary keys are saved one per column in the
student's row
}

Course.java {
 List - These students are saved one per column in the courses
row
}

We sometimes do this with playOrm and don't even bother with the S-SQL it
has which also means you don't need to worry about partitioning in that
case.

Later,
Dean

On 9/19/12 6:46 AM, "Hiller, Dean"  wrote:

>Yes, this scenario can occur(even with quorum writes/reads as you are
>dealing with different rows) as one write may be complete and the other
>not while someone else is reading from the cluster.  Generally though,
>you can do read repair when you read it in ;).  Ie. See if things are
>inconsistent when reading it and either 1. Wait and read again or 2.
>Figure out a way to display the results correctly based on merging data.
>In general #1 on a lot of systems is not a bad approach when you can't
>merge the data yourself because the conflicts are not happening to
>often(maybe < 1% of the time in a lot of cases)
>
>With playOrm if you have partitions, you can still have that relational
>model you described if you can figure out a partition strategy AND you
>can then query on it with joins and such.
>
>Later,
>Dean
>
>
>From: Roshni Rajagopal
>mailto:roshni_rajago...@hotmail.com>>
>Reply-To: "user@cassandra.apache.org"
>mailto:user@cassandra.apache.org>>
>Date: Wednesday, September 19, 2012 6:16 AM
>To: "user@cassandra.apache.org"
>mailto:user@cassandra.apache.org>>
>Subject: Data Model - Consistency question
>
>Hi Folks,
>
>In the relational world, if I needed to model students, courses
>relationship, I may have done
>a students -master table
>a course - master table
>a bridge table students-course which gives me the ids to students and the
>courses they are taking. This can answer both 'which students take course
>A', as well as 'which courses are taken by student B'
>
>In the cassandra world, I may design it like this
>a static student column family
>a static course column family
>a student-course column family with student id as key and dynamic list of
>course - ids to answer 'which courses are taken by student B'
>a course-student column family with course id as key and dynamic list of
>student ids  'which students take course A'
>
>A screen which displays some student entity details as well as all the
>courses she is taking will need to refer to 2 column families
>
>Suppose an application inserts a new row in student column family, and a
>new row in student-course column family, as transactions or consistency
>across column families is not guaranteed, there is a chance that the
>client receives information that a student is attending a course from
>student-course column family, but does not exist in the student column
>family.
>
>If we use Strong consistency from the  reads + writes combination - will
>this scenario not occur ?
>And if we dont, can this scenario occur?
>
>Regards,
>Roshni
>
>
>
>
>
>
>Regards,
>Roshni

Re: Data Modeling - JSON vs Composite columns

[Brian O'Neill]

Roshni,

We're going through the same debate right now.

I believe native support for JSON (or collections) is on the docket
for Cassandra.
Here is a discussion we had a few months ago on the topic:
http://comments.gmane.org/gmane.comp.db.cassandra.devel/5233

We presently store JSON, but we're considering a change to composite keys.

Presently, each client has to parse the JSON value.  If you are
retrieving lots of values, that's a lot of parsing.  Also, storing the
raw values allows for better integration with other tools, such as
reporting engines (e.g. JasperSoft).  Also, if you do want to update a
single value inside the json, you get into real trouble, because you
first need to read the value, update the field, then write the column
again.  The read before write is a problem, especially if you have a
lot of concurrency in your system.  (Two clients could read the old
value, then update different fields, and the second would overwrite
the firsts change)

One final note...
(As a side not, JSON values also complicated our wide-row indexing
mechanism: (https://github.com/hmsonline/cassandra-indexing))

For those reasons, we're considering a data model shift away from JSON.

That said, I'm keeping a close watch on:
https://issues.apache.org/jira/browse/CASSANDRA-3647

But if this is CQL only, I'm not sure how much use it will be for us
since we're coming in from different clients.
Anyone know how/if collections will be available from other clients?

-brian


On Wed, Sep 19, 2012 at 8:00 AM, Roshni Rajagopal
 wrote:
> Hi,
>
> There was a conversation on this some time earlier, and to continue it
>
> Suppose I want to associate a user to  an item, and I want to also store 3
> commonly used attributes without needing to go to an entity item column
> family , I have 2 options :-
>
> A) use composite columns
> UserId1 : {
>  : = Betty Crocker,
>  : = Cake
> : = 5
>  : = Nutella,
>  : = Choc spread
> : = 15
> }
>
> B) use a json with the data
> UserId1 : {
>   = {name: Betty Crocker,descr: Cake, Qty: 5},
>   ={name: Nutella,descr: Choc spread, Qty: 15}
> }
>
> Essentially A is better if one wants to update individual fields , while B
> is better if one wants easier paging, reading multiple items at once in one
> read. etc. The details are in this discussion thread
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Data-Modeling-another-question-td7581967.html
>
> I had an additional question,
> as its being said, that CQL is the direction in which cassandra is moving,
> and there's a lot of effort in making CQL the standard,
>
> How does approach B work in CQL. Can we read/write a JSON easily in CQL? Can
> we extract a field from a JSON in CQL or would that need to be done via the
> client code?
>
>
> Regards,
> Roshni



-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
Apache Cassandra MVP
mobile:215.588.6024
blog: http://brianoneill.blogspot.com/
twitter: @boneill42

higher layer library makes things faster?

[Hiller, Dean]

So there is this interesting case where a higher layer library makes things 
slower.  This is counter-intuitive as every abstraction usually makes things 
slower with an increase in productivity.It would be cool if more and more 
libraries supported something to help with this scenario I think.

The concept.   Once in a while you end up needing a new query into an noSQL 
data model that already exists, and you do something like this

UserRow user = fetchUser(rowKey);
List mappings = fetchRoleMappings(user.getRoleMappingIds())
List rowKeys = new ArrayList();
for(RoleMapping m : mappings) {
   rowKeys.addAll(m.getGroupIds());
}
List groups = fetchGroupRows(rowKeys);

It turns out if you index stuff, it is much faster in a lot of cases.  Instead 
you can scan just 2 index rows to find the keys for the groups and read them 
in.  Basically you do one scan on the RoleMapping where mapping has a FK to 
UserRow and you now have a list of primary keys for your RoleMapping.  Next, 
you do a scan of the GroupRow index feeding in the primary keys of your 
RoleMapping which feeds back your GroupRow primary keys that you need to 
lookup….in this way, you skip not only a lot of coding that goes into avoiding 
getting all the UserRow data back, and can simply scan the indexes.

That said, every time you update a row, you need to remove an old value from 
the index and a new one to the index.  Inserts only need to add.  Removes only 
need to remove.

Anyways, I have found this to be quite an interesting paradigm shift as right 
now doing the latter manually requires a lot more code (but then, that is why 
more and more libraries like playOrm I think will exist in the future as it 
makes the above very simple to do yourself).

Later,
Dean

Re: Solr Use Cases

[Brian O'Neill]

Roshni,

We're using SOLR to support ad hoc queries and fuzzy searches against
unstructured data stored in Cassandra.  Cassandra is great for storage
and you can create data models and indexes that support your queries,
provided you can anticipate those queries.  When you can't anticipate
the queries, or if you need to support a large permutation of
multi-dimensional queries, your probably better off using an index
like SOLR.

Since SOLR only supports a flat document structure, you may need to
perform transformation before inserting into SOLR.  We chose not to
use DSE, so we used a cassandra-triggers as our mechanism to integrate
SOLR. (https://github.com/hmsonline/cassandra-triggers)  We intercept
the mutation, transform the data into a document (w/ multi-value
fields) and POST it to SOLR.

More recently though, we're looking to roll out ElasticSearch.  As our
query demand increases, we expect SOLR to quickly become a PITA to
administrer.  (master->slave relationships)  IMHO, ElasticSearch's
architecture is a better match for Cassandra.  We are also looking to
substitute cassandra-triggers for Storm, allowing us to build a data
processing flow using Cassandra and ElasticSearch bolts.  (we've open
sourced the Cassandra bolt and we'll be open sourcing the elastic
search bolt shortly)

-brian


On Wed, Sep 19, 2012 at 8:27 AM, Roshni Rajagopal
 wrote:
> Hi,
>
> Im new to Solr, and I hear that Solr is a great tool for improving search
> performance
> Im unsure whether Solr or DSE Search is a must for all cassandra deployments
>
> 1. For performance - I thought cassandra had great read & write performance.
> When should solr be used ?
> Taking the following use cases for cassandra from the datastax FAQ page, in
> which cases would Solr be useful, and whether for all?
>
> Time series data management
> High-velocity device data ingestion and analysis
> Media streaming (e.g., music, movies)
> Social media input and analysis
> Online web retail (e.g., shopping carts, user transactions)
> Web log management / analysis
> Web click-stream analysis
> Real-time data analytics
> Online gaming (e.g., real-time messaging)
> Write-intensive transaction systems
> Buyer event analytics
> Risk analysis and management
>
>
> 2. what changes to cassandra data modeling does Solr bring? We have some
> guidelines & best practices around cassandra data modeling.
> Is Solr so powerful, that it does not matter how data is modelled in
> cassandra? Are there different best practices for cassandra data modeling
> when Solr is in the picture?
> Is this something we should keep in mind while modeling for cassandra today-
> that it should be  good to be used via Solr in future?
>
> 3. Does Solr come with any drawbacks like its not real time ?
>
> I can & should read the manual, but it will be great if someone can explain
> at a high level.
>
> Thank you!
>
>
> Regards,
> Roshni



-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)
Apache Cassandra MVP
mobile:215.588.6024
blog: http://brianoneill.blogspot.com/
twitter: @boneill42

Losing keyspace on cassandra upgrade

[Thomas Stets]

I consistently keep losing my keyspace on upgrading from cassandra 1.1.1 to
1.1.5

I have the same cassandra keyspace on all our staging systems:

development:  a 3-node cluster
integration: a 3-node cluster
QS: a 2-node cluster
(productive will be a 4-node cluster, which is as yet not active)

All clusters were running cassandra 1.1.1. Before going productive I wanted
to upgrade to the
latest productive version of cassandra.

In all cases my keyspace disappeared when I started the cluster with
cassandra 1.1.5.
On the development system I didn't realize at first what was happening. I
just wondered that nodetool
showed a very low amount of data. On integration I saw the problem quickly,
but could not recover the
data. I re-installed the cassandra cluster from scratch, and populated it
with our test data, so our
developers could work.

I am currently using the QS system to recreate the problem and try to find
what I am doing wrong,
and how I can avoid losing productive data once we are live.

Basically I was doing the following:

1. create a snapshot on every node
2. create a tar.gz of my data directory, just to be safe
3. shut down and re-start cassandra 1.1.1 (just to see that it is not the
re-start that is creating the problem)
4. verify that the keyspace is still known, and the data present.
5. shut down cassandra 1.1.1
6. copy the config to cassandra 1.1.5 (doing a diff of cassandra.yaml to
the new one first, to see whether anything important has changed)
7. start cassandra 1.1.5

In the log file, after the "Replaying ..." messages I find the following:

 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 759 mutations from unknown (probably removed) CF with id 1187
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 606 mutations from unknown (probably removed) CF with id 1186
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 53 mutations from unknown (probably removed) CF with id 1185
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 1945 mutations from unknown (probably removed) CF with id 1184
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 1945 mutations from unknown (probably removed) CF with id 1191
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103)
Skipped 7506 mutations from unknown (probably removed) CF with id 1190
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 88 mutations from unknown (probably removed) CF with id 1189
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 87 mutations from unknown (probably removed) CF with id 1188
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 354 mutations from unknown (probably removed) CF with id 1195
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 87 mutations from unknown (probably removed) CF with id 1194
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 45 mutations from unknown (probably removed) CF with id 1192
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 82 mutations from unknown (probably removed) CF with id 1197
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 46386 mutations from unknown (probably removed) CF with id 1177
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103)
Skipped 69 mutations from unknown (probably removed) CF with id 1178
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103)
Skipped 73 mutations from unknown (probably removed) CF with id 1179
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103)
Skipped 88 mutations from unknown (probably removed) CF with id 1181
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103)
Skipped 46386 mutations from unknown (probably removed) CF with id 1182
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103)
Skipped 7506 mutations from unknown (probably removed) CF with id 1183
 INFO [main] 2012-09-19 15:15:50,325 CommitLog.java (line 131) Log replay
complete, 0 replayed mutations

This is the first obvious indication something is wrong. Going further up
in the log file I discover that the SSTableReader logs only system keyspace
files.

Currently my cluster is in the folloing state:

node 1 runs cassandra 1.1.5, and doesn't know my keyspace
node 2 runs cassandra 1.1.1, and still nows my keyspace.

nodetool ring confirms this: node a has a load of 29kb, node 2 of roughly
1GB. The cluster itself is still intact, i.e. nodetool ring shows both
nodes.

I tried a nodetool resetlocalschema, and nodetool repair, but that didn't
change anything.

Any idea what I have been doing wrong (the preferred solution), or whether
I stumbled over a cassandra bug (not so nice)?


  TIA, Thomas

Re: Solr Use Cases

[Michael Kjellman]

If I were you I would look into ElasticSearch unless you are okay updating the 
search cache very infrequently.

I tried Solandra vs ElasticSearch in our use case and there was no contest.

Also, Cassandra is great for writes but not as optimized for reads. Honestly, 
it all depends on your use case and what brand of Lucine depends on it.

I would benchmark it and see what sticks.

On Sep 19, 2012, at 5:28 AM, "Roshni Rajagopal" 
mailto:roshni_rajago...@hotmail.com>> wrote:

Hi,

Im new to Solr, and I hear that Solr is a great tool for improving search 
performance
Im unsure whether Solr or DSE Search is a must for all cassandra deployments

1. For performance - I thought cassandra had great read & write performance. 
When should solr be used ?
Taking the following use cases for cassandra from the datastax FAQ page, in 
which cases would Solr be useful, and whether for all?

 *   Time series data management
 *   High-velocity device data ingestion and analysis
 *   Media streaming (e.g., music, movies)
 *   Social media input and analysis
 *   Online web retail (e.g., shopping carts, user transactions)
 *   Web log management / analysis
 *   Web click-stream analysis
 *   Real-time data analytics
 *   Online gaming (e.g., real-time messaging)
 *   Write-intensive transaction systems
 *   Buyer event analytics
 *   Risk analysis and management

2. what changes to cassandra data modeling does Solr bring? We have some 
guidelines & best practices around cassandra data modeling.
Is Solr so powerful, that it does not matter how data is modelled in cassandra? 
Are there different best practices for cassandra data modeling when Solr is in 
the picture?
Is this something we should keep in mind while modeling for cassandra today- 
that it should be  good to be used via Solr in future?

3. Does Solr come with any drawbacks like its not real time ?

I can & should read the manual, but it will be great if someone can explain at 
a high level.

Thank you!


Regards,
Roshni

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: Losing keyspace on cassandra upgrade

[Michael Kjellman]

Sounds like you are loosing your system keyspace. When you say nothing 
important changed between yaml files do you mean with or without your changes?

Did your data directories change in the migration? Permissions okay?

I've done a 1.1.1 to 1.1.5 upgrade on many of my nodes without issue..

On Sep 19, 2012, at 7:44 AM, "Thomas Stets"  wrote:

> I consistently keep losing my keyspace on upgrading from cassandra 1.1.1 to 
> 1.1.5
> 
> I have the same cassandra keyspace on all our staging systems:
> 
> development:  a 3-node cluster
> integration: a 3-node cluster
> QS: a 2-node cluster
> (productive will be a 4-node cluster, which is as yet not active)
> 
> All clusters were running cassandra 1.1.1. Before going productive I wanted 
> to upgrade to the
> latest productive version of cassandra.
> 
> In all cases my keyspace disappeared when I started the cluster with 
> cassandra 1.1.5.
> On the development system I didn't realize at first what was happening. I 
> just wondered that nodetool
> showed a very low amount of data. On integration I saw the problem quickly, 
> but could not recover the
> data. I re-installed the cassandra cluster from scratch, and populated it 
> with our test data, so our
> developers could work.
> 
> I am currently using the QS system to recreate the problem and try to find 
> what I am doing wrong,
> and how I can avoid losing productive data once we are live.
> 
> Basically I was doing the following:
> 
> 1. create a snapshot on every node
> 2. create a tar.gz of my data directory, just to be safe
> 3. shut down and re-start cassandra 1.1.1 (just to see that it is not the 
> re-start that is creating the problem)
> 4. verify that the keyspace is still known, and the data present.
> 5. shut down cassandra 1.1.1
> 6. copy the config to cassandra 1.1.5 (doing a diff of cassandra.yaml to the 
> new one first, to see whether anything important has changed)
> 7. start cassandra 1.1.5
> 
> In the log file, after the "Replaying ..." messages I find the following:
> 
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 759 mutations from unknown (probably removed) CF with id 1187
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 606 mutations from unknown (probably removed) CF with id 1186
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 53 mutations from unknown (probably removed) CF with id 1185
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 1945 mutations from unknown (probably removed) CF with id 1184
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 1945 mutations from unknown (probably removed) CF with id 1191
>  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) 
> Skipped 7506 mutations from unknown (probably removed) CF with id 1190
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 88 mutations from unknown (probably removed) CF with id 1189
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 87 mutations from unknown (probably removed) CF with id 1188
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 354 mutations from unknown (probably removed) CF with id 1195
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 87 mutations from unknown (probably removed) CF with id 1194
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 45 mutations from unknown (probably removed) CF with id 1192
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 82 mutations from unknown (probably removed) CF with id 1197
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 46386 mutations from unknown (probably removed) CF with id 1177
>  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) 
> Skipped 69 mutations from unknown (probably removed) CF with id 1178
>  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) 
> Skipped 73 mutations from unknown (probably removed) CF with id 1179
>  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) 
> Skipped 88 mutations from unknown (probably removed) CF with id 1181
>  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) 
> Skipped 46386 mutations from unknown (probably removed) CF with id 1182
>  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) 
> Skipped 7506 mutations from unknown (probably removed) CF with id 1183
>  INFO [main] 2012-09-19 15:15:50,325 CommitLog.java (line 131) Log replay 
> complete, 0 replayed mutations
> 
> This is the first obvious indication something is wrong. Going further up in 
> the log file I discover that the SSTableReader logs only system keyspace 
> files.
> 
> Currently my cluster is in the folloing state:
> 
> node 1 runs c

Re: Setting the default replication factor for Solandra cores

[Michael Kjellman]

If I recall correctly you should make those changes in the schema through the 
CLI.

I never ended up running Solandra in production though so I'm not sure if 
anyone else has better options. Why is the CLI not enough?

On Sep 19, 2012, at 5:56 AM, "Safdar Kureishy" 
mailto:safdar.kurei...@gmail.com>> wrote:

Hi,

This question is related to Solandra, but since it sits on top of Cassandra, I 
figured I'd use this mailing list (since there isn't another one I know of for 
Solandra). Apologies in advance if this is the wrong place for this.

I'm trying to setup a Solandra cluster, across 2 centers. I want to set the 
replication factor = 2, and the default replications strategy to be 
RackAwareStrategy. Is there somewhere in cassandra.yaml or solandra.properties 
that I can provide these parameters so that I won't need to use cassandra-cli 
manually? I couldn't find any such property so far...

Thanks in advance.

Safdar

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: Data Modeling - JSON vs Composite columns

[Michael Kjellman]

Client code. CQL will only deserialize composites as you mention in A.

On Sep 19, 2012, at 5:01 AM, "Roshni Rajagopal" 
mailto:roshni_rajago...@hotmail.com>> wrote:

Hi,

There was a conversation on this some time earlier, and to continue it

Suppose I want to associate a user to  an item, and I want to also store 3 
commonly used attributes without needing to go to an entity item column family 
, I have 2 options :-

A) use composite columns
UserId1 : {
 : = Betty Crocker,
 : = Cake
: = 5
 : = Nutella,
 : = Choc spread
: = 15
}

B) use a json with the data
UserId1 : {
  = {name: Betty Crocker,descr: Cake, Qty: 5},
  ={name: Nutella,descr: Choc spread, Qty: 15}
}

Essentially A is better if one wants to update individual fields , while B is 
better if one wants easier paging, reading multiple items at once in one read. 
etc. The details are in this discussion thread
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Data-Modeling-another-question-td7581967.html

I had an additional question,
as its being said, that CQL is the direction in which cassandra is moving, and 
there's a lot of effort in making CQL the standard,

How does approach B work in CQL. Can we read/write a JSON easily in CQL? Can we 
extract a field from a JSON in CQL or would that need to be done via the client 
code?


Regards,
Roshni

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: Losing keyspace on cassandra upgrade

[Edward Sargisson]

We've seen that before too - supposedly it was fixed in 1.1.5. Your 
experience casts some doubt on that.


Our workaround, thus far, is to shut down the entire ring and then bring 
each node back up starting with known good.
Then you do nodetool resetlocalschema on the node that's confused and 
make sure it gets the schema linked up properly.

Then nodetool repair.

I see you've done that but we found a complete ring restart was 
necessary. This was on Cass 1.1.1.


Cheers,
Edward

On 12-09-19 08:12 AM, Michael Kjellman wrote:

Sounds like you are loosing your system keyspace. When you say nothing 
important changed between yaml files do you mean with or without your changes?

Did your data directories change in the migration? Permissions okay?

I've done a 1.1.1 to 1.1.5 upgrade on many of my nodes without issue..

On Sep 19, 2012, at 7:44 AM, "Thomas Stets"  wrote:


I consistently keep losing my keyspace on upgrading from cassandra 1.1.1 to 
1.1.5

I have the same cassandra keyspace on all our staging systems:

development:  a 3-node cluster
integration: a 3-node cluster
QS: a 2-node cluster
(productive will be a 4-node cluster, which is as yet not active)

All clusters were running cassandra 1.1.1. Before going productive I wanted to 
upgrade to the
latest productive version of cassandra.

In all cases my keyspace disappeared when I started the cluster with cassandra 
1.1.5.
On the development system I didn't realize at first what was happening. I just 
wondered that nodetool
showed a very low amount of data. On integration I saw the problem quickly, but 
could not recover the
data. I re-installed the cassandra cluster from scratch, and populated it with 
our test data, so our
developers could work.

I am currently using the QS system to recreate the problem and try to find what 
I am doing wrong,
and how I can avoid losing productive data once we are live.

Basically I was doing the following:

1. create a snapshot on every node
2. create a tar.gz of my data directory, just to be safe
3. shut down and re-start cassandra 1.1.1 (just to see that it is not the 
re-start that is creating the problem)
4. verify that the keyspace is still known, and the data present.
5. shut down cassandra 1.1.1
6. copy the config to cassandra 1.1.5 (doing a diff of cassandra.yaml to the 
new one first, to see whether anything important has changed)
7. start cassandra 1.1.5

In the log file, after the "Replaying ..." messages I find the following:

  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
759 mutations from unknown (probably removed) CF with id 1187
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
606 mutations from unknown (probably removed) CF with id 1186
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
53 mutations from unknown (probably removed) CF with id 1185
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1184
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1191
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
7506 mutations from unknown (probably removed) CF with id 1190
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
88 mutations from unknown (probably removed) CF with id 1189
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1188
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
354 mutations from unknown (probably removed) CF with id 1195
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1194
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
45 mutations from unknown (probably removed) CF with id 1192
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
82 mutations from unknown (probably removed) CF with id 1197
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
46386 mutations from unknown (probably removed) CF with id 1177
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
69 mutations from unknown (probably removed) CF with id 1178
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
73 mutations from unknown (probably removed) CF with id 1179
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
88 mutations from unknown (probably removed) CF with id 1181
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
46386 mutations from unknown (probably removed) CF with id 1182
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
7506 mutation

Re: Losing keyspace on cassandra upgrade

[Michael Kjellman]

@Edward Do you have a bug number for that by chance?

On Sep 19, 2012, at 8:25 AM, "Edward Sargisson" 
mailto:edward.sargis...@globalrelay.net>> 
wrote:

We've seen that before too - supposedly it was fixed in 1.1.5. Your experience 
casts some doubt on that.

Our workaround, thus far, is to shut down the entire ring and then bring each 
node back up starting with known good.
Then you do nodetool resetlocalschema on the node that's confused and make sure 
it gets the schema linked up properly.
Then nodetool repair.

I see you've done that but we found a complete ring restart was necessary. This 
was on Cass 1.1.1.

Cheers,
Edward

On 12-09-19 08:12 AM, Michael Kjellman wrote:

Sounds like you are loosing your system keyspace. When you say nothing 
important changed between yaml files do you mean with or without your changes?

Did your data directories change in the migration? Permissions okay?

I've done a 1.1.1 to 1.1.5 upgrade on many of my nodes without issue..

On Sep 19, 2012, at 7:44 AM, "Thomas Stets" 
 wrote:



I consistently keep losing my keyspace on upgrading from cassandra 1.1.1 to 
1.1.5

I have the same cassandra keyspace on all our staging systems:

development:  a 3-node cluster
integration: a 3-node cluster
QS: a 2-node cluster
(productive will be a 4-node cluster, which is as yet not active)

All clusters were running cassandra 1.1.1. Before going productive I wanted to 
upgrade to the
latest productive version of cassandra.

In all cases my keyspace disappeared when I started the cluster with cassandra 
1.1.5.
On the development system I didn't realize at first what was happening. I just 
wondered that nodetool
showed a very low amount of data. On integration I saw the problem quickly, but 
could not recover the
data. I re-installed the cassandra cluster from scratch, and populated it with 
our test data, so our
developers could work.

I am currently using the QS system to recreate the problem and try to find what 
I am doing wrong,
and how I can avoid losing productive data once we are live.

Basically I was doing the following:

1. create a snapshot on every node
2. create a tar.gz of my data directory, just to be safe
3. shut down and re-start cassandra 1.1.1 (just to see that it is not the 
re-start that is creating the problem)
4. verify that the keyspace is still known, and the data present.
5. shut down cassandra 1.1.1
6. copy the config to cassandra 1.1.5 (doing a diff of cassandra.yaml to the 
new one first, to see whether anything important has changed)
7. start cassandra 1.1.5

In the log file, after the "Replaying ..." messages I find the following:

 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
759 mutations from unknown (probably removed) CF with id 1187
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
606 mutations from unknown (probably removed) CF with id 1186
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
53 mutations from unknown (probably removed) CF with id 1185
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1184
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1191
 INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
7506 mutations from unknown (probably removed) CF with id 1190
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
88 mutations from unknown (probably removed) CF with id 1189
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1188
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
354 mutations from unknown (probably removed) CF with id 1195
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1194
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
45 mutations from unknown (probably removed) CF with id 1192
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
82 mutations from unknown (probably removed) CF with id 1197
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
46386 mutations from unknown (probably removed) CF with id 1177
 INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
69 mutations from unknown (probably removed) CF with id 1178
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
73 mutations from unknown (probably removed) CF with id 1179
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
88 mutations from unknown (probably removed) CF with id 1181
 INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) 

Re: Losing keyspace on cassandra upgrade

[Edward Sargisson]

https://issues.apache.org/jira/browse/CASSANDRA-4583

On 12-09-19 08:30 AM, Michael Kjellman wrote:

@Edward Do you have a bug number for that by chance?

On Sep 19, 2012, at 8:25 AM, "Edward Sargisson" 
mailto:edward.sargis...@globalrelay.net>> wrote:

We've seen that before too - supposedly it was fixed in 1.1.5. Your experience 
casts some doubt on that.

Our workaround, thus far, is to shut down the entire ring and then bring each 
node back up starting with known good.
Then you do nodetool resetlocalschema on the node that's confused and make sure 
it gets the schema linked up properly.
Then nodetool repair.

I see you've done that but we found a complete ring restart was necessary. This 
was on Cass 1.1.1.

Cheers,
Edward

On 12-09-19 08:12 AM, Michael Kjellman wrote:

Sounds like you are loosing your system keyspace. When you say nothing 
important changed between yaml files do you mean with or without your changes?

Did your data directories change in the migration? Permissions okay?

I've done a 1.1.1 to 1.1.5 upgrade on many of my nodes without issue..

On Sep 19, 2012, at 7:44 AM, "Thomas Stets" 
 wrote:



I consistently keep losing my keyspace on upgrading from cassandra 1.1.1 to 
1.1.5

I have the same cassandra keyspace on all our staging systems:

development:  a 3-node cluster
integration: a 3-node cluster
QS: a 2-node cluster
(productive will be a 4-node cluster, which is as yet not active)

All clusters were running cassandra 1.1.1. Before going productive I wanted to 
upgrade to the
latest productive version of cassandra.

In all cases my keyspace disappeared when I started the cluster with cassandra 
1.1.5.
On the development system I didn't realize at first what was happening. I just 
wondered that nodetool
showed a very low amount of data. On integration I saw the problem quickly, but 
could not recover the
data. I re-installed the cassandra cluster from scratch, and populated it with 
our test data, so our
developers could work.

I am currently using the QS system to recreate the problem and try to find what 
I am doing wrong,
and how I can avoid losing productive data once we are live.

Basically I was doing the following:

1. create a snapshot on every node
2. create a tar.gz of my data directory, just to be safe
3. shut down and re-start cassandra 1.1.1 (just to see that it is not the 
re-start that is creating the problem)
4. verify that the keyspace is still known, and the data present.
5. shut down cassandra 1.1.1
6. copy the config to cassandra 1.1.5 (doing a diff of cassandra.yaml to the 
new one first, to see whether anything important has changed)
7. start cassandra 1.1.5

In the log file, after the "Replaying ..." messages I find the following:

  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
759 mutations from unknown (probably removed) CF with id 1187
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
606 mutations from unknown (probably removed) CF with id 1186
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
53 mutations from unknown (probably removed) CF with id 1185
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1184
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
1945 mutations from unknown (probably removed) CF with id 1191
  INFO [main] 2012-09-19 15:15:50,323 CommitLogReplayer.java (line 103) Skipped 
7506 mutations from unknown (probably removed) CF with id 1190
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
88 mutations from unknown (probably removed) CF with id 1189
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1188
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
354 mutations from unknown (probably removed) CF with id 1195
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
87 mutations from unknown (probably removed) CF with id 1194
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
45 mutations from unknown (probably removed) CF with id 1192
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
82 mutations from unknown (probably removed) CF with id 1197
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
46386 mutations from unknown (probably removed) CF with id 1177
  INFO [main] 2012-09-19 15:15:50,324 CommitLogReplayer.java (line 103) Skipped 
69 mutations from unknown (probably removed) CF with id 1178
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
73 mutations from unknown (probably removed) CF with id 1179
  INFO [main] 2012-09-19 15:15:50,325 CommitLogReplayer.java (line 103) Skipped 
88 mutations fr

Re: higher layer library makes things faster?

[jeffpk]

Actually its not uncommon at all.  Any caching implemented on a higher level 
will generally improve speed at a cost in memory.

Beware common wisdom, its seldom very wise
Sent from my Verizon Wireless BlackBerry

-Original Message-
From: "Hiller, Dean" 
Date: Wed, 19 Sep 2012 07:35:07 
To: user@cassandra.apache.org
Reply-To: user@cassandra.apache.org
Subject: higher layer library makes things faster?

So there is this interesting case where a higher layer library makes things 
slower.  This is counter-intuitive as every abstraction usually makes things 
slower with an increase in productivity.It would be cool if more and more 
libraries supported something to help with this scenario I think.

The concept.   Once in a while you end up needing a new query into an noSQL 
data model that already exists, and you do something like this

UserRow user = fetchUser(rowKey);
List mappings = fetchRoleMappings(user.getRoleMappingIds())
List rowKeys = new ArrayList();
for(RoleMapping m : mappings) {
   rowKeys.addAll(m.getGroupIds());
}
List groups = fetchGroupRows(rowKeys);

It turns out if you index stuff, it is much faster in a lot of cases.  Instead 
you can scan just 2 index rows to find the keys for the groups and read them 
in.  Basically you do one scan on the RoleMapping where mapping has a FK to 
UserRow and you now have a list of primary keys for your RoleMapping.  Next, 
you do a scan of the GroupRow index feeding in the primary keys of your 
RoleMapping which feeds back your GroupRow primary keys that you need to 
lookup….in this way, you skip not only a lot of coding that goes into avoiding 
getting all the UserRow data back, and can simply scan the indexes.

That said, every time you update a row, you need to remove an old value from 
the index and a new one to the index.  Inserts only need to add.  Removes only 
need to remove.

Anyways, I have found this to be quite an interesting paradigm shift as right 
now doing the latter manually requires a lot more code (but then, that is why 
more and more libraries like playOrm I think will exist in the future as it 
makes the above very simple to do yourself).

Later,
Dean

Re: higher layer library makes things faster?

[Hiller, Dean]

I guess you could look at that as a form of cachingŠdidn't think of it at
the timeŠ.I usually think of it as caching in RAM, but this I guess is
caching on disk(though hopefully if the row cache is used for the 3 index
tables playOrm uses, it should be blazingly fast).

Dean

On 9/19/12 10:59 AM, "jef...@gmail.com"  wrote:

>Actually its not uncommon at all.  Any caching implemented on a higher
>level will generally improve speed at a cost in memory.
>
>Beware common wisdom, its seldom very wise
>Sent from my Verizon Wireless BlackBerry
>
>-Original Message-
>From: "Hiller, Dean" 
>Date: Wed, 19 Sep 2012 07:35:07
>To: user@cassandra.apache.org
>Reply-To: user@cassandra.apache.org
>Subject: higher layer library makes things faster?
>
>So there is this interesting case where a higher layer library makes
>things slower.  This is counter-intuitive as every abstraction usually
>makes things slower with an increase in productivity.It would be cool
>if more and more libraries supported something to help with this scenario
>I think.
>
>The concept.   Once in a while you end up needing a new query into an
>noSQL data model that already exists, and you do something like this
>
>UserRow user = fetchUser(rowKey);
>List mappings =
>fetchRoleMappings(user.getRoleMappingIds())
>List rowKeys = new ArrayList();
>for(RoleMapping m : mappings) {
>   rowKeys.addAll(m.getGroupIds());
>}
>List groups = fetchGroupRows(rowKeys);
>
>It turns out if you index stuff, it is much faster in a lot of cases.
>Instead you can scan just 2 index rows to find the keys for the groups
>and read them in.  Basically you do one scan on the RoleMapping where
>mapping has a FK to UserRow and you now have a list of primary keys for
>your RoleMapping.  Next, you do a scan of the GroupRow index feeding in
>the primary keys of your RoleMapping which feeds back your GroupRow
>primary keys that you need to lookupŠ.in this way, you skip not only a
>lot of coding that goes into avoiding getting all the UserRow data back,
>and can simply scan the indexes.
>
>That said, every time you update a row, you need to remove an old value
>from the index and a new one to the index.  Inserts only need to add.
>Removes only need to remove.
>
>Anyways, I have found this to be quite an interesting paradigm shift as
>right now doing the latter manually requires a lot more code (but then,
>that is why more and more libraries like playOrm I think will exist in
>the future as it makes the above very simple to do yourself).
>
>Later,
>Dean

Data stax community

[Marcelo Elias Del Valle]

Not sure if this question should be asked in this list, if this is the
wrong place to ask this, please tell me.

Does anyone know if Data Stax community edition alows us to run in
production? I plan to use the enterprise edition later, but for now even
for production I am thinking in using community version.

-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: Data stax community

[Abhijit Chanda]

You better ask this question
http://www.datastax.com/support-forums/forum/datastax-enterprise there. Any
ways as far as i am concern it should not be problematic thing.

Regards,
Abhijit

On Thu, Sep 20, 2012 at 12:07 AM, Marcelo Elias Del Valle <
mvall...@gmail.com> wrote:

> Not sure if this question should be asked in this list, if this is the
> wrong place to ask this, please tell me.
>
> Does anyone know if Data Stax community edition alows us to run in
> production? I plan to use the enterprise edition later, but for now even
> for production I am thinking in using community version.
>
> --
> Marcelo Elias Del Valle
> http://mvalle.com - @mvallebr
>



-- 
Abhijit Chanda
Software Developer
VeHere Interactive Pvt. Ltd.
+91-974395

Re: Data stax community

[Tyler Hobbs]

DataStax Community is free for any type of use, including production.

On Wed, Sep 19, 2012 at 1:42 PM, Abhijit Chanda
wrote:

> You better ask this question
> http://www.datastax.com/support-forums/forum/datastax-enterprise there.
> Any ways as far as i am concern it should not be problematic thing.
>
> Regards,
> Abhijit
>
>
> On Thu, Sep 20, 2012 at 12:07 AM, Marcelo Elias Del Valle <
> mvall...@gmail.com> wrote:
>
>> Not sure if this question should be asked in this list, if this is the
>> wrong place to ask this, please tell me.
>>
>> Does anyone know if Data Stax community edition alows us to run in
>> production? I plan to use the enterprise edition later, but for now even
>> for production I am thinking in using community version.
>>
>> --
>> Marcelo Elias Del Valle
>> http://mvalle.com - @mvallebr
>>
>
>
>
> --
> Abhijit Chanda
> Software Developer
> VeHere Interactive Pvt. Ltd.
> +91-974395
>
>


-- 
Tyler Hobbs
DataStax 

Re: Data stax community

[Marcelo Elias Del Valle]

Thanks!

2012/9/19 Tyler Hobbs 

> DataStax Community is free for any type of use, including production.
>
>
> On Wed, Sep 19, 2012 at 1:42 PM, Abhijit Chanda  > wrote:
>
>> You better ask this question
>> http://www.datastax.com/support-forums/forum/datastax-enterprise there.
>> Any ways as far as i am concern it should not be problematic thing.
>>
>> Regards,
>> Abhijit
>>
>>
>> On Thu, Sep 20, 2012 at 12:07 AM, Marcelo Elias Del Valle <
>> mvall...@gmail.com> wrote:
>>
>>> Not sure if this question should be asked in this list, if this is the
>>> wrong place to ask this, please tell me.
>>>
>>> Does anyone know if Data Stax community edition alows us to run in
>>> production? I plan to use the enterprise edition later, but for now even
>>> for production I am thinking in using community version.
>>>
>>> --
>>> Marcelo Elias Del Valle
>>> http://mvalle.com - @mvallebr
>>>
>>
>>
>>
>> --
>> Abhijit Chanda
>> Software Developer
>> VeHere Interactive Pvt. Ltd.
>> +91-974395
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>
>


-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Correct model

[Marcelo Elias Del Valle]

I am new to Cassandra and NoSQL at all.
I built my first model and any comments would be of great help. I am
describing my thoughts bellow.

It's a very simple model. I will need to store several users and, for each
user, I will need to store several requests. It request has it's insertion
time. As the query comes first, here are the only queries I will need to
run against this model:
- Select all the requests for an user
- Select all the users which has new requests, since date D

I created the following model: an UserCF, whose key is a userID generated
by TimeUUID, and a RequestCF, whose key is composite: UserUUID + timestamp.
For each user, I will store basic data and, for each request, I will insert
a lot of columns.

My questions:
- Is the strategy of using a composite key good for this case? I thought in
other solutions, but this one seemed to be the best. Another solution would
be have a non-composite key of type UUID for the requests, and have another
CF to relate user and request.
- To perform the second query, instead of selecting if each user has a
request inserted after date D, I thought in storing the last request
insertion date into the userCF, everytime I have a new insert for the user.
It would be a data replication, but I would have no read-before-write and I
am guessing the second query would perform faster.

Any thoughts?

-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: Correct model

[Hiller, Dean]

Thinking out loud and I think a bit towards playOrm's model though you don’t' 
need to use playroom for this.

1. I would probably have a User with the requests either embedded in or the 
Foreign keys to the requests…either is fine as long as you get the user get ALL 
FK's and make one request to get the requests for that user

2. I would create rows for index and index each month of data OR maybe index 
each day of data(depends on your system).  Then, I can just query into the 
index for that one month.  With playOrm S-SQL, this is a simple PARTITIONS 
r(:thismonthParititonId) SELECT r FROM Request r where r.date > :date OR you 
just do a column range query doing the same thing into your index.  The index 
is basically the wide row pattern ;) with composite keys of .

Later,
Dean

From: Marcelo Elias Del Valle mailto:mvall...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, September 19, 2012 1:02 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Correct model

I am new to Cassandra and NoSQL at all.
I built my first model and any comments would be of great help. I am describing 
my thoughts bellow.

It's a very simple model. I will need to store several users and, for each 
user, I will need to store several requests. It request has it's insertion 
time. As the query comes first, here are the only queries I will need to run 
against this model:
- Select all the requests for an user
- Select all the users which has new requests, since date D

I created the following model: an UserCF, whose key is a userID generated by 
TimeUUID, and a RequestCF, whose key is composite: UserUUID + timestamp. For 
each user, I will store basic data and, for each request, I will insert a lot 
of columns.

My questions:
- Is the strategy of using a composite key good for this case? I thought in 
other solutions, but this one seemed to be the best. Another solution would be 
have a non-composite key of type UUID for the requests, and have another CF to 
relate user and request.
- To perform the second query, instead of selecting if each user has a request 
inserted after date D, I thought in storing the last request insertion date 
into the userCF, everytime I have a new insert for the user. It would be a data 
replication, but I would have no read-before-write and I am guessing the second 
query would perform faster.

Any thoughts?

--
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: Correct model

[Marcelo Elias Del Valle]

2012/9/19 Hiller, Dean 

> Thinking out loud and I think a bit towards playOrm's model though you
> don’t' need to use playroom for this.
>
> 1. I would probably have a User with the requests either embedded in or
> the Foreign keys to the requests…either is fine as long as you get the user
> get ALL FK's and make one request to get the requests for that user
>

This was my first option. However, everytime I have a new request I would
need to read the column "request_ids", update its value, and them write the
result. This would be a read-before-write, which is bad in Cassandra,
right? Or you were talking about other kinds of FKs?


> 2. I would create rows for index and index each month of data OR maybe
> index each day of data(depends on your system).  Then, I can just query
> into the index for that one month.  With playOrm S-SQL, this is a simple
> PARTITIONS r(:thismonthParititonId) SELECT r FROM Request r where r.date >
> :date OR you just do a column range query doing the same thing into your
> index.  The index is basically the wide row pattern ;) with composite keys
> of .
>

I would consider playOrm in a later step in my project, as my understanding
now is it is good to store relational data, structured data. I cannot
predict which columns I am going to store in requestCF. But regardless,
even in Cassandra, you would still use a composite key, but it seems you
would create an indexCf using the wide row pattern, and each request would
have its own id, right? But why? Wouldn't it be faster to have a composite
key in the requestCF itself?


From: Marcelo Elias Del Valle mailto:mvall...@gmail.com
> >>
> Reply-To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Date: Wednesday, September 19, 2012 1:02 PM
> To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Subject: Correct model
>
> I am new to Cassandra and NoSQL at all.
> I built my first model and any comments would be of great help. I am
> describing my thoughts bellow.
>
> It's a very simple model. I will need to store several users and, for each
> user, I will need to store several requests. It request has it's insertion
> time. As the query comes first, here are the only queries I will need to
> run against this model:
> - Select all the requests for an user
> - Select all the users which has new requests, since date D
>
> I created the following model: an UserCF, whose key is a userID generated
> by TimeUUID, and a RequestCF, whose key is composite: UserUUID + timestamp.
> For each user, I will store basic data and, for each request, I will insert
> a lot of columns.
>
> My questions:
> - Is the strategy of using a composite key good for this case? I thought
> in other solutions, but this one seemed to be the best. Another solution
> would be have a non-composite key of type UUID for the requests, and have
> another CF to relate user and request.
> - To perform the second query, instead of selecting if each user has a
> request inserted after date D, I thought in storing the last request
> insertion date into the userCF, everytime I have a new insert for the user.
> It would be a data replication, but I would have no read-before-write and I
> am guessing the second query would perform faster.
>
> Any thoughts?
>
> --
> Marcelo Elias Del Valle
> http://mvalle.com - @mvallebr
>



-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re: Correct model

[Hiller, Dean]

Uhm, unless I am mistaken, a NEW request implies a new UUID so you can just 
write it to both the index to the request row and to the user that request was 
for all in one shot with no need to read, right?

(Also, read before write is not necessarily bad…it really depends on your 
situation but in this case, I don't think you need read before write).

For your structured data comment….
Actually playOrm stores structured and unstructured data.  It follows the 
pattern cassandra is adopting more and more of "partial" schemas and plans to 
hold to that path.  It is a complete break from JPA due to noSQL being so 
different.

and each request would have its own id, right

Yes, in my design, I choose each request with it's own id.

Wouldn't it be faster to have a composite key in the requestCF itself?

In CQL, don't you have to have an == in the first part of the clause meaning 
you would have to select the user id, BUT you wanted requests > date no matter 
which user so the indices I gave you have that information with a simple column 
slice of the data.  The indices I gave you look like this(composite column 
names)…. .., .., .. 
 NOTE that each is a UUID there in the <> so are unique.

Maybe there is a way, but I am not sure on how to get all the latest request > 
data for every user….I guess you could always map/reduce but that is generally 
reserved for analytics or maybe updating new index tables you are creating for 
reading faster.

Later,
Dean

From: Marcelo Elias Del Valle mailto:mvall...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, September 19, 2012 1:47 PM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Correct model

2012/9/19 Hiller, Dean mailto:dean.hil...@nrel.gov>>
Thinking out loud and I think a bit towards playOrm's model though you don’t' 
need to use playroom for this.

1. I would probably have a User with the requests either embedded in or the 
Foreign keys to the requests…either is fine as long as you get the user get ALL 
FK's and make one request to get the requests for that user

This was my first option. However, everytime I have a new request I would need 
to read the column "request_ids", update its value, and them write the result. 
This would be a read-before-write, which is bad in Cassandra, right? Or you 
were talking about other kinds of FKs?

2. I would create rows for index and index each month of data OR maybe index 
each day of data(depends on your system).  Then, I can just query into the 
index for that one month.  With playOrm S-SQL, this is a simple PARTITIONS 
r(:thismonthParititonId) SELECT r FROM Request r where r.date > :date OR you 
just do a column range query doing the same thing into your index.  The index 
is basically the wide row pattern ;) with composite keys of .

I would consider playOrm in a later step in my project, as my understanding now 
is it is good to store relational data, structured data. I cannot predict which 
columns I am going to store in requestCF. But regardless, even in Cassandra, 
you would still use a composite key, but it seems you would create an indexCf 
using the wide row pattern, and each request would have its own id, right? But 
why? Wouldn't it be faster to have a composite key in the requestCF itself?


From: Marcelo Elias Del Valle 
mailto:mvall...@gmail.com>>>
Reply-To: 
"user@cassandra.apache.org>"
 
mailto:user@cassandra.apache.org>>>
Date: Wednesday, September 19, 2012 1:02 PM
To: 
"user@cassandra.apache.org>"
 
mailto:user@cassandra.apache.org>>>
Subject: Correct model

I am new to Cassandra and NoSQL at all.
I built my first model and any comments would be of great help. I am describing 
my thoughts bellow.

It's a very simple model. I will need to store several users and, for each 
user, I will need to store several requests. It request has it's insertion 
time. As the query comes first, here are the only queries I will need to run 
against this model:
- Select all the requests for an user
- Select all the users which has new requests, since date D

I created the following model: an UserCF, whose key is a userID generated by 
TimeUUID, and a RequestCF, whose key is composite: UserUUID + timestamp. For 
each user, I will store basic data and, for each request, I will insert a lot 
of columns.

My questions:
- Is the strategy of using a composite key good for this case? I thought in 
other solutions, but this one seemed to be the best. Another solution would be 
have a non-composite k

Re: Correct model

[Hiller, Dean]

Oh, quick correction, I was thinking your user row key was in the request
coming in from your first email.

In your first email, you get a request and seem to shove it and a user in
generating the ids which means that user never generates a request ever
again???  If a user sends multiple requests in, how are you looking up his
TimeUUID row key from your first email(I would do the same in my
implementation)?

If you had an ldap unique username, I would just use that as the primary
key meaning you NEVER have to do reads.  If you have a username and need
to lookup a UUID, you would have to do that in both implementationsŠnot a
real big deal thoughŠa quick quick lookup table does the trick there and
in most cases is still fast enough(ie. Read before write here is ok in a
lot of cases).

That X-ref table would simple be rowkey=username and value=users real
primary key

Though again, we use ldap and know no one's username is really going to
change so username is our primary key.

Later,
Dean


On 9/19/12 2:33 PM, "Hiller, Dean"  wrote:

>Uhm, unless I am mistaken, a NEW request implies a new UUID so you can
>just write it to both the index to the request row and to the user that
>request was for all in one shot with no need to read, right?
>
>(Also, read before write is not necessarily badŠit really depends on your
>situation but in this case, I don't think you need read before write).
>
>For your structured data commentŠ.
>Actually playOrm stores structured and unstructured data.  It follows the
>pattern cassandra is adopting more and more of "partial" schemas and
>plans to hold to that path.  It is a complete break from JPA due to noSQL
>being so different.
>
>and each request would have its own id, right
>
>Yes, in my design, I choose each request with it's own id.
>
>Wouldn't it be faster to have a composite key in the requestCF itself?
>
>In CQL, don't you have to have an == in the first part of the clause
>meaning you would have to select the user id, BUT you wanted requests >
>date no matter which user so the indices I gave you have that information
>with a simple column slice of the data.  The indices I gave you look like
>this(composite column names)Š. ..,
>.., ..  NOTE that each is a UUID
>there in the <> so are unique.
>
>Maybe there is a way, but I am not sure on how to get all the latest
>request > data for every userŠ.I guess you could always map/reduce but
>that is generally reserved for analytics or maybe updating new index
>tables you are creating for reading faster.
>
>Later,
>Dean
>
>From: Marcelo Elias Del Valle
>mailto:mvall...@gmail.com>>
>Reply-To: "user@cassandra.apache.org"
>mailto:user@cassandra.apache.org>>
>Date: Wednesday, September 19, 2012 1:47 PM
>To: "user@cassandra.apache.org"
>mailto:user@cassandra.apache.org>>
>Subject: Re: Correct model
>
>2012/9/19 Hiller, Dean mailto:dean.hil...@nrel.gov>>
>Thinking out loud and I think a bit towards playOrm's model though you
>don¹t' need to use playroom for this.
>
>1. I would probably have a User with the requests either embedded in or
>the Foreign keys to the requestsŠeither is fine as long as you get the
>user get ALL FK's and make one request to get the requests for that user
>
>This was my first option. However, everytime I have a new request I would
>need to read the column "request_ids", update its value, and them write
>the result. This would be a read-before-write, which is bad in Cassandra,
>right? Or you were talking about other kinds of FKs?
>
>2. I would create rows for index and index each month of data OR maybe
>index each day of data(depends on your system).  Then, I can just query
>into the index for that one month.  With playOrm S-SQL, this is a simple
>PARTITIONS r(:thismonthParititonId) SELECT r FROM Request r where r.date
>> :date OR you just do a column range query doing the same thing into
>>your index.  The index is basically the wide row pattern ;) with
>>composite keys of .
>
>I would consider playOrm in a later step in my project, as my
>understanding now is it is good to store relational data, structured
>data. I cannot predict which columns I am going to store in requestCF.
>But regardless, even in Cassandra, you would still use a composite key,
>but it seems you would create an indexCf using the wide row pattern, and
>each request would have its own id, right? But why? Wouldn't it be faster
>to have a composite key in the requestCF itself?
>
>
>From: Marcelo Elias Del Valle
>mailto:mvall...@gmail.com>>>
>Reply-To: 
>"user@cassandra.apache.orgassandra.apache.org>"
>mailto:user@cassandra.apache.org>assandra.apache.org>>
>Date: Wednesday, September 19, 2012 1:02 PM
>To: 
>"user@cassandra.apache.orgassandra.apache.o

Re: downgrade from 1.1.4 to 1.0.X

[aaron morton]

No. 
They use different minor file versions which are not backwards compatible. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 18/09/2012, at 11:18 PM, Arend-Jan Wijtzes  wrote:

> Hi,
> 
> We are running Cassandra 1.1.4 and like to experiment with
> Datastax Enterprise which uses 1.0.8. Can we safely downgrade
> a production cluster or is it incompatible? Any special steps
> involved?
> 
> Arend-Jan
> 
> -- 
> Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl

Re: Query advice to prevent node overload

[aaron morton]

> Wouldn't that return files from directories '/tmp1', '/tmp2', for example?
I believe so. 

> I thought the goal was to return files and subdirectories recursively inside 
> '/tmp'.
I'm not sure what the purpose of the query was. 
The query query will return inodes where the file path starts with "/tmp"
 
> select filename from inode where filename >= ‘/tmp/’ and filename < ‘/tmp0’ 
> and sentinel = ‘x’;
> 
> Would this work?
Sounds ok. 

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 18/09/2012, at 11:41 PM, André Cruz  wrote:

> On Sep 18, 2012, at 3:06 AM, aaron morton  wrote:
> 
>>> select filename from inode where filename > ‘/tmp’ and filename < ‘/tmq’ 
>>> and sentinel = ‘x’;
> 
> Wouldn't that return files from directories '/tmp1', '/tmp2', for example? I 
> thought the goal was to return files and subdirectories recursively inside 
> '/tmp'.
> 
> If we store the slash with the parent_path, so '/tmp/', maybe we can search 
> for:
> 
> select filename from inode where filename >= ‘/tmp/’ and filename < ‘/tmp0’ 
> and sentinel = ‘x’;
> 
> Would this work?
> 
> Thanks,
> André
> 
> 

Re: HTimedOutException and cluster not working

[aaron morton]

> No, all keyspaces that we created do not have secondary indexes. So probably 
> the settings 'memtable_flush_queue_size' is not relevant? 
It may be. 
If you had a lot a CF's and cassandra tried to flush more than 
memtable_flush_queue_size at once. 

> one would think that the compaction would probably hitting
> the limit I/O of the system and hence, the FlushWriter has to be block.
Maybe sort of. 
The flush writer will block if it cannot flush data fast enough. So it depends 
on the number of CF's being flushed and the speed it is written it. It's 
unlikely that compaction is hurting that much on one machine. 

> nodetool has a command 'setcompactionthroughput', would setting
> the throughput to a lower number to a number lower than 16 helps?
Cannot hurt to try. 

> What other methods to find out if all the nodes in the system is in OK or 
> problem from cassandra point of view? 
Monitoring of whatever flavour. e.g. http://www.datastax.com/products/opscenter

Look in the logs for nodes thinking node-3 is DOWN

> when the logging in cassandra is enable with DEBUG, here is the log I got for 
> node-1
This is node-1 failing to write hints to node-3. 


I forgot to ask, what consistency level are you using for writes ? 
Have you checked the disk health on node-3 ?

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 19/09/2012, at 1:10 AM, Jason Wee  wrote:

> Hi Aaron, thank you for your reply. Please read inline comment.
> 
> On Tue, Sep 18, 2012 at 7:36 PM, aaron morton  wrote:
> What version are you on ?
> cassandra version 1.0.8 
> 
>>  HTimedOutException is logged for all the nodes. 
> TimedOutException happens when less than CL replica nodes respond to the 
> coordinator in time. 
> You could get the error from all nodes in your cluster if the 3 nodes that 
> store the key are having problems. 
> we have check the system load and tpstats for all the nodes in the cluster. 
> Only node 3 having high active and pending 
> count for MutationStage and FlushWriter. 
> 
> 
>> MutationStage 16 2177067 879092633 0 0
> This looks like mutations are blocked or running very very slowly. 
> 
>> FlushWriter 0 0 5616 0 1321
> The All Timed Blocked number means there were 1,321 times a thread tried to 
> flush a memtable but the queue of flushers was full. Do you use secondary 
> indexes ? If so take a look at the comments for memtable_flush_queue_size in 
> the yaml file. 
> 
> No, all keyspaces that we created do not have secondary indexes. So probably 
> the settings 'memtable_flush_queue_size' is not relevant? 
> 
> What I interpret based on the output as above and the output of 
> compactionstats on node-3, one would think that the compaction would probably 
> hitting
> the limit I/O of the system and hence, the FlushWriter has to be block. I 
> read further, nodetool has a command 'setcompactionthroughput', would setting
> the throughput to a lower number to a number lower than 16 helps?
>>   and cluster settings, it should be possible in this scenario, write 
>> success 
>>   on one of the nodes even though node-3 is too busy or failing for any 
>> reason? 
> Yes. 
> If only one node fails to respond the write should succeed. If you got a 
> TimedOut with CL ONE it sounds like more nodes were having problems. 
> We check the nodes by running command like top or nodetool cfstats on all the 
> nodes in the cluster. The load for the rest of the nodes all below 1 and 
> there is no
> active or pending count for all the stages. What other methods to find out if 
> all the nodes in the system is in OK or problem from cassandra point of view? 
> Because 
> if there is a concrete method to gauge all the nodes, we will have better 
> clues where exactly is the problem.
> 
>> * when hector client failover to other nodes, basically all the nodes fail, 
>> why
>>   is this so?
> Sorry I don't understand this question. 
> 
> 14.Sep 12:14:04,430 WARN HConnectionManager [operateWithFailover]: Could not 
> fullfill request on this host CassandraClient
> 14.Sep 12:14:04,430 WARN HConnectionManager [operateWithFailover]: Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at 
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
> at 
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:252)
> ..
> ..
> ..
> 14.Sep 12:14:04,302 WARN HConnectionManager [operateWithFailover]: Could not 
> fullfill request on this host CassandraClient
> 14.Sep 12:14:04,302 WARN HConnectionManager [operateWithFailover]: Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at 
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:42)
> at 
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:252)
>  
> 
> the log show that simi

Re: updating CF from a mapper-only Hadoop job

[aaron morton]

> That job would consistently fail with a flurry of exceptions 
What were the exceptions ?

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 19/09/2012, at 2:16 AM, Brian Jeltema  wrote:

> I wrote a Hadoop mapper-only job that uses BulkOutputFormat to load a 
> Cassandra table.
> That job would consistently fail with a flurry of exceptions (primary cause 
> looks like EOFExceptions
> streaming between nodes).
> 
> I restructured the job to use an identity mapper and perform the updates in 
> the reducer. Same code other
> be moving to the reducer. That job runs correctly.
> 
> Is this expected behavior?
> 
> TIA
> 
> Brian

Re: Correct model

[Marcelo Elias Del Valle]

>
> In your first email, you get a request and seem to shove it and a user in
> generating the ids which means that user never generates a request ever
> again???  If a user sends multiple requests in, how are you looking up his
> TimeUUID row key from your first email(I would do the same in my
> implementation)?
>

Actually, I don't get it from Cassandra. I am using Cassandra for the
writes, but to find the userId I look on a pre-indexed structure, because I
think the reads would be faster this way. I need to find the userId by some
key fields, so I use an index like this:

user ID 5596 -> { name -> "john denver", phone -> " ", field3 ->
"field 3 data", field 10 -> "field 10 data"}

The values are just examples. This part is not implemented yet and I am
looking for alternatives. Currently we have some similar indexes in SOLR,
but we are thinking in keeping the index in memory and replicating manually
in the cluster, or using Voldemort, etc.
I might be wrong, but I think Cassandra is great for writes, but a solution
like this would be better for reads.



> If you had an ldap unique username, I would just use that as the primary
> key meaning you NEVER have to do reads.  If you have a username and need
> to lookup a UUID, you would have to do that in both implementationsŠnot a
> real big deal thoughŠa quick quick lookup table does the trick there and
> in most cases is still fast enough(ie. Read before write here is ok in a
> lot of cases).
>
> That X-ref table would simple be rowkey=username and value=users real
> primary key
>
> Though again, we use ldap and know no one's username is really going to
> change so username is our primary key.
>

In my case, a single user can have thousands of requests. In my userCF, I
will have just 1 user with uuid X, but I am not sure about what to have in
my requestCF.

-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Re:

[Michael Kjellman]

A few questions: what version of 1.1 are you running. What version of Hadoop?

What is your job config? What is the buffer size you've chosen? How much data 
are you dealing with?

On Sep 19, 2012, at 7:23 PM, "Manu Zhang"  wrote:

> I've been bulk loading data into Cassandra and seen the following exception:
> 
> ERROR 10:10:31,032 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.RuntimeException: Last written key DecoratedKey(-442063125946754, 
> 313130303136373a31) >= current key DecoratedKey(-465541023623745, 
> 313036393331333a33) writing into 
> /home/manuzhang/cassandra/data/tpch/lineitem/tpch-lineitem-tmp-ia-56-Data.db
>   at 
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:131)
>   at 
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:152)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:169)
>   at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$1.run(CompactionManager.java:152)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>   at java.lang.Thread.run(Thread.java:722)
> 
> The running Cassandra and that I load data into are the same one.
> 
> What's the cause?

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re:

[Manu Zhang]

cassandra-trunk (so it's 1.2); no Hadoop, bulk load example here
http://www.datastax.com/dev/blog/bulk-loading#comment-127019; buffer size
is 64 MB as in the example; I'm dealing with about 1GB data. job config,
you mean?

On Thu, Sep 20, 2012 at 10:32 AM, Michael Kjellman
wrote:

> A few questions: what version of 1.1 are you running. What version of
> Hadoop?
>
> What is your job config? What is the buffer size you've chosen? How much
> data are you dealing with?
>
> On Sep 19, 2012, at 7:23 PM, "Manu Zhang"  wrote:
>
> > I've been bulk loading data into Cassandra and seen the following
> exception:
> >
> > ERROR 10:10:31,032 Exception in thread
> Thread[CompactionExecutor:5,1,main]
> > java.lang.RuntimeException: Last written key
> DecoratedKey(-442063125946754, 313130303136373a31) >= current key
> DecoratedKey(-465541023623745, 313036393331333a33) writing into
> /home/manuzhang/cassandra/data/tpch/lineitem/tpch-lineitem-tmp-ia-56-Data.db
> >   at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:131)
> >   at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:152)
> >   at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:169)
> >   at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> >   at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> >   at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
> >   at
> org.apache.cassandra.db.compaction.CompactionManager$1.run(CompactionManager.java:152)
> >   at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >   at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >   at java.lang.Thread.run(Thread.java:722)
> >
> > The running Cassandra and that I load data into are the same one.
> >
> > What's the cause?
>
> 'Like' us on Facebook for exclusive content and other resources on all
> Barracuda Networks solutions.
>
> Visit http://barracudanetworks.com/facebook
>
>
>
>
>

Re:

[Michael Kjellman]

I assumed you were talking about BulkLoader. I haven't played with trunk yet so 
I'm afraid I won't be much help here...

On Sep 19, 2012, at 7:56 PM, "Manu Zhang" 
mailto:owenzhang1...@gmail.com>> wrote:

cassandra-trunk (so it's 1.2); no Hadoop, bulk load example here 
http://www.datastax.com/dev/blog/bulk-loading#comment-127019; buffer size is 64 
MB as in the example; I'm dealing with about 1GB data. job config, you mean?

On Thu, Sep 20, 2012 at 10:32 AM, Michael Kjellman 
mailto:mkjell...@barracuda.com>> wrote:
A few questions: what version of 1.1 are you running. What version of Hadoop?

What is your job config? What is the buffer size you've chosen? How much data 
are you dealing with?

On Sep 19, 2012, at 7:23 PM, "Manu Zhang" 
mailto:owenzhang1...@gmail.com>> wrote:

> I've been bulk loading data into Cassandra and seen the following exception:
>
> ERROR 10:10:31,032 Exception in thread Thread[CompactionExecutor:5,1,main]
> java.lang.RuntimeException: Last written key DecoratedKey(-442063125946754, 
> 313130303136373a31) >= current key DecoratedKey(-465541023623745, 
> 313036393331333a33) writing into 
> /home/manuzhang/cassandra/data/tpch/lineitem/tpch-lineitem-tmp-ia-56-Data.db
>   at 
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:131)
>   at 
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:152)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:169)
>   at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>   at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>   at 
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
>   at 
> org.apache.cassandra.db.compaction.CompactionManager$1.run(CompactionManager.java:152)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>   at java.lang.Thread.run(Thread.java:722)
>
> The running Cassandra and that I load data into are the same one.
>
> What's the cause?

'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.

Visit http://barracudanetworks.com/facebook






'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re:

[Manu Zhang]

Yeah, BulkLoader. You did help me to elaborate my question. Thanks!

On Thu, Sep 20, 2012 at 10:58 AM, Michael Kjellman
wrote:

> I assumed you were talking about BulkLoader. I haven't played with trunk
> yet so I'm afraid I won't be much help here...
>
> On Sep 19, 2012, at 7:56 PM, "Manu Zhang"  owenzhang1...@gmail.com>> wrote:
>
> cassandra-trunk (so it's 1.2); no Hadoop, bulk load example here
> http://www.datastax.com/dev/blog/bulk-loading#comment-127019; buffer size
> is 64 MB as in the example; I'm dealing with about 1GB data. job config,
> you mean?
>
> On Thu, Sep 20, 2012 at 10:32 AM, Michael Kjellman <
> mkjell...@barracuda.com> wrote:
> A few questions: what version of 1.1 are you running. What version of
> Hadoop?
>
> What is your job config? What is the buffer size you've chosen? How much
> data are you dealing with?
>
> On Sep 19, 2012, at 7:23 PM, "Manu Zhang"  owenzhang1...@gmail.com>> wrote:
>
> > I've been bulk loading data into Cassandra and seen the following
> exception:
> >
> > ERROR 10:10:31,032 Exception in thread
> Thread[CompactionExecutor:5,1,main]
> > java.lang.RuntimeException: Last written key
> DecoratedKey(-442063125946754, 313130303136373a31) >= current key
> DecoratedKey(-465541023623745, 313036393331333a33) writing into
> /home/manuzhang/cassandra/data/tpch/lineitem/tpch-lineitem-tmp-ia-56-Data.db
> >   at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:131)
> >   at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:152)
> >   at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:169)
> >   at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> >   at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> >   at
> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
> >   at
> org.apache.cassandra.db.compaction.CompactionManager$1.run(CompactionManager.java:152)
> >   at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >   at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >   at java.lang.Thread.run(Thread.java:722)
> >
> > The running Cassandra and that I load data into are the same one.
> >
> > What's the cause?
>
> 'Like' us on Facebook for exclusive content and other resources on all
> Barracuda Networks solutions.
>
> Visit http://barracudanetworks.com/facebook
>
>
>
>
>
>
> 'Like' us on Facebook for exclusive content and other resources on all
> Barracuda Networks solutions.
> Visit http://barracudanetworks.com/facebook
>
>
>

Re: HTimedOutException and cluster not working

[Jason Wee]

Hi Aaron, thank you for comment.

>It may be.
>If you had a lot a CF's and cassandra tried to flush more than
memtable_flush_queue_size at once.

We created 6 keyspaces where maximum CF in a keyspace is 5 CF. The setting
'memtable_flush_queue_size' is using the default value which is 4.

>Maybe sort of.
>The flush writer will block if it cannot flush data fast enough. So it
depends on the number of CF's being flushed and the speed it is written it.
It's unlikely that compaction is hurting that much on one machine.

Good comment. I managed to go through the log in node-3 and found out
GCInspector try to free up the heap size. It continue flushing a lot of CFs
as shown in the log and then no more showing flushing cf in the log but the
following two lines continue to shown until we restart node-3 as that time,
the hector client showing HTimedOutException and fail on all the nodes.

ERROR [Selector-Thread-1] 2012-09-14 07:43:07,795 TNonblockingServer.java
(line 467) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?
ERROR [Selector-Thread-0] 2012-09-14 07:43:08,311 TNonblockingServer.java
(line 467) Read an invalid frame size of 0. Are you using TFramedTransport
on the client side?

>Cannot hurt to try.

Thanks, will start with "iostat -x -m 1" and use the output iostat, value
in column wMB/s as a start to set.


>Monitoring of whatever flavour. e.g.
http://www.datastax.com/products/opscenter
>Look in the logs for nodes thinking node-3 is DOWN

Yes, we do have opscenter monitoring for the cluster. what we noticed in
opscenter, node-3 is shown doing compaction and it is up.
In the opscenter performance metrics, graph 'Write Request Latency' spike
to maximum 453652.23us/op where usually the write latency hovering on
average at 3000us/op.




>I forgot to ask, what consistency level are you using for writes ?
>Have you checked the disk health on node-3 ?

consistency level for write? mentioned in the first thread, we are using CL
ONE. But if you mean read CL? for read, we are using CL QUORUM.
no, will install smartmontools.


On Thu, Sep 20, 2012 at 6:56 AM, aaron morton wrote:

>
> No, all keyspaces that we created do not have secondary indexes. So
> probably the settings 'memtable_flush_queue_size' is not relevant?
>
> It may be.
> If you had a lot a CF's and cassandra tried to flush more than
> memtable_flush_queue_size at once.
>
> one would think that the compaction would probably hitting
> the limit I/O of the system and hence, the FlushWriter has to be block.
>
> Maybe sort of.
> The flush writer will block if it cannot flush data fast enough. So it
> depends on the number of CF's being flushed and the speed it is written it.
> It's unlikely that compaction is hurting that much on one machine.
>
> nodetool has a command 'setcompactionthroughput', would setting
> the throughput to a lower number to a number lower than 16 helps?
>
> Cannot hurt to try.
>
> What other methods to find out if all the nodes in the system is in OK or
> problem from cassandra point of view?
>
> Monitoring of whatever flavour. e.g.
> http://www.datastax.com/products/opscenter
>
> Look in the logs for nodes thinking node-3 is DOWN
>
> when the logging in cassandra is enable with DEBUG, here is the log I got
> for node-1
>
> This is node-1 failing to write hints to node-3.
>
>
> I forgot to ask, what consistency level are you using for writes ?
> Have you checked the disk health on node-3 ?
>
> Cheers
>
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/09/2012, at 1:10 AM, Jason Wee  wrote:
>
> Hi Aaron, thank you for your reply. Please read inline comment.
>
> On Tue, Sep 18, 2012 at 7:36 PM, aaron morton wrote:
>
>> What version are you on ?
>>
> cassandra version 1.0.8
>
>>
>>  HTimedOutException is logged for all the nodes.
>>
>> TimedOutException happens when less than CL replica nodes respond to the
>> coordinator in time.
>> You could get the error from all nodes in your cluster if the 3 nodes
>> that store the key are having problems.
>>
> we have check the system load and tpstats for all the nodes in the
> cluster. Only node 3 having high active and pending
> count for MutationStage and FlushWriter.
>
>
>> MutationStage 16 2177067 879092633 0 0
>>
>> This looks like mutations are blocked or running very very slowly.
>>
>> FlushWriter 0 0 5616 0 1321
>>
>> The All Timed Blocked number means there were 1,321 times a thread tried
>> to flush a memtable but the queue of flushers was full. Do you use
>> secondary indexes ? If so take a look at the comments for
>> memtable_flush_queue_size in the yaml file.
>>
>> No, all keyspaces that we created do not have secondary indexes. So
> probably the settings 'memtable_flush_queue_size' is not relevant?
>
> What I interpret based on the output as above and the output of
> compactionstats on node-3, one would think that the compaction would
> probably hitt

Re: persistent compaction issue (1.1.4 and 1.1.5)

[Віталій Тимчишин]

I did see problems with schema agreement on 1.1.4, but they did go away
after rolling restart (BTW: it would be still good to check describe schema
for unreachable). Same rolling restart helped to force compactions after
moving to Leveled compaction. If your compactions still don't go, you can
try removing *.json files from the data directory of the stopped node to
force moving all SSTables to level0.

Best regards, Vitalii Tymchyshyn

2012/9/19 Michael Kjellman 

> Potentially the pending compactions are a symptom and not the root
> cause/problem.
>
> When updating a 3rd column family with a larger sstable_size_in_mb it
> looks like the schema may not be in a good state
>
> [default@] UPDATE COLUMN FAMILY screenshots WITH
> compaction_strategy=LeveledCompactionStrategy AND
> compaction_strategy_options={sstable_size_in_mb: 200};
> 290cf619-57b0-3ad1-9ae3-e313290de9c9
> Waiting for schema agreement...
> Warning: unreachable nodes 10.8.30.102The schema has not settled in 10
> seconds; further migrations are ill-advised until it does.
> Versions are UNREACHABLE:[10.8.30.102],
> 290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13,
> 10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106],
> f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101]
>
>
> However, tpstats looks good. And the schema changes eventually do get
> applied on *all* the nodes (even the ones that seem to have different
> schema versions). There are no communications issues between the nodes and
> they are all in the same rack
>
> root@:~# nodetool tpstats
> Pool NameActive   Pending  Completed   Blocked
> All time blocked
> ReadStage 0 01254592 0
> 0
> RequestResponseStage  0 09480827 0
> 0
> MutationStage 0 08662263 0
> 0
> ReadRepairStage   0 0 339158 0
> 0
> ReplicateOnWriteStage 0 0  0 0
> 0
> GossipStage   0 01469197 0
> 0
> AntiEntropyStage  0 0  0 0
> 0
> MigrationStage0 0   1808 0
> 0
> MemtablePostFlusher   0 0248 0
> 0
> StreamStage   0 0  0 0
> 0
> FlushWriter   0 0248 0
> 4
> MiscStage 0 0  0 0
> 0
> commitlog_archiver0 0  0 0
> 0
> InternalResponseStage 0 0   5286 0
> 0
> HintedHandoff 0 0 21 0
> 0
>
> Message type   Dropped
> RANGE_SLICE  0
> READ_REPAIR  0
> BINARY   0
> READ 0
> MUTATION 0
> REQUEST_RESPONSE 0
>
> So I'm guessing maybe the different schema versions may be potentially
> stopping compactions? Will compactions still happen if there are different
> versions of the schema?
>
>
>
>
>
> On 9/18/12 11:38 AM, "Michael Kjellman"  wrote:
>
> >Thanks, I just modified the schema on the worse offending column family
> >(as determined by the .json) from 10MB to 200MB.
> >
> >Should I kick off a compaction on this cf now/repair?/scrub?
> >
> >Thanks
> >
> >-michael
> >
> >From: Віталій Тимчишин mailto:tiv...@gmail.com>>
> >Reply-To: "user@cassandra.apache.org"
> >mailto:user@cassandra.apache.org>>
> >To: "user@cassandra.apache.org"
> >mailto:user@cassandra.apache.org>>
> >Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)
> >
> >I've started to use LeveledCompaction some time ago and from my
> >experience this indicates some SST on lower levels than they should be.
> >The compaction is going, moving them up level by level, but total count
> >does not change as new data goes in.
> >The numbers are pretty high as for me. Such numbers mean a lot of files
> >(over 100K in single directory) and a lot of thinking for compaction
> >executor to decide what to compact next. I can see numbers like 5K-10K
> >and still thing this is high number. If I were you, I'd increase
> >sstable_size_in_mb 10-20 times it is now.
> >
> >2012/9/17 Michael Kjellman
> >mailto:mkjell...@barracuda.com>>
> >Hi All,
> >
> >I have an issue where each one of my nodes (currently all running at
> >1.1.5) is reporting around 30,000 pending compactions. I understand that
> >a pending compaction doesn't necessarily mean it is a scheduled task
> >however I'm confused why this behavior is occurring. It is the 

Re:

[Manu Zhang]

the problem seems to have gone away with changing Murmur3Partitioner back
to RandomPartitioner

On Thu, Sep 20, 2012 at 11:14 AM, Manu Zhang wrote:

> Yeah, BulkLoader. You did help me to elaborate my question. Thanks!
>
>
> On Thu, Sep 20, 2012 at 10:58 AM, Michael Kjellman <
> mkjell...@barracuda.com> wrote:
>
>> I assumed you were talking about BulkLoader. I haven't played with trunk
>> yet so I'm afraid I won't be much help here...
>>
>> On Sep 19, 2012, at 7:56 PM, "Manu Zhang" > > wrote:
>>
>> cassandra-trunk (so it's 1.2); no Hadoop, bulk load example here
>> http://www.datastax.com/dev/blog/bulk-loading#comment-127019; buffer
>> size is 64 MB as in the example; I'm dealing with about 1GB data. job
>> config, you mean?
>>
>> On Thu, Sep 20, 2012 at 10:32 AM, Michael Kjellman <
>> mkjell...@barracuda.com> wrote:
>> A few questions: what version of 1.1 are you running. What version of
>> Hadoop?
>>
>> What is your job config? What is the buffer size you've chosen? How much
>> data are you dealing with?
>>
>> On Sep 19, 2012, at 7:23 PM, "Manu Zhang" > > wrote:
>>
>> > I've been bulk loading data into Cassandra and seen the following
>> exception:
>> >
>> > ERROR 10:10:31,032 Exception in thread
>> Thread[CompactionExecutor:5,1,main]
>> > java.lang.RuntimeException: Last written key
>> DecoratedKey(-442063125946754, 313130303136373a31) >= current key
>> DecoratedKey(-465541023623745, 313036393331333a33) writing into
>> /home/manuzhang/cassandra/data/tpch/lineitem/tpch-lineitem-tmp-ia-56-Data.db
>> >   at
>> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:131)
>> >   at
>> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:152)
>> >   at
>> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:169)
>> >   at
>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>> >   at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>> >   at
>> org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:69)
>> >   at
>> org.apache.cassandra.db.compaction.CompactionManager$1.run(CompactionManager.java:152)
>> >   at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >   at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >   at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >   at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >   at java.lang.Thread.run(Thread.java:722)
>> >
>> > The running Cassandra and that I load data into are the same one.
>> >
>> > What's the cause?
>>
>> 'Like' us on Facebook for exclusive content and other resources on all
>> Barracuda Networks solutions.
>>
>> Visit http://barracudanetworks.com/facebook
>>
>>
>>
>>
>>
>>
>> 'Like' us on Facebook for exclusive content and other resources on all
>> Barracuda Networks solutions.
>> Visit http://barracudanetworks.com/facebook
>>
>>
>>
>

Re: persistent compaction issue (1.1.4 and 1.1.5)

[Michael Kjellman]

After changing my ss_table_size as recommended my pending compactions across 
the cluster have leveled off at 34808 but it isn't progressing after 24 hours 
at that level.

As I've already changed the most offending column families I think the only 
option I have left is to remove the .json files from all of the column families 
and do another rolling restart...

Developing... Thanks for the help so far

On Sep 19, 2012, at 10:35 PM, "Віталій Тимчишин" 
mailto:tiv...@gmail.com>> wrote:

I did see problems with schema agreement on 1.1.4, but they did go away after 
rolling restart (BTW: it would be still good to check describe schema for 
unreachable). Same rolling restart helped to force compactions after moving to 
Leveled compaction. If your compactions still don't go, you can try removing 
*.json files from the data directory of the stopped node to force moving all 
SSTables to level0.

Best regards, Vitalii Tymchyshyn

2012/9/19 Michael Kjellman 
mailto:mkjell...@barracuda.com>>
Potentially the pending compactions are a symptom and not the root
cause/problem.

When updating a 3rd column family with a larger sstable_size_in_mb it
looks like the schema may not be in a good state

[default@] UPDATE COLUMN FAMILY screenshots WITH
compaction_strategy=LeveledCompactionStrategy AND
compaction_strategy_options={sstable_size_in_mb: 200};
290cf619-57b0-3ad1-9ae3-e313290de9c9
Waiting for schema agreement...
Warning: unreachable nodes 10.8.30.102The schema has not settled in 10
seconds; further migrations are ill-advised until it does.
Versions are UNREACHABLE:[10.8.30.102],
290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13,
10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106],
f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101]


However, tpstats looks good. And the schema changes eventually do get
applied on *all* the nodes (even the ones that seem to have different
schema versions). There are no communications issues between the nodes and
they are all in the same rack

root@:~# nodetool tpstats
Pool NameActive   Pending  Completed   Blocked
All time blocked
ReadStage 0 01254592 0
0
RequestResponseStage  0 09480827 0
0
MutationStage 0 08662263 0
0
ReadRepairStage   0 0 339158 0
0
ReplicateOnWriteStage 0 0  0 0
0
GossipStage   0 01469197 0
0
AntiEntropyStage  0 0  0 0
0
MigrationStage0 0   1808 0
0
MemtablePostFlusher   0 0248 0
0
StreamStage   0 0  0 0
0
FlushWriter   0 0248 0
4
MiscStage 0 0  0 0
0
commitlog_archiver0 0  0 0
0
InternalResponseStage 0 0   5286 0
0
HintedHandoff 0 0 21 0
0

Message type   Dropped
RANGE_SLICE  0
READ_REPAIR  0
BINARY   0
READ 0
MUTATION 0
REQUEST_RESPONSE 0

So I'm guessing maybe the different schema versions may be potentially
stopping compactions? Will compactions still happen if there are different
versions of the schema?





On 9/18/12 11:38 AM, "Michael Kjellman" 
mailto:mkjell...@barracuda.com>> wrote:

>Thanks, I just modified the schema on the worse offending column family
>(as determined by the .json) from 10MB to 200MB.
>
>Should I kick off a compaction on this cf now/repair?/scrub?
>
>Thanks
>
>-michael
>
>From: Віталій Тимчишин 
>mailto:tiv...@gmail.com>>>
>Reply-To: 
>"user@cassandra.apache.org>"
>mailto:user@cassandra.apache.org>>>
>To: 
>"user@cassandra.apache.org>"
>mailto:user@cassandra.apache.org>>>
>Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)
>
>I've started to use LeveledCompaction some time ago and from my
>experience this indicates some SST on lower levels than they should be.
>The compaction is going, moving them up level by level, but total count
>does not change as new data goe

Re: Invalid Counter Shard errors?

[Peter Schuller]

> I don't understand what the three in parentheses values are exactly. I guess
> the last number is the count and the middle one is the number of increments,
> is that true ? What is the first string (identical in all the errors) ?

It's (UUID, clock, increment). Very  briefly, counter columns in
Cassandra are made up of multiple "shards". In the write path, a
particular counter increment is executed by one "leader" which is one
of the replicas of the counter. The leader will increment it's own
value, read it's own full value (this is why "Replicate On Write" has
to do reads in the write path for counters) and replicas to other
nodes.

UUID "roughly" corresponds to a node in the cluster (UUID:s are
sometimes refreshed, so it's not a strict correlation). Clockid is
supposed to be monotonically increasing for a given UUID.

> How can the last number (assuming it's the count) be negative knowing that I
> only sum positive numbers ?

I don't see a negative number in you paste?

> An other point is that the highest value seems to be *always* the good one
> (assuming this time that the middle number is the number of increments).

DISCLAIMER: This is me responding off the cuff without digging into it further.

Depends on the source of the problem. If the problem, as theorized in
the ticket, is caused by non-clean shutdown of nodes the expected
result *should* be that we effectively "loose" counter increments.
Given a particular leader among the replicas, suppose you increment
counter C by N1, followed by un-clean shutdown with the value never
having been written to the commit log. On the next increment of C by
N2, a counter shard would be generated which has the value being
base+N2 instead of base+N1 (assuming the memtable wasn't flushed and
no other writes to the same counter column happened).

When this gets replicated to other nodes, they would see a value based
on N1 and a value based on N2, both with the same clock. It would
choose the higher one. In either case as far as I can tell (off the
top of my head), *some* counter increment is lost. The only way I can
see (again off the top of my head) the resulting value being correct
is if the later increment (N2 in this case) is somehow including N1 as
well (e.g., because it was generated by first reading the current
counter value).

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: Invalid Counter Shard errors?

[Peter Schuller]

The significance I think is: If it is indeed the case that the higher
value is always *in fact* correct, I think that's inconsistent with
the hypothesis that unclean shutdown is the sole cause of these
problems - as long as the client is truly submitting non-idempotent
counter increments without a read-before-write.

As a side note: If hou're using these counters for stuff like
determining amounts of money to be payed by somebody, consider the
non-idempotense of counter increments. Any write that increments a
counter, that fails by e.g. Timeout *MAY OR MAY NOT* have been applied
and cannot be safely retried. Cassandra counters are generally not
useful if *strict* correctness is desired, for this reason.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)