Re: Help for creating a custom partitioner

[Clement Honore]

Hi,

thanks for your answer.

We plan to use manual indexing too (with native C* indexing for other
cases).
So, for one index, we will get plenty of FK and a MultiGet call to get all
the associated entities, with RP, would then spread all the cluster.
As we don't know the cluster size yet, and as it's expected to grow at an
unknown rate, we are thinking about alternatives, now, for scalability.

But, to tell the truth, so far, we have not done performance tests.
But as the choice of a partitioner is the first C* cornerstone, we are
already thinking about a new partitioner.
We are planning tests "random vs custom partitioner" => so, my questions
for creating, first, another one.

AFAIS, your partitioner (the higher bits of the hash from hashing the
category, and the lower bits of the hash from hashing the document id) will
put all the docs of a category in (in average) 1 node. Quite interesting,
thanks!
I could add such a partitioner to my test suite.

But, why not just hashing the "category" part of the row key ?
With such partitioner, as said before, many rows on *one* node are going to
have the same hash value.
- if it hurts Cassandra behavior/performance => I am curious to know why.
Anyway, in that case, I see your partitioner, so far, as the best answer to
my wishes!
- if it's NOT hurting Cassandra behavior/performance => it sounds, then, an
optimal partitioner for our needs.

Any idea about Cassandra behavior with such hash (category-only)
partitioner ?

Regards,
Clément

2012/9/28 Tim Wintle 

> On Fri, 2012-09-28 at 18:20 +0200, Clement Honore wrote:
> > Hi,
> >
> > ** **
> >
> > I have hierarchical data.
> >
> > I'm storing them in CF with rowkey somewhat like (category, doc id), and
> > plenty of columns for a doc definition.
> >
> > ** **
> >
> > I have hierarchical data traversal too.
> >
> > The user just chooses one category, and then, interact with docs
> belonging
> > only to this category.
> >
> > ** **
> >
> > 1) If I use RandomPartitioner, all docs could be spread within all nodes
> in
> > the cluster => bad performance.
> >
> > ** **
> >
> > 2) Using RandomPartitioner, an alternative design could be
> rowkey=category
> > and column name=(doc id, prop name)
> >
> > I don't want it because I need fixed column names for indexing purposes,
> > and the "category" is quite a log string.
> >
> > ** **
> >
> > 3) Then, I want to define a new partitioner for my rowkey (category, doc
> > id), doing MD5 only for the "category" part.
> >
> > ** **
> >
> > The question is : with such partitioner, many rows on *one* node are
> going
> > to have the same MD5 value, as a result of this new partitioner.
>
> If you do decide writing having rows on the same node is what you want,
> then you could take the higher bits of the hash from hashing the
> category, and the lower bits of the hash from hashing the document id.
>
> That would mean documents in a category would be close to each other in
> the ring - while being unlikely to share the same hash.
>
>
> However, If you're doing this then all reads/writes to the category are
> going to be to a single machine. That's not going to spread the load
> across the cluster very well as I assume a few categories are going to
> be far more popular than others.
>
> Have you tested that you actually get bad performance from
> RandomPartitioner?
>
> Tim
>
>

Re: Help for creating a custom partitioner

[Tim Wintle]

On Mon, 2012-10-01 at 10:45 +0200, Clement Honore wrote:
> We plan to use manual indexing too (with native C* indexing for other
> cases).
> So, for one index, we will get plenty of FK and a MultiGet call to get all
> the associated entities, with RP, would then spread all the cluster.
> As we don't know the cluster size yet, and as it's expected to grow at an
> unknown rate, we are thinking about alternatives, now, for scalability.
> 
> But, to tell the truth, so far, we have not done performance tests.
> But as the choice of a partitioner is the first C* cornerstone, we are
> already thinking about a new partitioner.
> We are planning tests "random vs custom partitioner" => so, my questions
> for creating, first, another one.
> 
> AFAIS, your partitioner (the higher bits of the hash from hashing the
> category, and the lower bits of the hash from hashing the document id) will
> put all the docs of a category in (in average) 1 node. Quite interesting,
> thanks!
> I could add such a partitioner to my test suite.
> 
> But, why not just hashing the "category" part of the row key ?
> With such partitioner, as said before, many rows on *one* node are going to
> have the same hash value.
> - if it hurts Cassandra behavior/performance => I am curious to know why.
> Anyway, in that case, I see your partitioner, so far, as the best answer to
> my wishes!
> - if it's NOT hurting Cassandra behavior/performance => it sounds, then, an
> optimal partitioner for our needs.
> 
> Any idea about Cassandra behavior with such hash (category-only)
> partitioner ?

I honestly don't know the code well enough - but I have always assumed
(perhaps incorrectly) that the whole SSTable / Memtable system was
sorted on the hash value rather than the key, so that range queries are
efficient - so if all items on a node have the same hash you would get
awful performance for (at least) reading specific rows from disk. I
could be wrong in my assumptions.

Certainly having lots of hash collisions is unusual behaviour - I don't
imagine the time behaviour has been tested against that situation
closely.

If you haven't yet tested it, then I'm not sure why you assume that
accesses from a single machine would be faster than from documents
spread around the ring - ethernet is fast, and if you're going to have
to do disk seeks to get any of this data then you can run the seeks in
parallel across a large number of spindles by spreading the load around
the cluster.

It also adds extra load onto machines handling popular categories -
assuming the number of categories is significantly smaller than the
number of documents that could make a major difference to latency.

Tim


> 
> Regards,
> Clément
> 
> 2012/9/28 Tim Wintle 
> 
> > On Fri, 2012-09-28 at 18:20 +0200, Clement Honore wrote:
> > > Hi,
> > >
> > > ** **
> > >
> > > I have hierarchical data.
> > >
> > > I'm storing them in CF with rowkey somewhat like (category, doc id), and
> > > plenty of columns for a doc definition.
> > >
> > > ** **
> > >
> > > I have hierarchical data traversal too.
> > >
> > > The user just chooses one category, and then, interact with docs
> > belonging
> > > only to this category.
> > >
> > > ** **
> > >
> > > 1) If I use RandomPartitioner, all docs could be spread within all nodes
> > in
> > > the cluster => bad performance.
> > >
> > > ** **
> > >
> > > 2) Using RandomPartitioner, an alternative design could be
> > rowkey=category
> > > and column name=(doc id, prop name)
> > >
> > > I don't want it because I need fixed column names for indexing purposes,
> > > and the "category" is quite a log string.
> > >
> > > ** **
> > >
> > > 3) Then, I want to define a new partitioner for my rowkey (category, doc
> > > id), doing MD5 only for the "category" part.
> > >
> > > ** **
> > >
> > > The question is : with such partitioner, many rows on *one* node are
> > going
> > > to have the same MD5 value, as a result of this new partitioner.
> >
> > If you do decide writing having rows on the same node is what you want,
> > then you could take the higher bits of the hash from hashing the
> > category, and the lower bits of the hash from hashing the document id.
> >
> > That would mean documents in a category would be close to each other in
> > the ring - while being unlikely to share the same hash.
> >
> >
> > However, If you're doing this then all reads/writes to the category are
> > going to be to a single machine. That's not going to spread the load
> > across the cluster very well as I assume a few categories are going to
> > be far more popular than others.
> >
> > Have you tested that you actually get bad performance from
> > RandomPartitioner?
> >
> > Tim
> >
> >

Re: Prevent queries from OOM nodes

[Віталій Тимчишин]

It's not about columns, it's about rows, see example statement.
In QueryProcessor#processStatement it reads rows into list, then does
list.size()

2012/10/1 aaron morton 

> CQL will read everything into List to make latter a count.
>
>
> From 1.0 onwards count paginated reading the columns. What version are you
> on ?
>
> https://issues.apache.org/jira/browse/CASSANDRA-2894
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/09/2012, at 8:26 PM, Віталій Тимчишин  wrote:
>
> Actually an easy way to put cassandra down is
> select count(*) from A limit 1000
> CQL will read everything into List to make latter a count.
>
> 2012/9/26 aaron morton 
>
>> Can you provide some information on the queries and the size of the data
>> they traversed ?
>>
>> The default maximum size for a single thrift message is 16MB, was it
>> larger than that ?
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>>
>> Cheers
>>
>>
>> On 25/09/2012, at 8:33 AM, Bryce Godfrey 
>> wrote:
>>
>> Is there anything I can do on the configuration side to prevent nodes
>> from going OOM due to queries that will read large amounts of data and
>> exceed the heap available? 
>> ** **
>> For the past few days of we had some nodes consistently freezing/crashing
>> with OOM.  We got a heap dump into MAT and figured out the nodes were dying
>> due to some queries for a few extremely large data sets.  Tracked it back
>> to an app that just didn't prevent users from doing these large queries,
>> but it seems like Cassandra could be smart enough to guard against this
>> type of thing?
>> ** **
>> Basically some kind of setting like "if the data to satisfy query >
>> available heap then throw an error to the caller and about query".  I would
>> much rather return errors to clients then crash a node, as the error is
>> easier to track down that way and resolve.
>> ** **
>> Thanks.
>>
>>
>>
>
>
> --
> Best regards,
>  Vitalii Tymchyshyn
>
>
>


-- 
Best regards,
 Vitalii Tymchyshyn

SST Inconsistency

[Daniel Doubleday]

Hi all

we are running c* 1.0.8 and found some strange row level tombstone problems.

Some rows (~50 in around 2B keys) have markedForDeleteAt timestamps in 
the future (so they 'drop' all writes) and 0 values as localDeletionTime.


A non-thorough check didn't bring up any code paths that could lead to a 
0 localDeletionTime and our client code is not capable to produce the 
markedForDeleteAt values we are seeing.
First we thought that it's file corruption. But we have multiple SSTs on 
different nodes that show the same pattern (though I did not check yet 
whether all belong to the same parition). Also all files are perfectly 
readable and the columns in the deleted rows are correct too.


I searched Jira but couldn't find anything related. So I was wondering 
if anyone is aware of a bug that could have led to this?


I'm going to patch scrub to fix the SSTs so it's not a big deal. Still 
strange ...


Cheers,
Daniel

Re: downgrade from 1.1.4 to 1.0.X

[Daniel Doubleday]

Since I was just fiddling around with sst2json: if you have row level 
deletes you might get problems since row level deletion info is not 
exported in at least 1.0.


But if you're not using those you might be fine.

Віталій Тимчишин wrote:


I suppose the way is to convert all SST to json, then install previous
version, convert back and load

2012/9/24 Arend-Jan Wijtzes mailto:ajwyt...@wise-guys.nl>>

On Thu, Sep 20, 2012 at 10:13:49AM +1200, aaron morton wrote:
> No.
> They use different minor file versions which are not backwards
compatible.

Thanks Aaron.

Is upgradesstables capable of downgrading the files to 1.0.8?
Looking for a way to make this work.

Regards,
Arend-Jan


> On 18/09/2012, at 11:18 PM, Arend-Jan Wijtzes
mailto:ajwyt...@wise-guys.nl>> wrote:
>
> > Hi,
> >
> > We are running Cassandra 1.1.4 and like to experiment with
> > Datastax Enterprise which uses 1.0.8. Can we safely downgrade
> > a production cluster or is it incompatible? Any special steps
> > involved?

--
Arend-Jan Wijtzes -- Wiseguys -- www.wise-guys.nl





--
Best regards,
Vitalii Tymchyshyn

Re: Help for creating a custom partitioner

[Hiller, Dean]

I would be surprised if random partitioner hurt your performance.  In general, 
doing performance tests on a 6 node cluster with PlayOrm Scalable SQL, even 
joins queries ended up faster as the parallel disks of reading all the rows was 
way faster than reading from a single machine(remember, one disk bottleneck can 
really hurt which is why random partitioner works out so well).

Later,
Dean

From: Clement Honore mailto:honor...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Monday, October 1, 2012 2:45 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Help for creating a custom partitioner

Hi,

thanks for your answer.

We plan to use manual indexing too (with native C* indexing for other cases).
So, for one index, we will get plenty of FK and a MultiGet call to get all the 
associated entities, with RP, would then spread all the cluster.
As we don't know the cluster size yet, and as it's expected to grow at an 
unknown rate, we are thinking about alternatives, now, for scalability.

But, to tell the truth, so far, we have not done performance tests.
But as the choice of a partitioner is the first C* cornerstone, we are already 
thinking about a new partitioner.
We are planning tests "random vs custom partitioner" => so, my questions for 
creating, first, another one.

AFAIS, your partitioner (the higher bits of the hash from hashing the category, 
and the lower bits of the hash from hashing the document id) will put all the 
docs of a category in (in average) 1 node. Quite interesting, thanks!
I could add such a partitioner to my test suite.

But, why not just hashing the "category" part of the row key ?
With such partitioner, as said before, many rows on *one* node are going to 
have the same hash value.
- if it hurts Cassandra behavior/performance => I am curious to know why. 
Anyway, in that case, I see your partitioner, so far, as the best answer to my 
wishes!
- if it's NOT hurting Cassandra behavior/performance => it sounds, then, an 
optimal partitioner for our needs.

Any idea about Cassandra behavior with such hash (category-only) partitioner ?

Regards,
Clément

2012/9/28 Tim Wintle mailto:timwin...@gmail.com>>
On Fri, 2012-09-28 at 18:20 +0200, Clement Honore wrote:
> Hi,
>
> ** **
>
> I have hierarchical data.
>
> I'm storing them in CF with rowkey somewhat like (category, doc id), and
> plenty of columns for a doc definition.
>
> ** **
>
> I have hierarchical data traversal too.
>
> The user just chooses one category, and then, interact with docs belonging
> only to this category.
>
> ** **
>
> 1) If I use RandomPartitioner, all docs could be spread within all nodes in
> the cluster => bad performance.
>
> ** **
>
> 2) Using RandomPartitioner, an alternative design could be rowkey=category
> and column name=(doc id, prop name)
>
> I don't want it because I need fixed column names for indexing purposes,
> and the "category" is quite a log string.
>
> ** **
>
> 3) Then, I want to define a new partitioner for my rowkey (category, doc
> id), doing MD5 only for the "category" part.
>
> ** **
>
> The question is : with such partitioner, many rows on *one* node are going
> to have the same MD5 value, as a result of this new partitioner.

If you do decide writing having rows on the same node is what you want,
then you could take the higher bits of the hash from hashing the
category, and the lower bits of the hash from hashing the document id.

That would mean documents in a category would be close to each other in
the ring - while being unlikely to share the same hash.


However, If you're doing this then all reads/writes to the category are
going to be to a single machine. That's not going to spread the load
across the cluster very well as I assume a few categories are going to
be far more popular than others.

Have you tested that you actually get bad performance from
RandomPartitioner?

Tim

Re: Rebalancing cluster

[Darvin Denmian]

Hi,

1) Yes. I'm using Random Partitioner
2) All 3 nodes in the cluster have used "auto_bootstrap" to acquire their
part of data (I really don't know if this is correct).

Regards.

On Sat, Sep 29, 2012 at 5:56 AM, Bradford Toney wrote:

> Are you using the random partitoner? Did you set up the tokens the way
> the manual says?
>
> Sent from my iPhone
>
> On Sep 28, 2012, at 7:55 AM, Darvin  Denmian 
> wrote:
>
> > Hi,
> >
> > currently my Cassandra cluster has 3 nodes and I'm facing a very common
> problem... Rebalance the data of the cluster.
> > At this point I have:
> >
> > node A: 37,69% of data
> > node B: 7,95% of data
> > node C: 54,36% of data
> >
> > all of them running Cassandra 1.1.3.
> >
> > Can somebody suggest something to solve this problem?
> >
> > Regards.
>

Re: Rebalancing cluster

[Hiller, Dean]

You should check the cassandra.yaml file.  There is an initial_token in that 
file that you should have set.  The comment above that property reads

# You should always specify InitialToken when setting up a production
# cluster for the first time, and often when adding capacity later.
# The principle is that each node should be given an equal slice of
# the token ring; see http://wiki.apache.org/cassandra/Operations
# for more details.
#
# If blank, Cassandra will request a token bisecting the range of
# the heaviest-loaded existing node.  If there is no load information
# available, such as is the case with a new cluster, it will pick
# a random token, which will lead to hot spots.

Naturally, he is referring to that link when he says manual ;) AND notice the 
"which will lead to hot spots" comment.

Later,
Dean

From: Darvin Denmian mailto:darvin.denm...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Monday, October 1, 2012 6:56 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Rebalancing cluster

Did you set up the tokens the way
the manual says?

Advice on correct storage configuration

[Lewis John Mcgibbney]

Hi,

I wish to confirm whether the current mapping (storage) configuration
I have is suited to store data commonly extracted field data from Web
Pages.

My mapping can be seen here [0] which basically specifies three column
families e.g. parse (p), fetch (f) and super columns (sc) within the
webpage keyspace.

Each column family subsequently includes several fields which for
clarity include comments. Current CF configuration is as follows:

- fetch CF includes 11 columns
- parse CF including 4
- super column CF including 7

I am trying to ascertain why the 7 super column fields are currently
configured to be super columns as oppose to standard columns!
I therefore wonder if someone can please clarify if such a
configuration is suited to storing data of this nature.

Thank you in advance. if this is too vague an explanation the please
say so and I will be happy to expand on any aspect in an attempt to
fully understand the data model and the configuration.

Thank you

Lewis

[0] 
http://svn.apache.org/viewvc/nutch/branches/2.x/conf/gora-cassandra-mapping.xml?view=markup



-- 
Lewis

Re: Rebalancing cluster

[Darvin Denmian]

When I execute the command "nodetool ring" the following output appears:

Address DC  RackStatus State   Load
Effective-Ownership Token


15977425217612630204561194765242499
10.36.214.118   datacenter1 rack1   Up Normal  16.62 GB
54.36%  82119805777845627277766502152998847873
10.36.214.66datacenter1 rack1   Up Normal  6.44 GB
7.95% 95648023217342394827789894311936282285
10.36.214.122   datacenter1 rack1   Up Normal  10.85 GB
37.69%  15977425217612630204561194765242499

as you can see there is no "Zero Token". Maybe I did something wrong.
Will I need to "reset" my entire cluster?

Regards.

On Mon, Oct 1, 2012 at 10:18 AM, Hiller, Dean  wrote:
>
> You should check the cassandra.yaml file.  There is an initial_token in that 
> file that you should have set.  The comment above that property reads
>
> # You should always specify InitialToken when setting up a production
> # cluster for the first time, and often when adding capacity later.
> # The principle is that each node should be given an equal slice of
> # the token ring; see http://wiki.apache.org/cassandra/Operations
> # for more details.
> #
> # If blank, Cassandra will request a token bisecting the range of
> # the heaviest-loaded existing node.  If there is no load information
> # available, such as is the case with a new cluster, it will pick
> # a random token, which will lead to hot spots.
>
> Naturally, he is referring to that link when he says manual ;) AND notice the 
> "which will lead to hot spots" comment.
>
> Later,
> Dean
>
> From: Darvin Denmian 
> mailto:darvin.denm...@gmail.com>>
> Reply-To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Date: Monday, October 1, 2012 6:56 AM
> To: "user@cassandra.apache.org" 
> mailto:user@cassandra.apache.org>>
> Subject: Re: Rebalancing cluster
>
> Did you set up the tokens the way
> the manual says?

Re: Rebalancing cluster

[Hiller, Dean]

Nodetool has a move command so you can move to a new better token.  Read
up on the documentation there.  I have not used it yet myselfŠ.good idea
to test it on your test cluster first.

Dean

On 10/1/12 8:03 AM, "Darvin Denmian"  wrote:

>as you can see there is no "Zero Token". Maybe I did something wrong.
>Will I need to "reset" my entire cluster?

Re: Advice on correct storage configuration

[Hiller, Dean]

What is really going to matter is what is the applications trying to read?
 That is really the critical piece of context.  Without knowing what the
application needs to read, it is very hard to design.

One example from a previous post that was a great questions wasŠ
1. I need to get the last 100 requests no matter which user
2. I need to get the last 100 requests of a specific user

This gives anyone on the list an idea of how the model should look AND
also is why on the web you will find many many many references that noSQL
is designed from the queries.  (that said, I have seen one project where
denormalization caused them to write 1 meg of data on one request so there
is a balanceŠ.and boy was that slow on the writes).

Later,
Dean

On 10/1/12 7:27 AM, "Lewis John Mcgibbney" 
wrote:

>Hi,
>
>I wish to confirm whether the current mapping (storage) configuration
>I have is suited to store data commonly extracted field data from Web
>Pages.
>
>My mapping can be seen here [0] which basically specifies three column
>families e.g. parse (p), fetch (f) and super columns (sc) within the
>webpage keyspace.
>
>Each column family subsequently includes several fields which for
>clarity include comments. Current CF configuration is as follows:
>
>- fetch CF includes 11 columns
>- parse CF including 4
>- super column CF including 7
>
>I am trying to ascertain why the 7 super column fields are currently
>configured to be super columns as oppose to standard columns!
>I therefore wonder if someone can please clarify if such a
>configuration is suited to storing data of this nature.
>
>Thank you in advance. if this is too vague an explanation the please
>say so and I will be happy to expand on any aspect in an attempt to
>fully understand the data model and the configuration.
>
>Thank you
>
>Lewis
>
>[0] 
>http://svn.apache.org/viewvc/nutch/branches/2.x/conf/gora-cassandra-mappin
>g.xml?view=markup
>
>
>
>-- 
>Lewis

Getting serialized Rows from CommitLogSegment file

[Felipe Schmidt]

Hello.

I'm trying to catch the serialized RowMutations from a CommitLogSegment to
capture the data change, but I don't have much idea about how to proceed.
Some one know a way of how to do it? I supposed that it would be kind of
simple.

Regards,
Felipe Mathias Schmidt
*(Computer Science UFRGS, RS, Brazil)*

Re: Cassandra supercolumns with same name

[Cyril Auburtin]

Yep Tyler is right
It seems I have trailing *\u* (null) characters , (one column name is
mymed_embrun.ma...@gmail.com the other mymed_embrun.ma...@gmail.com\u\u
for example)

I'm trying to know at what point they are created...
Thx

2012/9/21 Tyler Hobbs 

> If you're seeing that in cassandra-cli, it's possible that there are some
> non-printable characters in the name that the cli doesn't display, like the
> NUL char (ascii 0).  I opened a ticket for that somewhere, but in the
> meantime, you may want to verify that they are identical with a real client.
>
>
> On Tue, Sep 18, 2012 at 4:03 AM, aaron morton wrote:
>
>> They are. Can you provide some more information ?
>>
>> What happens when you read the super column ?
>>
>> Cheers
>>
>>   -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 18/09/2012, at 5:33 AM, Cyril Auburtin 
>> wrote:
>>
>> First sorry but I'm using an old version 0.7.10
>>
>> and recently I've come up seeing this
>>
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=permission, value=1, timestamp=1347895421475))
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=email, value=embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=id, value=mymed_embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=permission, value=0, timestamp=1347894698217)
>>  (column=profile, value=e24af776b4a025456bd50f55633b2419,
>> timestamp=1347894698217))
>>
>> as a part of of a supercolumnFamily
>>
>> I thought supercolumn was meant to be unique?
>>
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>
>

Re: Cassandra supercolumns with same name

[Cyril Auburtin]

Yep Tyler is right
It seems I have trailing *\u* (null) characters , (one column name is
mymed_embrun.ma...@gmail.com the other mymed_embrun.ma...@gmail.com\u\u
for example)

I'm trying to know at what point they are created...
Thx

2012/9/21 Tyler Hobbs 

> If you're seeing that in cassandra-cli, it's possible that there are some
> non-printable characters in the name that the cli doesn't display, like the
> NUL char (ascii 0).  I opened a ticket for that somewhere, but in the
> meantime, you may want to verify that they are identical with a real client.
>
>
> On Tue, Sep 18, 2012 at 4:03 AM, aaron morton wrote:
>
>> They are. Can you provide some more information ?
>>
>> What happens when you read the super column ?
>>
>> Cheers
>>
>>   -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 18/09/2012, at 5:33 AM, Cyril Auburtin 
>> wrote:
>>
>> First sorry but I'm using an old version 0.7.10
>>
>> and recently I've come up seeing this
>>
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=permission, value=1, timestamp=1347895421475))
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=email, value=embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=id, value=mymed_embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=permission, value=0, timestamp=1347894698217)
>>  (column=profile, value=e24af776b4a025456bd50f55633b2419,
>> timestamp=1347894698217))
>>
>> as a part of of a supercolumnFamily
>>
>> I thought supercolumn was meant to be unique?
>>
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>
>

[solved] Cassandra columns with same name

[Cyril Auburtin]

well don't know why I couldn't reply to the initial thread..

yep Tyler is right I have columns with trailing null chars
mycolumn, mycolumn\u\u prints the same in cassandra-cli, but not if
you write the output in a file
I haven't find where they come from, but no matters the architecture we are
using is hopeless, will rather try to make sure to trim column names before
insertion

2012/9/21 Tyler Hobbs 

> If you're seeing that in cassandra-cli, it's possible that there are some
> non-printable characters in the name that the cli doesn't display, like the
> NUL char (ascii 0).  I opened a ticket for that somewhere, but in the
> meantime, you may want to verify that they are identical with a real client.
>
>
> On Tue, Sep 18, 2012 at 4:03 AM, aaron morton 
>  wrote:
>
>> They are. Can you provide some more information ?
>>
>> What happens when you read the super column ?
>>
>> Cheers
>>
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 18/09/2012, at 5:33 AM, Cyril Auburtin 
>> wrote:
>>
>> First sorry but I'm using an old version 0.7.10
>>
>> and recently I've come up seeing this
>>
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=permission, value=1, timestamp=1347895421475))
>> => (super_column=mymed_embrun.ma...@gmail.com,
>>  (column=email, value=embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=id, value=mymed_embrun.ma...@gmail.com,
>> timestamp=1347894698217)
>>  (column=permission, value=0, timestamp=1347894698217)
>>  (column=profile, value=e24af776b4a025456bd50f55633b2419,
>> timestamp=1347894698217))
>>
>> as a part of of a supercolumnFamily
>>
>> I thought supercolumn was meant to be unique?
>>
>>
>>
>
>
> --
> Tyler Hobbs
> DataStax 
>
>

Cassandra vs Couchbase benchmarks

[Andy Cobley]

There are some interesting results in the benchmarks below:

http://www.slideshare.net/renatko/couchbase-performance-benchmarking

Without starting a flame war etc, I'm interested if these results should
be considered "Fair and Balanced" or if the methodology is flawed in some
way ? (for instance is the use of Amazon EC2 sensible for Cassandra
deployment) ?

Andy



The University of Dundee is a Scottish Registered Charity, No. SC015096.

Re: Cassandra vs Couchbase benchmarks

[Peter Lin]

Here is my own experience testing couchdb versus cassandra for an
internal application.

My test wasn't some dummy test case, it was realistic workloads that
is 95% write and 5% read. We insert data in batches to maximize
throughput. The critical thing for my use case was to answer "when
does the server crash and what happens?"

Each row was approximately 4-8K. I tested with a variety of batch
sizes and found that couchdb choked around 20K rows of data. When I
looked at couchdb logs, there was no indication of why it crashed or
what caused it. I tested cassandra up to 50K rows per batch and didn't
have time to reach the failure point.

The troubling issue for me is the lack of information on the cause of
the crash. All server crash at some point. Having details on why and
how it crashed to me is critical to debugging and improving
performance. Having zero indication for me suggests major
architectural flaws and issues in the design and implementation. I
haven't had time to investigate the real cause, since the logs didn't
give any hint. If the log had details, I would have atleast taken a
look at the source file and try to debug it.



On Mon, Oct 1, 2012 at 11:05 AM, Andy Cobley
 wrote:
> There are some interesting results in the benchmarks below:
>
> http://www.slideshare.net/renatko/couchbase-performance-benchmarking
>
> Without starting a flame war etc, I'm interested if these results should
> be considered "Fair and Balanced" or if the methodology is flawed in some
> way ? (for instance is the use of Amazon EC2 sensible for Cassandra
> deployment) ?
>
> Andy
>
>
>
> The University of Dundee is a Scottish Registered Charity, No. SC015096.
>
>

Re: 1000's of column families

[Brian O'Neill]

Dean,

We have the same question...

We have thousands of separate feeds of data as well (20,000+).  To
date, we've been using a CF per feed strategy, but as we scale this
thing out to accommodate all of those feeds, we're trying to figure
out if we're going to blow out the memory.

The initial documentation for heap sizing had column families in the equation:
http://www.datastax.com/docs/0.7/operations/tuning#heap-sizing

But in the more recent documentation, it looks like they removed the
column family variable with the introduction of the universal
key_cache_size.
http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size

We haven't committed either way yet, but given Ed Anuff's presentation
on virtual keyspaces, we were leaning towards a single column family
approach:
http://blog.apigee.com/detail/building_a_mobile_data_platform_with_cassandra_-_apigee_under_the_hood/?

Definitely let us know what you decide.

-brian

On Fri, Sep 28, 2012 at 11:48 AM, Flavio Baronti
 wrote:
> We had some serious trouble with dynamically adding CFs, although last time
> we tried we were using version 0.7, so maybe
> that's not an issue any more.
> Our problems were two:
> - You are (were?) not supposed to add CFs concurrently. Since we had more
> servers talking to the same Cassandra cluster,
> we had to use distributed locks (Hazelcast) to avoid concurrency.
> - You must be very careful to add new CFs to different Cassandra nodes. If
> you do that fast enough, and the clocks of
> the two servers are skewed, you will severely compromise your schema
> (Cassandra will not understand in which order the
> updates must be applied).
>
> As I said, this applied to version 0.7, maybe current versions solved these
> problems.
>
> Flavio
>
>
> Il 2012/09/27 16:11 PM, Hiller, Dean ha scritto:
>> We have 1000's of different building devices and we stream data from these
> devices.  The format and data from each one varies so one device has 
> temperature
> at timeX with some other variables, another device has CO2 percentage and 
> other
> variables.  Every device is unique and streams it's own data.  We dynamically
> discover devices and register them.  Basically, one CF or table per thing 
> really
> makes sense in this environment.  While we could try to find out which devices
> "are" similar, this would really be a pain and some devices add some new
> variable into the equation.  NOT only that but researchers can register new
> datasets and upload them as well and each dataset they have they do NOT want 
> to
> share with other researches necessarily so we have security groups and each CF
> belongs to security groups.  We dynamically create CF's on the fly as people
> register new datasets.
>>
>> On top of that, when the data sets get too large, we probably want to
> partition a single CF into time partitions.  We could create one CF and put 
> all
> the data and have a partition per device, but then a time partition will 
> contain
> "multiple" devices of data meaning we need to shrink our time partition size
> where if we have CF per device, the time partition can be larger as it is only
> for that one device.
>>
>> THEN, on top of that, we have a meta CF for these devices so some people want
> to query for streams that match criteria AND which returns a CF name and they
> query that CF name so we almost need a query with variables like select cfName
> from Meta where x = y and then select * from cfName where x. Which we can 
> do
> today.
>>
>> Dean
>>
>> From: Marcelo Elias Del Valle mailto:mvall...@gmail.com>>
>> Reply-To: "user@cassandra.apache.org"
> mailto:user@cassandra.apache.org>>
>> Date: Thursday, September 27, 2012 8:01 AM
>> To: "user@cassandra.apache.org"
> mailto:user@cassandra.apache.org>>
>> Subject: Re: 1000's of column families
>>
>> Out of curiosity, is it really necessary to have that amount of CFs?
>> I am probably still used to relational databases, where you would use a new
> table just in case you need to store different kinds of data. As Cassandra
> stores anything in each CF, it might probably make sense to have a lot of CFs 
> to
> store your data...
>> But why wouldn't you use a single CF with partitions in these case? Wouldn't
> it be the same thing? I am asking because I might learn a new modeling 
> technique
> with the answer.
>>
>> []s
>>
>> 2012/9/26 Hiller, Dean mailto:dean.hil...@nrel.gov>>
>> We are streaming data with 1 stream per 1 CF and we have 1000's of CF.  When
> using the tools they are all geared to analyzing ONE column family at a time 
> :(.
> If I remember correctly, Cassandra supports as many CF's as you want, correct?
> Even though I am going to have tons of funs with limitations on the tools,
> correct?
>>
>> (I may end up wrapping the node tool with my own aggregate calls if needed to
> sum up multiple column families and such).
>>
>> Thanks,
>> Dean
>>
>>
>>
>> --
>> Marcelo Elias Del V

Re: Cassandra vs Couchbase benchmarks

[Michael Kjellman]

From their wiki: "The replication is an incremental one way process
involving two databases (a source and a destination).
The aim of the replication is that at the end of the process, all active
documents on the source database are also in the destination database and
all documents that were deleted in the source databases are also deleted
(if exists) on the destination database."

CouchDB =! Cassandra for replication. From everything I've seen it behaves
much more like mysql replication than anything else.

If anything they tested replication. Also they barely tuned Cassandra from
those slides so I wonder if compactions etc bit them.

Finally, there are some very high profile people using Cassandra on Amazon
EC2. My understanding is that disk IO is the biggest limitation here.

My 2 cents.

Best,
michael

On 10/1/12 8:05 AM, "Andy Cobley"  wrote:

>There are some interesting results in the benchmarks below:
>
>http://www.slideshare.net/renatko/couchbase-performance-benchmarking
>
>Without starting a flame war etc, I'm interested if these results should
>be considered "Fair and Balanced" or if the methodology is flawed in some
>way ? (for instance is the use of Amazon EC2 sensible for Cassandra
>deployment) ?
>
>Andy
>
>
>
>The University of Dundee is a Scottish Registered Charity, No. SC015096.
>
>


'Like' us on Facebook for exclusive content and other resources on all 
Barracuda Networks solutions.
Visit http://barracudanetworks.com/facebook

Re: Collections in native protocol

[Jonathan Rudenberg]

Thanks. Do you want me to open a JIRA ticket?

On Oct 1, 2012, at 2:45 AM, Sylvain Lebresne  wrote:

> Ok, I'll look what's wrong.
> 
> --
> Sylvain

Re: Cassandra vs Couchbase benchmarks

[horschi]

Hi Andy,

things I find odd:

- Replicacount=1 for mongo and couchdb. How is that a realistic benchmark?
I always want at least 2 replicas for my data. Maybe thats just me.
- On the Mongo Config slide they said they disabled journaling. Why do you
disable all safety mechanisms that you would want in a production
environment? Maybe they should have added /dev/null to their benchmark ;-)
- I dont see the replicacount for Cassandra in the slides. Also CL is not
specified. Imho the important stuff is missing in the cassandra
configuration.
- In the goals section it said "more data than RAM". But they only have
12GB data per node, with 15GB of RAM per node!

I am very interested in a recent cassandra-benchmark, but I find this
benchmark very disappointing.

cheers,
Christian


On Mon, Oct 1, 2012 at 5:05 PM, Andy Cobley
wrote:

> There are some interesting results in the benchmarks below:
>
> http://www.slideshare.net/renatko/couchbase-performance-benchmarking
>
> Without starting a flame war etc, I'm interested if these results should
> be considered "Fair and Balanced" or if the methodology is flawed in some
> way ? (for instance is the use of Amazon EC2 sensible for Cassandra
> deployment) ?
>
> Andy
>
>
>
> The University of Dundee is a Scottish Registered Charity, No. SC015096.
>
>
>

Re: Ball is rolling on High Performance Cassandra Cookbook second edition

[Edward Capriolo]

Hello all,

Work has begun on the second edition!  Keep hitting me up with ideas.
In particular I am looking for someone who has done work with
flume+Cassandra and pig+Cassandra. Both of these things topics will be
covered to some extent in the second edition, but these are two
instances in which I could use some help as I do not have extensive
experience with these two combinations.

Contact me if you have any other ideas as well.

Edward

On Tue, Jun 26, 2012 at 5:25 PM, Edward Capriolo  wrote:
> Hello all,
>
> It has not been very long since the first book was published but
> several things have been added to Cassandra and a few things have
> changed. I am putting together a list of changed content, for example
> features like the old per Column family memtable flush settings versus
> the new system with the global variable.
>
> My editors have given me the green light to grow the second edition
> from ~200 pages currently up to 300 pages! This gives us the ability
> to add more items/sections to the text.
>
> Some things were missing from the first edition such as Hector
> support. Nate has offered to help me in this area. Please feel contact
> me with any ideas and suggestions of recipes you would like to see in
> the book. Also get in touch if you want to write a recipe. Several
> people added content to the first edition and it would be great to see
> that type of participation again.
>
> Thank you,
> Edward

Re: Collections in native protocol

[Sylvain Lebresne]

Sure.

On Mon, Oct 1, 2012 at 5:33 PM, Jonathan Rudenberg
 wrote:
> Thanks. Do you want me to open a JIRA ticket?
>
> On Oct 1, 2012, at 2:45 AM, Sylvain Lebresne  wrote:
>
>> Ok, I'll look what's wrong.
>>
>> --
>> Sylvain

Re: 1000's of column families

[Hiller, Dean]

Well, I am now thinking of adding a virtual capability to PlayOrm which we
currently use to allow grouping entities into one column family.  Right
now the CF creation comes from a single entity so this then may change for
those entities that define they are in a single CF groupŠ.This should not
be a very hard change if we decide to do that.

This makes us rely even more on PlayOrm's command line tool(instead of
cassandra-cli) as I can't stand reading hex all the time nor do I like
switching my "assume validator to utf8 to decimal, to integer just so I
can read stuff".

Later,
Dean

On 10/1/12 9:22 AM, "Brian O'Neill"  wrote:

>Dean,
>
>We have the same question...
>
>We have thousands of separate feeds of data as well (20,000+).  To
>date, we've been using a CF per feed strategy, but as we scale this
>thing out to accommodate all of those feeds, we're trying to figure
>out if we're going to blow out the memory.
>
>The initial documentation for heap sizing had column families in the
>equation:
>http://www.datastax.com/docs/0.7/operations/tuning#heap-sizing
>
>But in the more recent documentation, it looks like they removed the
>column family variable with the introduction of the universal
>key_cache_size.
>http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size
>
>We haven't committed either way yet, but given Ed Anuff's presentation
>on virtual keyspaces, we were leaning towards a single column family
>approach:
>http://blog.apigee.com/detail/building_a_mobile_data_platform_with_cassand
>ra_-_apigee_under_the_hood/?
>
>Definitely let us know what you decide.
>
>-brian
>
>On Fri, Sep 28, 2012 at 11:48 AM, Flavio Baronti
> wrote:
>> We had some serious trouble with dynamically adding CFs, although last
>>time
>> we tried we were using version 0.7, so maybe
>> that's not an issue any more.
>> Our problems were two:
>> - You are (were?) not supposed to add CFs concurrently. Since we had
>>more
>> servers talking to the same Cassandra cluster,
>> we had to use distributed locks (Hazelcast) to avoid concurrency.
>> - You must be very careful to add new CFs to different Cassandra nodes.
>>If
>> you do that fast enough, and the clocks of
>> the two servers are skewed, you will severely compromise your schema
>> (Cassandra will not understand in which order the
>> updates must be applied).
>>
>> As I said, this applied to version 0.7, maybe current versions solved
>>these
>> problems.
>>
>> Flavio
>>
>>
>> Il 2012/09/27 16:11 PM, Hiller, Dean ha scritto:
>>> We have 1000's of different building devices and we stream data from
>>>these
>> devices.  The format and data from each one varies so one device has
>>temperature
>> at timeX with some other variables, another device has CO2 percentage
>>and other
>> variables.  Every device is unique and streams it's own data.  We
>>dynamically
>> discover devices and register them.  Basically, one CF or table per
>>thing really
>> makes sense in this environment.  While we could try to find out which
>>devices
>> "are" similar, this would really be a pain and some devices add some new
>> variable into the equation.  NOT only that but researchers can register
>>new
>> datasets and upload them as well and each dataset they have they do NOT
>>want to
>> share with other researches necessarily so we have security groups and
>>each CF
>> belongs to security groups.  We dynamically create CF's on the fly as
>>people
>> register new datasets.
>>>
>>> On top of that, when the data sets get too large, we probably want to
>> partition a single CF into time partitions.  We could create one CF and
>>put all
>> the data and have a partition per device, but then a time partition
>>will contain
>> "multiple" devices of data meaning we need to shrink our time partition
>>size
>> where if we have CF per device, the time partition can be larger as it
>>is only
>> for that one device.
>>>
>>> THEN, on top of that, we have a meta CF for these devices so some
>>>people want
>> to query for streams that match criteria AND which returns a CF name
>>and they
>> query that CF name so we almost need a query with variables like select
>>cfName
>> from Meta where x = y and then select * from cfName where x. Which
>>we can do
>> today.
>>>
>>> Dean
>>>
>>> From: Marcelo Elias Del Valle
>>>mailto:mvall...@gmail.com>>
>>> Reply-To: "user@cassandra.apache.org"
>> mailto:user@cassandra.apache.org>>
>>> Date: Thursday, September 27, 2012 8:01 AM
>>> To: "user@cassandra.apache.org"
>> mailto:user@cassandra.apache.org>>
>>> Subject: Re: 1000's of column families
>>>
>>> Out of curiosity, is it really necessary to have that amount of CFs?
>>> I am probably still used to relational databases, where you would use
>>>a new
>> table just in case you need to store different kinds of data. As
>>Cassandra
>> stores anything in each CF, it might probably make sense to have a lot
>>of CFs to
>> store your data...
>>> But why wouldn'

read-repair and deletes / forgotten deletes

[Hiller, Dean]

I know there is a 10 day limit if you have a node out of the cluster where you 
better be running read-repair or you end up with forgotten deletes, but what 
about on a clean cluster with all nodes always available?  Shouldn't the 
deletes eventually take place or does one have to keep running read-repair 
manually all the time(god, I hope not as that should be automated for deletes).

I am seeing the empty rows in the CLI right now like so…(and would prefer they 
go away on their own)…

RowKey: post:10079376916:b2:4045302966737879216
---
RowKey: post:10079386137:b2:3120637961487430473
---
RowKey: post:10079377944:b2:5118918738052923761
---
RowKey: post:10079378212:b2:1449128414219430571
---
RowKey: post:10079386010:b2:8567843818586569179

If not, someone has to already have an automated project for this, anyone know 
of one??

Thanks,
Dean

Re: read-repair and deletes / forgotten deletes

[Aaron Turner]

the 10 days is actually configurable... look into gc_grace.

Basically, you always need to run repair once per/gc_grace period.
You won't see empty/deleted rows go away until they're compacted away.

On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean  wrote:
> I know there is a 10 day limit if you have a node out of the cluster where 
> you better be running read-repair or you end up with forgotten deletes, but 
> what about on a clean cluster with all nodes always available?  Shouldn't the 
> deletes eventually take place or does one have to keep running read-repair 
> manually all the time(god, I hope not as that should be automated for 
> deletes).
>
> I am seeing the empty rows in the CLI right now like so…(and would prefer 
> they go away on their own)…
>
> RowKey: post:10079376916:b2:4045302966737879216
> ---
> RowKey: post:10079386137:b2:3120637961487430473
> ---
> RowKey: post:10079377944:b2:5118918738052923761
> ---
> RowKey: post:10079378212:b2:1449128414219430571
> ---
> RowKey: post:10079386010:b2:8567843818586569179
>
> If not, someone has to already have an automated project for this, anyone 
> know of one??
>
> Thanks,
> Dean
>



-- 
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"

Re: Rebalancing cluster

[Darvin Denmian]

Hey,

now I got it. I'll try it this night :)
Thanks for all replies.

Regards.

On Mon, Oct 1, 2012 at 11:08 AM, Hiller, Dean  wrote:
> Nodetool has a move command so you can move to a new better token.  Read
> up on the documentation there.  I have not used it yet myselfŠ.good idea
> to test it on your test cluster first.
>
> Dean
>
> On 10/1/12 8:03 AM, "Darvin Denmian"  wrote:
>
>>as you can see there is no "Zero Token". Maybe I did something wrong.
>>Will I need to "reset" my entire cluster?
>

Re: read-repair and deletes / forgotten deletes

[Hiller, Dean]

Thanks, (actually new it was configurable) BUT what I don't get is why I
have to run a repair.  IF all nodes became consistent on the delete, it
should not be possible to get a forgotten delete, correct.  The forgotten
delete will only occur if I have a node down and out for 10 days and it
comes back online because by then the nodes no longer have the delete
anymore and the new node has data so getting to a consistent state the
node with data would win.

S, if I run repair say every 20 days, isn't it true, I would have no
problems as long as I did not have a node outage?

And most importantly, does anyone know of an automated tool for running
repairs every X days(this should really be an automated/schedulable
thing)???

Thanks,
Dean

On 10/1/12 11:57 AM, "Aaron Turner"  wrote:

>the 10 days is actually configurable... look into gc_grace.
>
>Basically, you always need to run repair once per/gc_grace period.
>You won't see empty/deleted rows go away until they're compacted away.
>
>On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean  wrote:
>> I know there is a 10 day limit if you have a node out of the cluster
>>where you better be running read-repair or you end up with forgotten
>>deletes, but what about on a clean cluster with all nodes always
>>available?  Shouldn't the deletes eventually take place or does one have
>>to keep running read-repair manually all the time(god, I hope not as
>>that should be automated for deletes).
>>
>> I am seeing the empty rows in the CLI right now like soŠ(and would
>>prefer they go away on their own)Š
>>
>> RowKey: post:10079376916:b2:4045302966737879216
>> ---
>> RowKey: post:10079386137:b2:3120637961487430473
>> ---
>> RowKey: post:10079377944:b2:5118918738052923761
>> ---
>> RowKey: post:10079378212:b2:1449128414219430571
>> ---
>> RowKey: post:10079386010:b2:8567843818586569179
>>
>> If not, someone has to already have an automated project for this,
>>anyone know of one??
>>
>> Thanks,
>> Dean
>>
>
>
>
>-- 
>Aaron Turner
>http://synfin.net/ Twitter: @synfinatic
>http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
>Windows
>Those who would give up essential Liberty, to purchase a little temporary
>Safety, deserve neither Liberty nor Safety.
>-- Benjamin Franklin
>"carpe diem quam minimum credula postero"

Re: read-repair and deletes / forgotten deletes

[Hiller, Dean]

Oh, and I have been reading Aaron Mortan's article here

http://thelastpickle.com/2011/05/15/Deletes-and-Tombstones/


On 10/1/12 12:46 PM, "Hiller, Dean"  wrote:

>Thanks, (actually new it was configurable) BUT what I don't get is why I
>have to run a repair.  IF all nodes became consistent on the delete, it
>should not be possible to get a forgotten delete, correct.  The forgotten
>delete will only occur if I have a node down and out for 10 days and it
>comes back online because by then the nodes no longer have the delete
>anymore and the new node has data so getting to a consistent state the
>node with data would win.
>
>S, if I run repair say every 20 days, isn't it true, I would have no
>problems as long as I did not have a node outage?
>
>And most importantly, does anyone know of an automated tool for running
>repairs every X days(this should really be an automated/schedulable
>thing)???
>
>Thanks,
>Dean
>
>On 10/1/12 11:57 AM, "Aaron Turner"  wrote:
>
>>the 10 days is actually configurable... look into gc_grace.
>>
>>Basically, you always need to run repair once per/gc_grace period.
>>You won't see empty/deleted rows go away until they're compacted away.
>>
>>On Mon, Oct 1, 2012 at 6:32 PM, Hiller, Dean 
>>wrote:
>>> I know there is a 10 day limit if you have a node out of the cluster
>>>where you better be running read-repair or you end up with forgotten
>>>deletes, but what about on a clean cluster with all nodes always
>>>available?  Shouldn't the deletes eventually take place or does one have
>>>to keep running read-repair manually all the time(god, I hope not as
>>>that should be automated for deletes).
>>>
>>> I am seeing the empty rows in the CLI right now like soŠ(and would
>>>prefer they go away on their own)Š
>>>
>>> RowKey: post:10079376916:b2:4045302966737879216
>>> ---
>>> RowKey: post:10079386137:b2:3120637961487430473
>>> ---
>>> RowKey: post:10079377944:b2:5118918738052923761
>>> ---
>>> RowKey: post:10079378212:b2:1449128414219430571
>>> ---
>>> RowKey: post:10079386010:b2:8567843818586569179
>>>
>>> If not, someone has to already have an automated project for this,
>>>anyone know of one??
>>>
>>> Thanks,
>>> Dean
>>>
>>
>>
>>
>>-- 
>>Aaron Turner
>>http://synfin.net/ Twitter: @synfinatic
>>http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
>>Windows
>>Those who would give up essential Liberty, to purchase a little temporary
>>Safety, deserve neither Liberty nor Safety.
>>-- Benjamin Franklin
>>"carpe diem quam minimum credula postero"
>

Re: read-repair and deletes / forgotten deletes

[Aaron Turner]

inline...

On Mon, Oct 1, 2012 at 7:46 PM, Hiller, Dean  wrote:
> Thanks, (actually new it was configurable) BUT what I don't get is why I
> have to run a repair.  IF all nodes became consistent on the delete, it
> should not be possible to get a forgotten delete, correct.  The forgotten
> delete will only occur if I have a node down and out for 10 days and it
> comes back online because by then the nodes no longer have the delete
> anymore and the new node has data so getting to a consistent state the
> node with data would win.
>
> S, if I run repair say every 20 days, isn't it true, I would have no
> problems as long as I did not have a node outage?

Basically if you know for certain that you have 100% uptime for all
your nodes and you haven't lost any updates due to overload/etc then
you don't need to run repair.  You run repair to ensure that all the
tombstones are replicated to all the necessary nodes prior to gc_grace
so that the data doesn't come back from the dead after a compaction.
Generally speaking it's a lot safer/easier to just always run repair.

> And most importantly, does anyone know of an automated tool for running
> repairs every X days(this should really be an automated/schedulable
> thing)???

I use a cron job.  It's a good idea to use the '-pr' flag btw.  Also,
you only need to run repair against CF's which actually have deletes.

-- 
Aaron Turner
http://synfin.net/ Twitter: @synfinatic
http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary
Safety, deserve neither Liberty nor Safety.
-- Benjamin Franklin
"carpe diem quam minimum credula postero"

Re: cassandra key cache question

[Tamar Fraenkel]

Created https://issues.apache.org/jira/browse/CASSANDRA-4742
Any clue regarding the first question in this thread (key cache > number of
keys in CF, and not many deletes on that CF)?

Thanks,

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

ta...@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Mon, Oct 1, 2012 at 2:28 AM, aaron morton wrote:

> I had a quick look at the code (1.1X) and could not see a facility to
> purge dropped cf's from key or row caches.
>
> Could you create a ticket on
> https://issues.apache.org/jira/browse/CASSANDRA ?
>
> Cheers
>
>   -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/09/2012, at 9:58 PM, Tamar Fraenkel  wrote:
>
> Hi!
>
> One more question:
> I have couple of dropped column families, and in the JMX console I don't
> see them under org.apache.cassandra.db.ColumnFamilies, *BUT *I do see
> them under org.apache.cassandra.db.Caches, and the cache is not empty!
> Does it mean that Cassandra still keep memory busy doing caching for a
> non-existing column family? If so, how do I remove those caches?
>
> Thanks!
>
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> 
>
>
> ta...@tok-media.com
> Tel:   +972 2 6409736
> Mob:  +972 54 8356490
> Fax:   +972 2 5612956
>
>
>
>
>
> On Thu, Sep 27, 2012 at 11:45 AM, Tamar Fraenkel wrote:
>
>> Hi!
>> Is it possible that in JMX and cfstats the Key cache size is much bigger
>> than the number of keys in the CF?
>> Thanks,
>>
>> *Tamar Fraenkel *
>> Senior Software Engineer, TOK Media
>>
>> 
>>
>>
>> ta...@tok-media.com
>> Tel:   +972 2 6409736
>> Mob:  +972 54 8356490
>> Fax:   +972 2 5612956
>>
>>
>>
>>
>
>
<>

Re: 1000's of column families

[Ben Hood]

Brian,

On Mon, Oct 1, 2012 at 4:22 PM, Brian O'Neill  wrote:
> We haven't committed either way yet, but given Ed Anuff's presentation
> on virtual keyspaces, we were leaning towards a single column family
> approach:
> http://blog.apigee.com/detail/building_a_mobile_data_platform_with_cassandra_-_apigee_under_the_hood/?

Is this doing something special or is this just a convenience way of
prefixing keys to make the storage space multi-tenanted?

Cheers,

Ben

Re: 1000's of column families

[Brian O'Neill]

Its just a convenient way of prefixing:
http://hector-client.github.com/hector/build/html/content/virtual_keyspaces.html

-brian

On Mon, Oct 1, 2012 at 4:22 PM, Ben Hood <0x6e6...@gmail.com> wrote:
> Brian,
>
> On Mon, Oct 1, 2012 at 4:22 PM, Brian O'Neill  wrote:
>> We haven't committed either way yet, but given Ed Anuff's presentation
>> on virtual keyspaces, we were leaning towards a single column family
>> approach:
>> http://blog.apigee.com/detail/building_a_mobile_data_platform_with_cassandra_-_apigee_under_the_hood/?
>
> Is this doing something special or is this just a convenience way of
> prefixing keys to make the storage space multi-tenanted?
>
> Cheers,
>
> Ben



-- 
Brian ONeill
Lead Architect, Health Market Science (http://healthmarketscience.com)

mobile:215.588.6024
blog: http://brianoneill.blogspot.com/
twitter: @boneill42

Re: pig and widerows

[aaron morton]

That looks like it may be a bug, can you create a ticket on 
https://issues.apache.org/jira/browse/CASSANDRA ?

Thanks

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 28/09/2012, at 7:50 AM, William Oberman  wrote:

> I don't want to switch my cassandra to HEAD, but looking at the newest code 
> for CassandraStorage, I'm concerned the Uri parsing for widerows isn't going 
> to work.  setLocation first calls setLocationFromUri (which sets widerows to 
> the Uri value), but then sets widerows to a static value (which is defined as 
> false), and then it sets widerows to the system setting if it exists.  That 
> doesn't seem right...  ?
> 
> But setLocationFromUri also gets called from setStoreLocation, and I don't 
> really know the difference between setLocation and setStoreLocation in terms 
> of what is going on in terms of the integration between cassandra/pig/hadoop.
> 
> will
> 
> On Thu, Sep 27, 2012 at 3:26 PM, William Oberman  
> wrote:
> The next painful lesson for me was figuring out how to get logging working 
> for a distributed hadoop process.   In my test environment, I have a single 
> node that runs name/secondaryname/data/job trackers (call it "central"), and 
> I have two cassandra nodes running tasktrackers.  But, I also have cassandra 
> libraries on the central box, and invoke my pig script from there.   I had 
> been patching and recompiling cassandra (1.1.5 with my logging, and the 
> system env fix) on that central box, and SOME of the logging was appearing in 
> the pig output.  But, eventually I decided to move that recompiled code to 
> the tasktracker boxes, and then I found even more of the logging I had added 
> in:
> /var/log/hadoop/userlogs/JOB_ID
> on each of the tasktrackers.
> 
> Based on this new logging, I found out that the widerows setting wasn't 
> propagating from the central box to the tasktrackers.  I added:
> export PIG_WIDEROW_INPUT=true
> To hadoop-env.sh on each of the tasktrackers and it finally worked!  
> 
> So, long story short, to actually get all columns for a key I had to:
> 1.) patch 1.1.5 to honor the "PIG_WIDEROW_INPUT=true" system setting
> 2.) add the system setting to ALL nodes in the hadoop cluster
> 
> I'm going to try to undo all of my other hacks to get logging/printing 
> working to confirm if those were actually the only two changes I had to make.
> 
> will
> 
> 
> On Thu, Sep 27, 2012 at 1:43 PM, William Oberman  
> wrote:
> Ok, this is painful.  The first problem I found is in stock 1.1.5 there is no 
> way to set widerows to true!  The new widerows URI parsing is NOT in 1.1.5.  
> And for extra fun, getting the value from the system property is BROKEN (at 
> least in my centos linux environment).
> 
> Here are the key lines of code (in CassandraStorage), note the different ways 
> of getting the property!  getenv in the test, and getProperty in the set:
> widerows = DEFAULT_WIDEROW_INPUT;
> if (System.getenv(PIG_WIDEROW_INPUT) != null)
> widerows = Boolean.valueOf(System.getProperty(PIG_WIDEROW_INPUT));
> 
> I added this logging:
> logger.warn("widerows = " + widerows + " getenv=" + 
> System.getenv(PIG_WIDEROW_INPUT) + " 
> getProp="+System.getProperty(PIG_WIDEROW_INPUT));
> 
> And I saw:
> org.apache.cassandra.hadoop.pig.CassandraStorage - widerows = false 
> getenv=true getProp=null
> So for me getProperty != getenv :-(
> 
> For people trying to figure out how to debug cassandra + hadoop + pig, for me 
> the key to get debugging and logging working was to focus on /etc/hadoop/conf 
> (not /etc/pig/conf as I expected).  
> 
> Also, if you want to compile your own cassandra (to add logging messages), 
> make sure it's appears first on the pig classpath (use pig -secretDebugCmd to 
> see the fully qualified command line).
> 
> The next thing I'm trying to figure out is why when widerows == true I'm 
> STILL not seeing more than 1024 columns :-( 
> 
> will
>  
> On Wed, Sep 26, 2012 at 3:42 PM, William Oberman  
> wrote:
> Hi,
> 
> I'm trying to figure out what's going on with my cassandra/hadoop/pig system. 
>  I created a "mini" copy of my main cassandra data by randomly subsampling to 
> get ~50,000 keys.  I was then writing pig scripts but also the equivalent 
> operation using simple single threaded code to double check pig.
> 
> Of course my very first test failed.  After doing a pig DUMP on the raw data, 
> what appears to be happening is I'm only getting the first 1024 columns of a 
> key.  After some googling, this seems to be known behavior unless you add 
> "?widerows=true" to the pig load URI. I tried this, but it didn't seem to fix 
> anything :-(   Here's the the start of my pig script:
> foo = LOAD 'cassandra://KEYSPACE/COLUMN_FAMILY?widerows=true' USING 
> CassandraStorage() AS (key:chararray, columns:bag {column:tuple (name, 
> value)});
> 
> I'm using cassandra 1.1.5 from datastax rpms.  I'm using hadoop 
> (0.20.2+923.418-1) and pig (0.8

Re: How to stop streaming

[aaron morton]

I assume you have resolved this but if someone else is looking…

You can stop a validation compaction with nodetool stop

AFAIk the only way to stop streaming is to restart the node. 

You can also use nodetool setstreamthroughput to reduce the streaming 
throughput. 

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 29/09/2012, at 4:02 PM, Senthilvel Rangaswamy  wrote:

> Hello,
> 
> We started a repair on one of our nodes without the -pr option. That started 
> a series
> of repair on all ranges and we see a ton of streaming activity between lot of 
> nodes in
> the cluster. This is putting enormous strain on the cluster. How can I stop 
> all the streaming.
> 
> The original repair is already over.
> 
> -- 
> ..Senthil
> 
> "If there's anything more important than my ego around, I want it 
>  caught and shot now."
> - Douglas Adams.
> 

Re: Remove node from cluster and have it run as a single node cluster by itself

[aaron morton]

> The other nodes may be trying to connect to it - it may be listed as a
> seed node on the other machines?
The other nodes will be looking for it. 

Change the Cluster Name in the yaml file. 

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 30/09/2012, at 12:04 AM, Tim Wintle  wrote:

> On Fri, 2012-09-28 at 18:53 +, Xu, Zaili wrote:
>> Hi,
>> 
>> I have an existing Cassandra Cluster. I removed a node from the cluster. 
>> Then I decommissioned the removed node, stopped it,  updated its config so 
>> that it only has itself as the seed and in the cassandra-topology.properties 
>> file, even deleted the data, commitlog, and saved_caches. But as soon as I 
>> start it backup it is able to join back to the cluster.  How does this node 
>> know the information of the existing cluster and was able to join it ?
> 
> The other nodes may be trying to connect to it - it may be listed as a
> seed node on the other machines?
> 
> Tim
> 
> 

Re: 1000's of column families

[Ben Hood]

On Mon, Oct 1, 2012 at 9:38 PM, Brian O'Neill  wrote:
> Its just a convenient way of prefixing:
> http://hector-client.github.com/hector/build/html/content/virtual_keyspaces.html

So given that it is possible to use a CF per tenant, should we assume
that there at sufficient scale that there is less overhead to prefix
keys than there is to manage multiple CFs?

Ben

Re: Data Modeling: Comments with Voting

[aaron morton]

You cannot (and probably do not want to) sort continually when the voting is 
going on. 

You can store the votes using CounterColumnTypes in column values. When someone 
votes you then (somehow) queue a job that will read the vote counts for the 
post / comment, pivot and sort on the vote count, and then write the updated 
leader board to cassandra. 

Alternatively if you have a small number of comments for a post just read all 
the votes and sort them as part of the read. 

Cheers
  
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 30/09/2012, at 8:25 AM, Drew Kutcharian  wrote:

> Thanks Roshni,
> 
> I'm not sue how #d will work when users are actually voting on a comment. 
> What happens when two users vote on the same comment simultaneously? How do 
> you update the entries in #d column family to prevent duplicates?
> 
>  Also #a and #c can be combined together using TimeUUID as comment ids.
> 
> - Drew
> 
> 
> 
> On Sep 27, 2012, at 2:13 AM, Roshni Rajagopal  
> wrote:
> 
>> Hi Drew,
>> 
>> I think you have 4 requirements. Here are my suggestions.
>> 
>> a) store comments : have a static column family for comments with master 
>> data like created date, created by , length etc
>> b) when a person votes for a comment, increment a vote counter : have a 
>> counter column family for incrementing the votes for each comment
>> c) display comments sorted by date created: have a column family with a 
>> dummy row id  'sort_by_time_list',  column names can be date 
>> created(timeUUID), and column value can be comment id 
>> d) display comments sorted by number of votes: have a column family with a 
>> dummy row id 'sort_by_votes_list' and column names can be a composite of 
>> number of votes , and comment id ( as more than 1 comment can have the same 
>> votes)
>> 
>> 
>> Regards,
>> Roshni
>> 
>> > Date: Wed, 26 Sep 2012 17:36:13 -0700
>> > From: k...@mustardgrain.com
>> > To: user@cassandra.apache.org
>> > CC: d...@venarc.com
>> > Subject: Re: Data Modeling: Comments with Voting
>> > 
>> > Depending on your needs, you could simply duplicate the comments in two 
>> > separate CFs with the column names including time in one and the vote in 
>> > the other. If you allow for updates to the comments, that would pose 
>> > some issues you'd need to solve at the app level.
>> > 
>> > On 9/26/12 4:28 PM, Drew Kutcharian wrote:
>> > > Hi Guys,
>> > >
>> > > Wondering what would be the best way to model a flat (no sub comments, 
>> > > i.e. twitter) comments list with support for voting (where I can sort by 
>> > > create time or votes) in Cassandra?
>> > >
>> > > To demonstrate:
>> > >
>> > > Sorted by create time:
>> > > - comment 1 (5 votes)
>> > > - comment 2 (1 votes)
>> > > - comment 3 (no votes)
>> > > - comment 4 (10 votes)
>> > >
>> > > Sorted by votes:
>> > > - comment 4 (10 votes)
>> > > - comment 1 (5 votes)
>> > > - comment 2 (1 votes)
>> > > - comment 3 (no votes)
>> > >
>> > > It's the sorted-by-votes that I'm having a bit of a trouble with. I'm 
>> > > looking for a roll-your-own approach and prefer not to use secondary 
>> > > indexes and CQL sorting.
>> > >
>> > > Thanks,
>> > >
>> > > Drew
>> > >
>> > 

Re: Prevent queries from OOM nodes

[aaron morton]

> It's not about columns, it's about rows, see example statement.
My bad, misread the CQL.

My jira search fu is poor, but I could not find an open ticket for paging row 
counts. Could you create one ? 
https://issues.apache.org/jira/browse/CASSANDRA

Cheers

 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 2/10/2012, at 12:17 AM, Віталій Тимчишин  wrote:

> It's not about columns, it's about rows, see example statement.
> In QueryProcessor#processStatement it reads rows into list, then does 
> list.size()
> 
> 2012/10/1 aaron morton 
>> CQL will read everything into List to make latter a count.
> 
>  
> From 1.0 onwards count paginated reading the columns. What version are you on 
> ? 
> 
> https://issues.apache.org/jira/browse/CASSANDRA-2894
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 26/09/2012, at 8:26 PM, Віталій Тимчишин  wrote:
> 
>> Actually an easy way to put cassandra down is
>> select count(*) from A limit 1000
>> CQL will read everything into List to make latter a count.
>> 
>> 2012/9/26 aaron morton 
>> Can you provide some information on the queries and the size of the data 
>> they traversed ? 
>> 
>> The default maximum size for a single thrift message is 16MB, was it larger 
>> than that ? 
>> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L375
>> 
>> Cheers
>> 
>> 
>> On 25/09/2012, at 8:33 AM, Bryce Godfrey  wrote:
>> 
>>> Is there anything I can do on the configuration side to prevent nodes from 
>>> going OOM due to queries that will read large amounts of data and exceed 
>>> the heap available? 
>>>  
>>> For the past few days of we had some nodes consistently freezing/crashing 
>>> with OOM.  We got a heap dump into MAT and figured out the nodes were dying 
>>> due to some queries for a few extremely large data sets.  Tracked it back 
>>> to an app that just didn’t prevent users from doing these large queries, 
>>> but it seems like Cassandra could be smart enough to guard against this 
>>> type of thing?
>>>  
>>> Basically some kind of setting like “if the data to satisfy query > 
>>> available heap then throw an error to the caller and about query”.  I would 
>>> much rather return errors to clients then crash a node, as the error is 
>>> easier to track down that way and resolve.
>>>  
>>> Thanks.
>> 
>> 
>> 
>> 
>> -- 
>> Best regards,
>>  Vitalii Tymchyshyn
> 
> 
> 
> 
> -- 
> Best regards,
>  Vitalii Tymchyshyn

Read latency issue

[Arindam Barua]

We are trying to setup a Cassandra cluster and have low read latency 
requirements. Running some tests, we do not see the performance that we were 
hoping for. Wanted to check if anyone has thoughts on:

1.   If these are expected latency times for the data/machine config, etc

2.   If not, can do something to improve our read times?

We set up 4 boxes as a ring running Cassandra 1.1.5, and setup a keyspace with 
replication 3, and strategy_class SimpleStrategy. The column family being 
tested has 12 columns, 4 of which form a composite key.
We then wrote in 192,000 randomly generated test data rows into the column 
family. Most columns are either randomly generated UUIDs, or short strings. One 
of them however is a blob consisting of around 1K data (we later reduced the 
size of this blob data, but didn't seem to change our read times much)

Running a query to like "select * from  where atag=", where 
'atag' is the first column of the composite key, from either JDBC or Hector 
(equivalent code), results in read times of 200-300ms from a remote host on the 
same network. The query returned around 800 results. Running the same query on 
a Cassandra host results in a read time of ~110-130 ms.
Using read consistency of ONE reduces the read latency by ~20ms, compared to 
using QUORUM.

Enabling row cache did not seem to change the performance much. Moreover, the 
row cache 'size' according to nodetool was very tiny. Here is a snapshot of the 
nodetool info after running few read tests:
Key Cache: size 2448 (bytes), capacity 104857584 (bytes), 231 hits, 266 
requests, 1.000 recent hit rate, 14400 save period in seconds
Row Cache: size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 
requests, NaN recent hit rate, 0 save period in seconds

Hardware/OS specs:
Intel(R) Xeon(R) CPU L5640
OS: Solaris 5.10
RAM: 32 GB
Hard disk: 1 TB disk magnetic (not SSD)

Thanks,
Arindam

RE: Data Modeling: Comments with Voting

[Roshni Rajagopal]

Hi , 
To explain my suggestions - my thoughts were 
a) you need to store entity type information about a comment like date created, 
comment text, commented by etc. I cant think of any other master information 
for a comment, but in general one starts with entities in a standard static 
column family.  If you store an entity in a dynamic denormailized form, if any 
master data changes you would need to iterate across all rows and update it 
which is expensive in cassandra. Here comment text is editable.
b) So when a comment is created it goes to the static column family. Also an 
entry is made in the dynamic sort_by_time_list column family with column as 
time created. I didn't suggest a and c be clubbed so that master information 
remains in one place. The other approach would be to have a comment stored as a 
JSON in the column value. However if you need to update comment text, it 
would be hard to identify the comment column and update it. c) when a comment 
gets a vote, the counter column family is incremented to know the number of 
votes for a comment. Also to sort by number of votes  , after incrementing the 
counter you need to write the current number of votes, and the comment id in 
the column family d. But I see now that you also need to delete the old number 
of votes & comment id column and add a new  column with current number of votes 
and comment id. It would be sorted by number of votes.
If there are many ways to sort, its better to do it in the application to avoid 
having a new column family for each type of sort...however Im not certain over 
time and volume which approach would perform better.Sorting can be complex - 
aaron's blog post http://thelastpickle.com/2012/08/18/Sorting-Lists-For-Humans/ 
 
Welcome any feedback on my suggestions.


From: aa...@thelastpickle.com
Subject: Re: Data Modeling: Comments with Voting
Date: Tue, 2 Oct 2012 10:39:42 +1300
To: user@cassandra.apache.org

You cannot (and probably do not want to) sort continually when the voting is 
going on. 
You can store the votes using CounterColumnTypes in column values. When someone 
votes you then (somehow) queue a job that will read the vote counts for the 
post / comment, pivot and sort on the vote count, and then write the updated 
leader board to cassandra. 
Alternatively if you have a small number of comments for a post just read all 
the votes and sort them as part of the read. 
Cheers  

-Aaron MortonFreelance 
Developer@aaronmortonhttp://www.thelastpickle.com



On 30/09/2012, at 8:25 AM, Drew Kutcharian  wrote:Thanks 
Roshni,
I'm not sue how #d will work when users are actually voting on a comment. What 
happens when two users vote on the same comment simultaneously? How do you 
update the entries in #d column family to prevent duplicates?
 Also #a and #c can be combined together using TimeUUID as comment ids.
- Drew



On Sep 27, 2012, at 2:13 AM, Roshni Rajagopal  
wrote:





Hi Drew,
I think you have 4 requirements. Here are my suggestions.
a) store comments : have a static column family for comments with master data 
like created date, created by , length etcb) when a person votes for a comment, 
increment a vote counter : have a counter column family for incrementing the 
votes for each commentc) display comments sorted by date created: have a column 
family with a dummy row id  'sort_by_time_list',  column names can be date 
created(timeUUID), and column value can be comment id d) display comments 
sorted by number of votes: have a column family with a dummy row id 
'sort_by_votes_list' and column names can be a composite of number of votes , 
and comment id ( as more than 1 comment can have the same votes)

Regards,Roshni

> Date: Wed, 26 Sep 2012 17:36:13 -0700
> From: k...@mustardgrain.com
> To: user@cassandra.apache.org
> CC: d...@venarc.com
> Subject: Re: Data Modeling: Comments with Voting
> 
> Depending on your needs, you could simply duplicate the comments in two 
> separate CFs with the column names including time in one and the vote in 
> the other. If you allow for updates to the comments, that would pose 
> some issues you'd need to solve at the app level.
> 
> On 9/26/12 4:28 PM, Drew Kutcharian wrote:
> > Hi Guys,
> >
> > Wondering what would be the best way to model a flat (no sub comments, i.e. 
> > twitter) comments list with support for voting (where I can sort by create 
> > time or votes) in Cassandra?
> >
> > To demonstrate:
> >
> > Sorted by create time:
> > - comment 1 (5 votes)
> > - comment 2 (1 votes)
> > - comment 3 (no votes)
> > - comment 4 (10 votes)
> >
> > Sorted by votes:
> > - comment 4 (10 votes)
> > - comment 1 (5 votes)
> > - comment 2 (1 votes)
> > - comment 3 (no votes)
> >
> > It's the sorted-by-votes that I'm having a bit of a trouble with. I'm 
> > looking for a roll-your-own approach and prefer not to use secondary 
> > indexes and CQL sorting.
> >
> > Thanks,
> >
> > Drew
> >
> 
  

Re: How to stop streaming

[Senthilvel Rangaswamy]

Thanks Aaron. That's what I did.

Also, how do I lookup the current value of the setstreamthroughput.

On Mon, Oct 1, 2012 at 2:32 PM, aaron morton wrote:

> I assume you have resolved this but if someone else is looking…
>
> You can stop a validation compaction with nodetool stop
>
> AFAIk the only way to stop streaming is to restart the node.
>
> You can also use nodetool setstreamthroughput to reduce the streaming
> throughput.
>
> Cheers
>
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 29/09/2012, at 4:02 PM, Senthilvel Rangaswamy 
> wrote:
>
> Hello,
>
> We started a repair on one of our nodes without the -pr option. That
> started a series
> of repair on all ranges and we see a ton of streaming activity between lot
> of nodes in
> the cluster. This is putting enormous strain on the cluster. How can I
> stop all the streaming.
>
> The original repair is already over.
>
> --
> ..Senthil
>
> "If there's anything more important than my ego around, I want it
>  caught and shot now."
> - Douglas Adams.
>
>
>


-- 
..Senthil

"If there's anything more important than my ego around, I want it
 caught and shot now."
- Douglas Adams.

Re: Why data tripled in size after repair?

[Peter Schuller]

> It looks like what I need. Couple questions.
> Does it work with RandomPartinioner only? I use ByteOrderedPartitioner.

I believe it should work with BOP based on cursory re-examination of
the patch. I could be wrong.

> I don't see it as part of any release. Am I supposed to build my own
> version of cassandra?

It's in the 1.1 branch; I don't remember if it went into a release
yet. If not, it'll be in the next 1.1.x release.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)