node repair eat up all disk io and slow down entire cluster(3 nodes)

[Yan Chunlu]

at the beginning of using cassandra, I have no idea that I should run "node
repair" frequently, so basically, I have 3 nodes with RF=3 and have not run
node repair for months, the data size is 20G.

the problem is when I start running node repair now, it eat up all disk io
and the server load became 20+ and increasing, the worst thing is, the
entire cluster has slowed down and can not handle request. so I have to stop
it immediately because it make my web service unavailable.

the server has Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] and 8G memory,
with Western Digital WD RE3 WD1002FBYS SATA disk.

I really have no idea what to do now, as currently I have already found some
data loss, any suggestions would be appreciated.

Cassandra training in Bangalore, India

[CASSANDRA learner]

Is there anywhere the training on cassandra is available in bamgalore, India

RE: How to keep only exactly column of key

[Lior Golan]

Thanks Sylvain

Can you please point us to what interface should be implemented in order to 
write our own custom compaction. And how is it supposed to be configured?

-Original Message-
From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
Sent: Tuesday, July 19, 2011 11:40 AM
To: user@cassandra.apache.org
Subject: Re: How to keep only exactly column of key

On Tue, Jul 19, 2011 at 10:15 AM, Lior Golan  wrote:
> Can't this capping be done (approximately) during compaction. 
> Something
> like:
>
> 1.   Ability to define for a column family that it's a "capped 
> collection" with at most N columns per row
>
> 2.   During write - just add the column
>
> 3.   During reads - get a slice with the most recent / top N 
> column (in terms of column order)
>
> 4.   During compaction - if the number of columns in the row is 
> more than N, trim it to the top N columns (by replacing the rest of 
> the columns with a tombstone in the compacted row)
>
> Since I guess the purpose of this is for automated cleanup, and not 
> for enforcing exactly N columns, I think this would be sufficient

The problem with that is that we cannot enforce this on the query side.
Or more precisely, returning the top N first columns is fine, but what with 
query like "M columns starting from 'b'" ? Or columns by name ?
We cannot do those efficiently while enforcing that we won't return any columns 
after the N first ones. The only solution would be to always query the first N 
ones and then filter afterwards, but that's not efficient.

What I mean here is that it is hard to add that as a column family option given 
the limitation it would entail. That being said, 1.0 will add pluggable 
compaction (it's already in trunk) and it will be very easy to have a 
compaction that just drop columns after the N first. It would then be on the 
client side to deal with the possibility to get more that the first N ones, but 
as you said, if it is for automated cleanup, that will be enough.

--
Sylvain

> From: Tupshin Harper [mailto:tups...@tupshin.com]
> Sent: Tuesday, July 19, 2011 10:04 AM
> To: user@cassandra.apache.org
> Subject: Re: How to keep only exactly column of key
>
>
>
> Speaking from practical experience, it is possible to simulate this 
> feature by retrieving a slice of your row that only contains the most 
> recent 100 items. You can then prevent the rows from growing out of 
> control by checking the size of the row and pruning it back to 100 
> every N writes, where N is small enough to prevent excessive growth, 
> but large enough to prevent excessive overhead. A value of 50 or so 
> for N worked reasonably well for me for. If you do go down this path, 
> though, keep in mind that rapid writes and deletes to a single column 
> are basically a Cassandra anti-pattern due to performance problems with huge 
> numbers of tombstones.
>
>
>
> I would love to see a feature added similar to MongoDB's "capped 
> collections", but I don't believe there is any easy way to retrofit it 
> into Cassandra's sstable approach.
> http://www.mongodb.org/display/DOCS/Capped+Collections
>
>
>
> -Tupshin
>
> On Mon, Jul 18, 2011 at 8:22 AM, JKnight JKnight 
> wrote:
>
> Dear all,
>
>
>
> I want to keep only 100 column of a key: when I add a column for a 
> key, if the number column of key is 100, another column (by order) will be 
> deleted.
>
>
>
> Does Cassandra have setting for that?
>
> --
> Best regards,
> JKnight
>
>

best example of indexing

[CASSANDRA learner]

Hi Guys,

Can you please give me the best example of creating index on a column
family. As I am completely new to this, Can you please give me a simple and
good example.

Re: Need help json2sstable

[Nilabja Banerjee]

Yes.Actually, I was just asking you guys to give me one example with one
sample of small json structure.

Thank you in advance :)

On 20 July 2011 11:53, Sasha Dolgy  wrote:

> You are missing " after  
>
> On Wed, Jul 20, 2011 at 8:03 AM, Nilabja Banerjee
>  wrote:
> > Hi All,
> >
> > Here Is my Json structure.
> >
> >
> > {"Fetch_CC" :{
> > "cc":{ "":"1000",
> >  ":"ICICI",
> >  "":"",
> >  "city":{
> >  "name":"banglore"
> >};
> >};
> > }
> >
> > If the structure is incorrect, please give me one small structre to use
> > below utility.
> > I am using 0.7.5 version.
> > Now how can I can use Json2SStable utilities? Please provide me the
> steps.
> > What are the things I have configure?
> >
> > Thank You
> >
>
>
>
> --
> Sasha Dolgy
> sasha.do...@gmail.com
>

Re: Need help json2sstable

[Nilabja Banerjee]

On 20 July 2011 11:33, Nilabja Banerjee  wrote:

> Hi All,
>
> Here Is my Json structure.
>
>
> {"Fetch_CC" :{
> "cc":{ "":"1000",
>  ":"ICICI",
>  "":"",
>  "city":{
>  "name":"banglore"
>};
>};
> }
>
> If the structure is incorrect, please give me one small structre to use
> below utility.
> I am using 0.7.5 version.
> Now how can I can use Json2SStable utilities? Please provide me the steps.
> What are the things I have configure?
>
> Thank You
>

Re: best example of indexing

[Sasha Dolgy]

Examples exist in the conf directory of the distribution...
On Jul 20, 2011 11:48 AM, "CASSANDRA learner" 
wrote:
> Hi Guys,
>
> Can you please give me the best example of creating index on a column
> family. As I am completely new to this, Can you please give me a simple
and
> good example.

Re: best example of indexing

[CASSANDRA learner]

where can i get that. Can you please help me out

On Wed, Jul 20, 2011 at 3:39 PM, Sasha Dolgy  wrote:

> Examples exist in the conf directory of the distribution...
> On Jul 20, 2011 11:48 AM, "CASSANDRA learner" 
> wrote:
> > Hi Guys,
> >
> > Can you please give me the best example of creating index on a column
> > family. As I am completely new to this, Can you please give me a simple
> and
> > good example.
>

Cassandra & CLOUD . How its related

[CASSANDRA learner]

Hi Guys,

When we talk about cassandra, any how we connect it to cloud. I dont
understand how it is connected to cloud. Whats this Cassandra Cloud.

2800 file descriptors?

[cbert...@libero.it]

Hi all,
I wonder if is normal that Cassandra (5 nodes, 0.75) has more than 2800 fd 
open and growing.
I still have the problem that during repair I get into the "too many open 
files"

Best regards

What is the nodeId for?

[Boris Yen]

Hi,

I think we might have screwed our data up. I saw multiple columns inside
record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
I was wondering if anyone could tell me what the NodeId is for? so that I
might be able to duplicate this.

Thanks in advance
Boris

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

[Yan Chunlu]

just found this:
https://issues.apache.org/jira/browse/CASSANDRA-2156

but seems only available to 0.8 and people submitted a patch for 0.6, I am
using 0.7.4, do I need to dig into the code and make my own patch?

does add compaction throttle solve the io problem?  thanks!

On Wed, Jul 20, 2011 at 4:44 PM, Yan Chunlu  wrote:

> at the beginning of using cassandra, I have no idea that I should run "node
> repair" frequently, so basically, I have 3 nodes with RF=3 and have not run
> node repair for months, the data size is 20G.
>
> the problem is when I start running node repair now, it eat up all disk io
> and the server load became 20+ and increasing, the worst thing is, the
> entire cluster has slowed down and can not handle request. so I have to stop
> it immediately because it make my web service unavailable.
>
> the server has Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] and 8G memory,
> with Western Digital WD RE3 WD1002FBYS SATA disk.
>
> I really have no idea what to do now, as currently I have already found
> some data loss, any suggestions would be appreciated.
>



-- 
闫春路

Re: 2800 file descriptors?

[Boris Yen]

For the "too many open files" issue, maybe you could try:  ulimit -n 5000 &&
.

On Wed, Jul 20, 2011 at 6:47 PM, cbert...@libero.it wrote:

> Hi all,
> I wonder if is normal that Cassandra (5 nodes, 0.75) has more than 2800 fd
> open and growing.
> I still have the problem that during repair I get into the "too many open
> files"
>
> Best regards
>

Re: What is the nodeId for?

[Sam Overton]

The NodeId is used in counter replication. Counters are stored on each
replica as a set of "shards," where each shard corresponds to the local
count of one of the replicas for that counter, as identified by the NodeId.

A NodeId is generated the first time cassandra starts, and might be renewed
during cleanup, or if you explicitly tell cassandra to generate a new one by
starting it with -D*cassandra*.renew_counter_id

Do either of the NodeIds in the CurrentLocal row also appear in the Local
row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
belonging to this node. If one appears in the history and the other does
not, it is probably safe to assume that the old NodeId was not deleted from
the CurrentLocal row correctly, and so you could try removing it from there
manually.

Sam
--
Sam Overton
Acunu | http://www.acunu.com | @acunu

On 20 July 2011 12:25, Boris Yen  wrote:

> Hi,
>
> I think we might have screwed our data up. I saw multiple columns inside
> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
> I was wondering if anyone could tell me what the NodeId is for? so that I
> might be able to duplicate this.
>
> Thanks in advance
> Boris
>

network bandwidth question

[Arijit Mukherjee]

Hi All

We're trying to set up a Cassandra cluster (initially with 3 nodes). Each
node will generate data @ 32MB per second. What would be the likely network
usage for this (say with a replication factor of 3)?

I mean, if I use simple arithmetic, I can say 32MBps per node, and hence
96MBps in total as network traffic - but should I consider any other factors
(like handshaking or keep-alive packets)?

Regards
Arijit

-- 
"And when the night is cloudy,
There is still a light that shines on me,
Shine on until tomorrow, let it be."

R: Re: 2800 file descriptors?

[cbert...@libero.it]

> For the "too many open files" issue, maybe you could try:  ulimit -n 5000 
> && .
Ok, thanks for the tip but I get this error running nodetool repair and not 
during cassandra execution. 
I however wonder if this is normal or not ... in production do you get similar 
numbers? Isn't it too much?
best regards




 

Re: network bandwidth question

[Jonathan Ellis]

You can assume that's negligible compared to the data traffic.

On Wed, Jul 20, 2011 at 7:02 AM, Arijit Mukherjee  wrote:
> Hi All
>
> We're trying to set up a Cassandra cluster (initially with 3 nodes). Each
> node will generate data @ 32MB per second. What would be the likely network
> usage for this (say with a replication factor of 3)?
>
> I mean, if I use simple arithmetic, I can say 32MBps per node, and hence
> 96MBps in total as network traffic - but should I consider any other factors
> (like handshaking or keep-alive packets)?
>
> Regards
> Arijit
>
> --
> "And when the night is cloudy,
> There is still a light that shines on me,
> Shine on until tomorrow, let it be."
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Re: 2800 file descriptors?

[Jonathan Ellis]

Repair does normally stream lots of small sstables.

It's normal to set open fd to unlimited, but a higher limit like 64K
would also be reasonable.

On Wed, Jul 20, 2011 at 7:02 AM, cbert...@libero.it  wrote:
>> For the "too many open files" issue, maybe you could try:  ulimit -n 5000
>> && .
>
> Ok, thanks for the tip but I get this error running nodetool repair and not
> during cassandra execution.
> I however wonder if this is normal or not ... in production do you get
> similar numbers? Isn't it too much?
>
> best regards
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Repair taking a long, long time

[David Boxenhorn]

I have this problem too, and I don't understand why.

I can repair my nodes very quickly by looping though all my data (when you
read your data it does read-repair), but nodetool repair takes forever. I
understand that nodetool repair builds merkle trees, etc. etc., so it's a
different algorithm, but why can't nodetool repair be smart enough to choose
the best algorithm? Also, I don't understand what's special about my data
that makes nodetool repair so much slower than looping through all my data.


On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin  wrote:

> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
> pretty fast.
> Seriously, in my house I copy complete DVD images from my bedroom to
> the living room downstairs via WiFi, and a dozen of GB does not seem like a
> problem, on dirt cheap hardware (Patriot Box Office).
>
> I also have just _one_ column major family but caveat emptor -- 8 indexes
> attached to
> it (and there will be more). There is one accounting CF which is small,
> can't possibly
> make a difference.
>
> By contrast, compaction (as in nodetool) performs quite well on this
> cluster. I start suspecting some
> sort of malfunction.
>
> Looked at the system log during the "repair", there is some compaction
> agent doing
> work that I'm not sure makes sense (and I didn't call for it). Disk
> utilization all of a sudden goes up to 40%
> per Ganglia, and stays there, this is pretty silly considering the cluster
> is IDLE and we have SSDs. No external writes,
> no reads. There are occasional GC stoppages, but these I can live with.
>
> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
> production soon
> and that doesn't look good at all. If I can't manage a system that simple
> (and/or get help
> on this list) I may have to cut losses i.e. stay with Oracle.
>
> Regards,
>
> Maxim
>
>
>
>
> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>
>>
>> Well most SSD's are pretty fast. There is one more to consider. If
>> Cassandra determines nodes are out of sync it has to transfer data across
>> the network. If that is the case you have to look at 'nodetool streams' and
>> determine how much data is being transferred between nodes. There are some
>> open tickets where with larger tables repair is streaming more then it needs
>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>> is not trivial.
>>
>> If you have multiple keyspaces and column families repair one at a time
>> might make the process more manageable.
>>
>
>

Re: What is the nodeId for?

[Boris Yen]

Hi Sam,

Thanks for the explanation.

The NodeIds do appear in the Local row of NodeIdInfo, and after manually
deleting two (I got three before I deleted them) of them from CurrentLocal
row, the cassandra can be restarted now. I was just thinking what could be
the possible cause for this? and wondering if anyone has any idea about
this?

Boris

On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton  wrote:

> The NodeId is used in counter replication. Counters are stored on each
> replica as a set of "shards," where each shard corresponds to the local
> count of one of the replicas for that counter, as identified by the NodeId.
>
> A NodeId is generated the first time cassandra starts, and might be renewed
> during cleanup, or if you explicitly tell cassandra to generate a new one by
> starting it with -D*cassandra*.renew_counter_id
>
> Do either of the NodeIds in the CurrentLocal row also appear in the Local
> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
> belonging to this node. If one appears in the history and the other does
> not, it is probably safe to assume that the old NodeId was not deleted from
> the CurrentLocal row correctly, and so you could try removing it from there
> manually.
>
> Sam
> --
> Sam Overton
> Acunu | http://www.acunu.com | @acunu
>
> On 20 July 2011 12:25, Boris Yen  wrote:
>
>> Hi,
>>
>> I think we might have screwed our data up. I saw multiple columns inside
>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
>> I was wondering if anyone could tell me what the NodeId is for? so that I
>> might be able to duplicate this.
>>
>> Thanks in advance
>> Boris
>>
>
>

Re: Repair taking a long, long time

[Maxim Potekhin]

I can re-load all data that I have in the cluster, from a flat-file 
cache I have

on NFS, many times faster than the nodetool repair takes. And that's not
even accurate because as other noted nodetool repair eats up disk space
for breakfast and takes more than 24hrs on 200GB data load, at which point
I have to cancel. That's not acceptable. I simply don't know what to do now.


On 7/20/2011 8:47 AM, David Boxenhorn wrote:

I have this problem too, and I don't understand why.

I can repair my nodes very quickly by looping though all my data (when 
you read your data it does read-repair), but nodetool repair takes 
forever. I understand that nodetool repair builds merkle trees, etc. 
etc., so it's a different algorithm, but why can't nodetool repair be 
smart enough to choose the best algorithm? Also, I don't understand 
what's special about my data that makes nodetool repair so much slower 
than looping through all my data.



On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin > wrote:


Thanks Edward. I'm told by our IT that the switch connecting the
nodes is pretty fast.
Seriously, in my house I copy complete DVD images from my bedroom to
the living room downstairs via WiFi, and a dozen of GB does not
seem like a
problem, on dirt cheap hardware (Patriot Box Office).

I also have just _one_ column major family but caveat emptor -- 8
indexes attached to
it (and there will be more). There is one accounting CF which is
small, can't possibly
make a difference.

By contrast, compaction (as in nodetool) performs quite well on
this cluster. I start suspecting some
sort of malfunction.

Looked at the system log during the "repair", there is some
compaction agent doing
work that I'm not sure makes sense (and I didn't call for it).
Disk utilization all of a sudden goes up to 40%
per Ganglia, and stays there, this is pretty silly considering the
cluster is IDLE and we have SSDs. No external writes,
no reads. There are occasional GC stoppages, but these I can live
with.

This repair debacle happens 2nd time in a row. Cr@p. I need to go
to production soon
and that doesn't look good at all. If I can't manage a system that
simple (and/or get help
on this list) I may have to cut losses i.e. stay with Oracle.

Regards,

Maxim




On 7/19/2011 12:16 PM, Edward Capriolo wrote:


Well most SSD's are pretty fast. There is one more to
consider. If Cassandra determines nodes are out of sync it has
to transfer data across the network. If that is the case you
have to look at 'nodetool streams' and determine how much data
is being transferred between nodes. There are some open
tickets where with larger tables repair is streaming more then
it needs to. But even if the transfers are only 10% of your
200GB. Transferring 20 GB is not trivial.

If you have multiple keyspaces and column families repair one
at a time might make the process more manageable.

Re: Repair taking a long, long time

[Boris Yen]

We also got the same problem when using 0.8.0. As far as I know, there are a
few issues relative to 'repair' has been marked as resolved at 0.8.1. Hope
this could really solve our problem.

On Wed, Jul 20, 2011 at 8:47 PM, David Boxenhorn  wrote:

> I have this problem too, and I don't understand why.
>
> I can repair my nodes very quickly by looping though all my data (when you
> read your data it does read-repair), but nodetool repair takes forever. I
> understand that nodetool repair builds merkle trees, etc. etc., so it's a
> different algorithm, but why can't nodetool repair be smart enough to choose
> the best algorithm? Also, I don't understand what's special about my data
> that makes nodetool repair so much slower than looping through all my data.
>
>
>
> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin  wrote:
>
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
>> pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like
>> a
>> problem, on dirt cheap hardware (Patriot Box Office).
>>
>> I also have just _one_ column major family but caveat emptor -- 8 indexes
>> attached to
>> it (and there will be more). There is one accounting CF which is small,
>> can't possibly
>> make a difference.
>>
>> By contrast, compaction (as in nodetool) performs quite well on this
>> cluster. I start suspecting some
>> sort of malfunction.
>>
>> Looked at the system log during the "repair", there is some compaction
>> agent doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk
>> utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster
>> is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>>
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
>> production soon
>> and that doesn't look good at all. If I can't manage a system that simple
>> (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>>
>> Regards,
>>
>> Maxim
>>
>>
>>
>>
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>>
>>>
>>> Well most SSD's are pretty fast. There is one more to consider. If
>>> Cassandra determines nodes are out of sync it has to transfer data across
>>> the network. If that is the case you have to look at 'nodetool streams' and
>>> determine how much data is being transferred between nodes. There are some
>>> open tickets where with larger tables repair is streaming more then it needs
>>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>>> is not trivial.
>>>
>>> If you have multiple keyspaces and column families repair one at a time
>>> might make the process more manageable.
>>>
>>
>>
>

Re: Repair taking a long, long time

[David Boxenhorn]

As I indicated below (but didn't say specifically) another option is to set
read repair chance to 1.0 for all your CFs and loop over all your data,
since read triggers a read repair.

On Wed, Jul 20, 2011 at 4:58 PM, Maxim Potekhin  wrote:

> **
> I can re-load all data that I have in the cluster, from a flat-file cache I
> have
> on NFS, many times faster than the nodetool repair takes. And that's not
> even accurate because as other noted nodetool repair eats up disk space
> for breakfast and takes more than 24hrs on 200GB data load, at which point
> I have to cancel. That's not acceptable. I simply don't know what to do
> now.
>
>
>
> On 7/20/2011 8:47 AM, David Boxenhorn wrote:
>
> I have this problem too, and I don't understand why.
>
> I can repair my nodes very quickly by looping though all my data (when you
> read your data it does read-repair), but nodetool repair takes forever. I
> understand that nodetool repair builds merkle trees, etc. etc., so it's a
> different algorithm, but why can't nodetool repair be smart enough to choose
> the best algorithm? Also, I don't understand what's special about my data
> that makes nodetool repair so much slower than looping through all my data.
>
>
> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin  wrote:
>
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is
>> pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like
>> a
>> problem, on dirt cheap hardware (Patriot Box Office).
>>
>> I also have just _one_ column major family but caveat emptor -- 8 indexes
>> attached to
>> it (and there will be more). There is one accounting CF which is small,
>> can't possibly
>> make a difference.
>>
>> By contrast, compaction (as in nodetool) performs quite well on this
>> cluster. I start suspecting some
>> sort of malfunction.
>>
>> Looked at the system log during the "repair", there is some compaction
>> agent doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk
>> utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster
>> is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>>
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to
>> production soon
>> and that doesn't look good at all. If I can't manage a system that simple
>> (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>>
>> Regards,
>>
>> Maxim
>>
>>
>>
>>
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>>
>>>
>>> Well most SSD's are pretty fast. There is one more to consider. If
>>> Cassandra determines nodes are out of sync it has to transfer data across
>>> the network. If that is the case you have to look at 'nodetool streams' and
>>> determine how much data is being transferred between nodes. There are some
>>> open tickets where with larger tables repair is streaming more then it needs
>>> to. But even if the transfers are only 10% of your 200GB. Transferring 20 GB
>>> is not trivial.
>>>
>>> If you have multiple keyspaces and column families repair one at a time
>>> might make the process more manageable.
>>>
>>
>>
>
>

disable compaction

[Nikolai Kopylov]

Hi everyone,

finding out recently that cassandra have no upper limit for sstable files to
grow, I decided to move to deletion of CF with obsolete data.
So that I will not remove columns and there is no need in compaction at all.
How can I completely disable the compaction process?

Thanx for your attention,
Nikolai

Re: disable compaction

[Edward Capriolo]

On Wed, Jul 20, 2011 at 11:13 AM, Nikolai Kopylov  wrote:

> Hi everyone,
>
> finding out recently that cassandra have no upper limit for sstable files
> to grow, I decided to move to deletion of CF with obsolete data.
> So that I will not remove columns and there is no need in compaction at
> all. How can I completely disable the compaction process?
>
> Thanx for your attention,
> Nikolai
>
>
This can be done from the CLI.
[default@unknown] help update column family;
update column family Bar;
update column family Bar with =;
update column family Bar with = and =...;

- min_compaction_threshold: Avoid minor compactions of less than this
number of sstable files
- max_compaction_threshold: Compact no more than this number of sstable
files at once

update column family xxx with min_compaction_threshold=0 and
max_compaction_threshold=0;

Rolling over data into new column families is one extreme way to avoid
compaction :)

Edward

Re: What is the nodeId for?

[Sylvain Lebresne]

Possibly, you've hitted this:
https://issues.apache.org/jira/browse/CASSANDRA-2824
Should be fixed in next minor release.

In the meantime, you "fix" should be alright.

--
Sylvain


On Wed, Jul 20, 2011 at 3:47 PM, Boris Yen  wrote:
> Hi Sam,
> Thanks for the explanation.
> The NodeIds do appear in the Local row of NodeIdInfo, and after manually
> deleting two (I got three before I deleted them) of them from CurrentLocal
> row, the cassandra can be restarted now. I was just thinking what could be
> the possible cause for this? and wondering if anyone has any idea about
> this?
> Boris
>
> On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton  wrote:
>>
>> The NodeId is used in counter replication. Counters are stored on each
>> replica as a set of "shards," where each shard corresponds to the local
>> count of one of the replicas for that counter, as identified by the NodeId.
>> A NodeId is generated the first time cassandra starts, and might be
>> renewed during cleanup, or if you explicitly tell cassandra to generate a
>> new one by starting it with -Dcassandra.renew_counter_id
>> Do either of the NodeIds in the CurrentLocal row also appear in the Local
>> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
>> belonging to this node. If one appears in the history and the other does
>> not, it is probably safe to assume that the old NodeId was not deleted from
>> the CurrentLocal row correctly, and so you could try removing it from there
>> manually.
>>
>> Sam
>> --
>> Sam Overton
>> Acunu | http://www.acunu.com | @acunu
>> On 20 July 2011 12:25, Boris Yen  wrote:
>>>
>>> Hi,
>>> I think we might have screwed our data up. I saw multiple columns inside
>>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead forever.
>>> I was wondering if anyone could tell me what the NodeId is for? so that I
>>> might be able to duplicate this.
>>> Thanks in advance
>>> Boris
>
>

Re: How to keep only exactly column of key

[Sylvain Lebresne]

The ticket for pluggable compaction is
https://issues.apache.org/jira/browse/CASSANDRA-1610.
It's not released yet, so there is not real documentation for this
yet. But if you really want to look into it, you can start looking at
AbstractCompactionStragegy in trunk.

--
Sylvain


On Wed, Jul 20, 2011 at 10:57 AM, Lior Golan  wrote:
> Thanks Sylvain
>
> Can you please point us to what interface should be implemented in order to 
> write our own custom compaction. And how is it supposed to be configured?
>
> -Original Message-
> From: Sylvain Lebresne [mailto:sylv...@datastax.com]
> Sent: Tuesday, July 19, 2011 11:40 AM
> To: user@cassandra.apache.org
> Subject: Re: How to keep only exactly column of key
>
> On Tue, Jul 19, 2011 at 10:15 AM, Lior Golan  wrote:
>> Can't this capping be done (approximately) during compaction.
>> Something
>> like:
>>
>> 1.   Ability to define for a column family that it's a "capped
>> collection" with at most N columns per row
>>
>> 2.   During write - just add the column
>>
>> 3.   During reads - get a slice with the most recent / top N
>> column (in terms of column order)
>>
>> 4.   During compaction - if the number of columns in the row is
>> more than N, trim it to the top N columns (by replacing the rest of
>> the columns with a tombstone in the compacted row)
>>
>> Since I guess the purpose of this is for automated cleanup, and not
>> for enforcing exactly N columns, I think this would be sufficient
>
> The problem with that is that we cannot enforce this on the query side.
> Or more precisely, returning the top N first columns is fine, but what with 
> query like "M columns starting from 'b'" ? Or columns by name ?
> We cannot do those efficiently while enforcing that we won't return any 
> columns after the N first ones. The only solution would be to always query 
> the first N ones and then filter afterwards, but that's not efficient.
>
> What I mean here is that it is hard to add that as a column family option 
> given the limitation it would entail. That being said, 1.0 will add pluggable 
> compaction (it's already in trunk) and it will be very easy to have a 
> compaction that just drop columns after the N first. It would then be on the 
> client side to deal with the possibility to get more that the first N ones, 
> but as you said, if it is for automated cleanup, that will be enough.
>
> --
> Sylvain
>
>> From: Tupshin Harper [mailto:tups...@tupshin.com]
>> Sent: Tuesday, July 19, 2011 10:04 AM
>> To: user@cassandra.apache.org
>> Subject: Re: How to keep only exactly column of key
>>
>>
>>
>> Speaking from practical experience, it is possible to simulate this
>> feature by retrieving a slice of your row that only contains the most
>> recent 100 items. You can then prevent the rows from growing out of
>> control by checking the size of the row and pruning it back to 100
>> every N writes, where N is small enough to prevent excessive growth,
>> but large enough to prevent excessive overhead. A value of 50 or so
>> for N worked reasonably well for me for. If you do go down this path,
>> though, keep in mind that rapid writes and deletes to a single column
>> are basically a Cassandra anti-pattern due to performance problems with huge 
>> numbers of tombstones.
>>
>>
>>
>> I would love to see a feature added similar to MongoDB's "capped
>> collections", but I don't believe there is any easy way to retrofit it
>> into Cassandra's sstable approach.
>> http://www.mongodb.org/display/DOCS/Capped+Collections
>>
>>
>>
>> -Tupshin
>>
>> On Mon, Jul 18, 2011 at 8:22 AM, JKnight JKnight 
>> wrote:
>>
>> Dear all,
>>
>>
>>
>> I want to keep only 100 column of a key: when I add a column for a
>> key, if the number column of key is 100, another column (by order) will be 
>> deleted.
>>
>>
>>
>> Does Cassandra have setting for that?
>>
>> --
>> Best regards,
>> JKnight
>>
>>
>
>
>

Re: best example of indexing

[Konstantin Naryshkin]

In the Cassandra CLI tutorial(http://wiki.apache.org/cassandra/CassandraCli), 
there is an example of creating a secondary index.

Konstantin

- Original Message -
From: "CASSANDRA learner" 
To: user@cassandra.apache.org
Sent: Wednesday, July 20, 2011 9:47:28 AM
Subject: best example of indexing

Hi Guys,

Can you please give me the best example of creating index on a column
family. As I am completely new to this, Can you please give me a simple and
good example.

Data Visualization Best Practices

[Selcuk Bozdag]

Hi,

Cassandra provides a flexible scheme-less data storage facility which
is a perfect match for one of our projects. However, regarding the
requirements it is also necessary to list the CFs in a tabular
fashion. I searched on the Internet for some guidelines but could not
get a handy practice for viewing such scheme-less data.

Have you experienced such a case where you required to show CFs (which
obviously may not have the same columns) inside tables? What would be
the most relevant way of showing such data?

Regards,

Selcuk

My "nodetool" in Java

[cbert...@libero.it]

Hi all,
I'd like to build something like "nodetool" to show the status of the ring 
(nodes up-down, info on single node) all via JAVA.
Do you have any tip for this? (I don't want to run the nodetool through java 
and capture the output ...).

I have really no idea on how to do it ... :-)

Re: My "nodetool" in Java

[Jeremy Hanna]

If you look at the bin/nodetool file, it's just a shell script to run 
org.apache.cassandra.tools.NodeCmd.  You could probably call that directly from 
your code.

On Jul 20, 2011, at 3:18 PM, cbert...@libero.it wrote:

> Hi all,
> I'd like to build something like "nodetool" to show the status of the ring 
> (nodes up-down, info on single node) all via JAVA.
> Do you have any tip for this? (I don't want to run the nodetool through java 
> and capture the output ...).
> 
> I have really no idea on how to do it ... :-)

b-tree

[Eldad Yamin]

Hello,
Is there any good way of storing a binary-tree in Cassandra?
I wonder if someone already implement something like that and how
accomplished that without transaction supports (while the tree keep
evolving)?

I'm asking that becouse I want to save geospatial-data, and SimpleGeo did it
using b-tree:
http://www.readwriteweb.com/cloud/2011/02/video-simplegeo-cassandra.php

Thanks!

Re: b-tree

[Jeffrey Kesselman]

Im not sure if I have an answer for you, anyway, but I'm curious

A b-tree and a binary tree are not the same thing.  A binary tree is a
basic fundamental data structure,  A b-tree is an approach to storing and
indexing data on disc for a database.

Which do you mean?

On Wed, Jul 20, 2011 at 4:30 PM, Eldad Yamin  wrote:

> Hello,
> Is there any good way of storing a binary-tree in Cassandra?
> I wonder if someone already implement something like that and how
> accomplished that without transaction supports (while the tree keep
> evolving)?
>
> I'm asking that becouse I want to save geospatial-data, and SimpleGeo did
> it using b-tree:
> http://www.readwriteweb.com/cloud/2011/02/video-simplegeo-cassandra.php
>
> Thanks!
>



-- 
It's always darkest just before you are eaten by a grue.

Re: disable compaction

[Nikolai Kopylov]

Thanx a lot Edward, will follow your advice.

On Wed, Jul 20, 2011 at 7:28 PM, Edward Capriolo wrote:

>
>
> On Wed, Jul 20, 2011 at 11:13 AM, Nikolai Kopylov wrote:
>
>> Hi everyone,
>>
>> finding out recently that cassandra have no upper limit for sstable files
>> to grow, I decided to move to deletion of CF with obsolete data.
>> So that I will not remove columns and there is no need in compaction at
>> all. How can I completely disable the compaction process?
>>
>> Thanx for your attention,
>> Nikolai
>>
>>
> This can be done from the CLI.
> [default@unknown] help update column family;
> update column family Bar;
> update column family Bar with =;
> update column family Bar with = and =...;
>
> - min_compaction_threshold: Avoid minor compactions of less than this
> number of sstable files
> - max_compaction_threshold: Compact no more than this number of sstable
> files at once
>
> update column family xxx with min_compaction_threshold=0 and
> max_compaction_threshold=0;
>
> Rolling over data into new column families is one extreme way to avoid
> compaction :)
>
> Edward
>
>


-- 
C уважением, Николай.

Re: Need help json2sstable

[Tyler Hobbs]

The sstable2json/json2sstable format is detailed here:
http://www.datastax.com/docs/0.7/utilities/sstable2json

On Wed, Jul 20, 2011 at 4:58 AM, Nilabja Banerjee
 wrote:
>
>
>
>
> On 20 July 2011 11:33, Nilabja Banerjee  wrote:
>>
>> Hi All,
>>
>> Here Is my Json structure.
>>
>>
>> {"Fetch_CC" :{
>>                 "cc":{ "":"1000",
>>                      ":"ICICI",
>>                          "":"",
>>                          "city":{
>>                              "name":"banglore"
>>    };
>>    };
>> }
>>
>> If the structure is incorrect, please give me one small structre to use
>> below utility.
>> I am using 0.7.5 version.
>> Now how can I can use Json2SStable utilities? Please provide me the steps.
>> What are the things I have configure?
>>
>> Thank You
>
>



-- 
Tyler Hobbs
Software Engineer, DataStax
Maintainer of the pycassa Cassandra Python client library

Re: b-tree

[aaron morton]

Just throwing out a (half baked) idea, perhaps the Nested Set Model of trees 
would work  http://en.wikipedia.org/wiki/Nested_set_model

* Ever row would represent a set with a left and right encoded into the key
* Members are inserted as columns into *every* set / row they are a member. So 
we are de-normalising and trading space for time. 
* May need to maintain a custom secondary index of the materialised sets. e.g. 
slice a row to get the first column >= the left value you are interested in, 
that is the key for the set. 

I've not thought it through much further than that, a lot would depend on your 
data. The top sets may get very big, . 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jul 2011, at 08:33, Jeffrey Kesselman wrote:

> Im not sure if I have an answer for you, anyway, but I'm curious
> 
> A b-tree and a binary tree are not the same thing.  A binary tree is a basic 
> fundamental data structure,  A b-tree is an approach to storing and indexing 
> data on disc for a database.
> 
> Which do you mean?
> 
> On Wed, Jul 20, 2011 at 4:30 PM, Eldad Yamin  wrote:
> Hello,
> Is there any good way of storing a binary-tree in Cassandra?
> I wonder if someone already implement something like that and how 
> accomplished that without transaction supports (while the tree keep evolving)?
> 
> I'm asking that becouse I want to save geospatial-data, and SimpleGeo did it 
> using b-tree:
> http://www.readwriteweb.com/cloud/2011/02/video-simplegeo-cassandra.php
> 
> Thanks!
> 
> 
> 
> -- 
> It's always darkest just before you are eaten by a grue.

Re: Cassandra & CLOUD . How its related

[Sameer Farooqui]

Are you talking about cloudsandra.com? Check out their website.

Cassandra is a database. Cloud is just a fancy term for remote hosting. The
two aren't really related.

On Wed, Jul 20, 2011 at 3:19 AM, CASSANDRA learner <
cassandralear...@gmail.com> wrote:

> Hi Guys,
>
> When we talk about cassandra, any how we connect it to cloud. I dont
> understand how it is connected to cloud. Whats this Cassandra Cloud.
>

Re: Cassandra training in Bangalore, India

[Sameer Farooqui]

Check out this DataStax documentation:
http://www.datastax.com/docs/0.8/index

DataStax has training in California and the U.S. every once in a while. No
training in India as far as I know.

O'Reilly Has a book on Cassandra called "Cassandra: The Definitive Guide"
and it's kinda outdated, but it's a decent learning guide if you're just
getting started with NoSQL.

Also, there's tons of videos and slides online for getting started. You can
also check out this beginner's guide, even though it's a bit outdated:
https://sites.google.com/a/techlabs.accenture.com/cassandra/

Here are my bookmarks on Cassandra, if you want to browse through them:
http://www.delicious.com/blueplastic/cassandra
http://www.delicious.com/blueplastic/nosql


On Wed, Jul 20, 2011 at 1:54 AM, CASSANDRA learner <
cassandralear...@gmail.com> wrote:

> Is there anywhere the training on cassandra is available in bamgalore,
> India
>

Re: best example of indexing

[Sameer Farooqui]

More info:
http://www.datastax.com/docs/0.8/data_model/secondary_indexes
http://www.datastax.com/docs/0.8/data_model/cfs_as_indexes


On Wed, Jul 20, 2011 at 10:49 AM, Konstantin Naryshkin  wrote:

> In the Cassandra CLI tutorial(
> http://wiki.apache.org/cassandra/CassandraCli), there is an example of
> creating a secondary index.
>
> Konstantin
>
> - Original Message -
> From: "CASSANDRA learner" 
> To: user@cassandra.apache.org
> Sent: Wednesday, July 20, 2011 9:47:28 AM
> Subject: best example of indexing
>
> Hi Guys,
>
> Can you please give me the best example of creating index on a column
> family. As I am completely new to this, Can you please give me a simple and
> good example.
>

Re: Data Visualization Best Practices

[aaron morton]

This project may provide some inspiration 
https://github.com/driftx/chiton

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jul 2011, at 06:36, Selcuk Bozdag wrote:

> Hi,
> 
> Cassandra provides a flexible scheme-less data storage facility which
> is a perfect match for one of our projects. However, regarding the
> requirements it is also necessary to list the CFs in a tabular
> fashion. I searched on the Internet for some guidelines but could not
> get a handy practice for viewing such scheme-less data.
> 
> Have you experienced such a case where you required to show CFs (which
> obviously may not have the same columns) inside tables? What would be
> the most relevant way of showing such data?
> 
> Regards,
> 
> Selcuk

Re: Repair taking a long, long time

[aaron morton]

The first thing to do is understand what the server is doing. 

As Edward said, there are two phases to the repair first the differences are 
calculated and then they are shared between the neighbours. Lets an a third 
step, once the neighbour gets the data it has to rebuild the indexes and bloom 
filter, not huge but lets include it for completeness. 

So...

0. Check for ERRORS in the log.
1. check nodetool compactstats , if the Merkle tree build is going on it will 
say "Validation Compaction". Run it twice and check for progress.
2. check nodetool netstats, this will show which segments of the data are been 
streamed. Run it twice and check for progress. 
3. check nodetool compactstats, if the data has completed streaming and indexes 
are been built it will say "SSTable build"

Once we know what stage of the repair your server is at it's possible to reason 
about what is going on.

If you want to dive deeper look for a log messages on the machine you started 
the repair on from the AnitEntropyService. 

Hope that helps. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jul 2011, at 02:31, David Boxenhorn wrote:

> As I indicated below (but didn't say specifically) another option is to set 
> read repair chance to 1.0 for all your CFs and loop over all your data, since 
> read triggers a read repair. 
> 
> On Wed, Jul 20, 2011 at 4:58 PM, Maxim Potekhin  wrote:
> I can re-load all data that I have in the cluster, from a flat-file cache I 
> have
> on NFS, many times faster than the nodetool repair takes. And that's not
> even accurate because as other noted nodetool repair eats up disk space
> for breakfast and takes more than 24hrs on 200GB data load, at which point
> I have to cancel. That's not acceptable. I simply don't know what to do now.
> 
> 
> 
> On 7/20/2011 8:47 AM, David Boxenhorn wrote:
>> 
>> I have this problem too, and I don't understand why.
>> 
>> I can repair my nodes very quickly by looping though all my data (when you 
>> read your data it does read-repair), but nodetool repair takes forever. I 
>> understand that nodetool repair builds merkle trees, etc. etc., so it's a 
>> different algorithm, but why can't nodetool repair be smart enough to choose 
>> the best algorithm? Also, I don't understand what's special about my data 
>> that makes nodetool repair so much slower than looping through all my data.
>> 
>> 
>> On Wed, Jul 20, 2011 at 12:18 AM, Maxim Potekhin  wrote:
>> Thanks Edward. I'm told by our IT that the switch connecting the nodes is 
>> pretty fast.
>> Seriously, in my house I copy complete DVD images from my bedroom to
>> the living room downstairs via WiFi, and a dozen of GB does not seem like a
>> problem, on dirt cheap hardware (Patriot Box Office).
>> 
>> I also have just _one_ column major family but caveat emptor -- 8 indexes 
>> attached to
>> it (and there will be more). There is one accounting CF which is small, 
>> can't possibly
>> make a difference.
>> 
>> By contrast, compaction (as in nodetool) performs quite well on this 
>> cluster. I start suspecting some
>> sort of malfunction.
>> 
>> Looked at the system log during the "repair", there is some compaction agent 
>> doing
>> work that I'm not sure makes sense (and I didn't call for it). Disk 
>> utilization all of a sudden goes up to 40%
>> per Ganglia, and stays there, this is pretty silly considering the cluster 
>> is IDLE and we have SSDs. No external writes,
>> no reads. There are occasional GC stoppages, but these I can live with.
>> 
>> This repair debacle happens 2nd time in a row. Cr@p. I need to go to 
>> production soon
>> and that doesn't look good at all. If I can't manage a system that simple 
>> (and/or get help
>> on this list) I may have to cut losses i.e. stay with Oracle.
>> 
>> Regards,
>> 
>> Maxim
>> 
>> 
>> 
>> 
>> On 7/19/2011 12:16 PM, Edward Capriolo wrote:
>> 
>> Well most SSD's are pretty fast. There is one more to consider. If Cassandra 
>> determines nodes are out of sync it has to transfer data across the network. 
>> If that is the case you have to look at 'nodetool streams' and determine how 
>> much data is being transferred between nodes. There are some open tickets 
>> where with larger tables repair is streaming more then it needs to. But even 
>> if the transfers are only 10% of your 200GB. Transferring 20 GB is not 
>> trivial.
>> 
>> If you have multiple keyspaces and column families repair one at a time 
>> might make the process more manageable.
>> 
>> 
> 
> 

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

[Aaron Morton]

If you have never run repair also check the section on repair on this page 
http://wiki.apache.org/cassandra/Operations About how frequently it should be 
run.

There is an issue where repair can stream too much data, and this can lead to 
excessive disk use.

My non scientific approach to the never run repair before problem is to repair 
a single CF at a time, starting with the small ones that are less likely to 
have differences as they will stream the smallest amount of data. 

If you really want to conserve disk IO during the repair consider disabling the 
minor compaction by setting the min and max thresholds to 0 via node tool.

hope that helps.


-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 20/07/2011, at 11:46 PM, Yan Chunlu  wrote:

> just found this:
> https://issues.apache.org/jira/browse/CASSANDRA-2156
> 
> but seems only available to 0.8 and people submitted a patch for 0.6, I am 
> using 0.7.4, do I need to dig into the code and make my own patch?
> 
> does add compaction throttle solve the io problem?  thanks!
> 
> On Wed, Jul 20, 2011 at 4:44 PM, Yan Chunlu  wrote:
> at the beginning of using cassandra, I have no idea that I should run "node 
> repair" frequently, so basically, I have 3 nodes with RF=3 and have not run 
> node repair for months, the data size is 20G.
> 
> the problem is when I start running node repair now, it eat up all disk io 
> and the server load became 20+ and increasing, the worst thing is, the entire 
> cluster has slowed down and can not handle request. so I have to stop it 
> immediately because it make my web service unavailable.
> 
> the server has Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] and 8G memory, 
> with Western Digital WD RE3 WD1002FBYS SATA disk.
> 
> I really have no idea what to do now, as currently I have already found some 
> data loss, any suggestions would be appreciated.
> 
> 
> 
> -- 
> 闫春路

Re: PHPCassa get number of rows

[Aaron Morton]

Cassandra does not provide a way to count the number of rows, the best you can 
do is a series of range calls and count them on the client side 
http://thobbs.github.com/phpcassa/tutorial.html

If this is something you need in your app consider creating a custom secondary 
index to store the row keys and counting the columns. NOTE: counting columns 
just reads aol the columns, for a big row it can result in an OOM.

Cheers
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 20/07/2011, at 8:29 AM, Jean-Nicolas Boulay Desjardins 
 wrote:

> Hi,
> 
> How can I get the number of rows with PHPCassa?
> 
> Thanks in advance.

Re: What is the nodeId for?

[Boris Yen]

Not sure if this is the same. I saw exceptions like this:

 INFO 15:33:49,336 Finished reading
/root/commitlog_tmp/CommitLog-1311135088656.log
ERROR 15:33:49,336 Exception encountered during startup.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.AssertionError
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:509)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:332)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:174)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:315)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:80)
Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:505)
... 5 more
Caused by: java.lang.AssertionError
at
org.apache.cassandra.db.SystemTable.getCurrentLocalNodeId(SystemTable.java:383)
at
org.apache.cassandra.utils.NodeId$LocalNodeIdHistory.(NodeId.java:179)
at org.apache.cassandra.utils.NodeId.(NodeId.java:38)
at
org.apache.cassandra.db.context.CounterContext$ContextState.getNodeId(CounterContext.java:722)
at
org.apache.cassandra.db.context.CounterContext.merge(CounterContext.java:333)
at
org.apache.cassandra.db.CounterColumn.reconcile(CounterColumn.java:159)
at
org.apache.cassandra.db.SuperColumn.addColumn(SuperColumn.java:185)
at
org.apache.cassandra.db.SuperColumn.putColumn(SuperColumn.java:204)
at
org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:228)
at
org.apache.cassandra.db.ColumnFamily.addAll(ColumnFamily.java:131)
at
org.apache.cassandra.db.ColumnFamily.resolve(ColumnFamily.java:414)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:218)
at org.apache.cassandra.db.Memtable.put(Memtable.java:148)
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:774)
at org.apache.cassandra.db.Table.apply(Table.java:451)
at
org.apache.cassandra.db.commitlog.CommitLog$2.runMayThrow(CommitLog.java:324)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)

On Wed, Jul 20, 2011 at 11:42 PM, Sylvain Lebresne wrote:

> Possibly, you've hitted this:
> https://issues.apache.org/jira/browse/CASSANDRA-2824
> Should be fixed in next minor release.
>
> In the meantime, you "fix" should be alright.
>
> --
> Sylvain
>
>
> On Wed, Jul 20, 2011 at 3:47 PM, Boris Yen  wrote:
> > Hi Sam,
> > Thanks for the explanation.
> > The NodeIds do appear in the Local row of NodeIdInfo, and after manually
> > deleting two (I got three before I deleted them) of them from
> CurrentLocal
> > row, the cassandra can be restarted now. I was just thinking what could
> be
> > the possible cause for this? and wondering if anyone has any idea about
> > this?
> > Boris
> >
> > On Wed, Jul 20, 2011 at 8:00 PM, Sam Overton  wrote:
> >>
> >> The NodeId is used in counter replication. Counters are stored on each
> >> replica as a set of "shards," where each shard corresponds to the local
> >> count of one of the replicas for that counter, as identified by the
> NodeId.
> >> A NodeId is generated the first time cassandra starts, and might be
> >> renewed during cleanup, or if you explicitly tell cassandra to generate
> a
> >> new one by starting it with -Dcassandra.renew_counter_id
> >> Do either of the NodeIds in the CurrentLocal row also appear in the
> Local
> >> row of the NodeIdInfo CF? the Local row is a history of previous NodeIds
> >> belonging to this node. If one appears in the history and the other does
> >> not, it is probably safe to assume that the old NodeId was not deleted
> from
> >> the CurrentLocal row correctly, and so you could try removing it from
> there
> >> manually.
> >>
> >> Sam
> >> --
> >> Sam Overton
> >> Acunu | http://www.acunu.com | @acunu
> >> On 20 July 2011 12:25, Boris Yen  wrote:
> >>>
> >>> Hi,
> >>> I think we might have screwed our data up. I saw multiple columns
> inside
> >>> record: System.NodeIdInfo.CurrentLocal. It makes our cassandra dead
> forever.
> >>> I was wondering if an

with proof Re: cassandra goes infinite loop and data lost.....

[Yan Chunlu]

this time it is another node, the node goes down during repair, and come
back but never up, I change log level to "DEBUG" and found out it print out
the following message infinitely

DEBUG [main] 2011-07-20 20:58:16,286 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:6@1311207851757243
DEBUG [main] 2011-07-20 20:58:16,319 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:98@1306722716288857
DEBUG [main] 2011-07-20 20:58:16,424 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:95@1311089980134545
DEBUG [main] 2011-07-20 20:58:16,611 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:85@1311154048866767
DEBUG [main] 2011-07-20 20:58:16,754 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:366@1311207176880564
DEBUG [main] 2011-07-20 20:58:16,770 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:80@1310443605930900
DEBUG [main] 2011-07-20 20:58:16,816 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:486@1311173929610402
DEBUG [main] 2011-07-20 20:58:16,870 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:101@1310818289021118
DEBUG [main] 2011-07-20 20:58:17,041 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:677@1311202595772170
DEBUG [main] 2011-07-20 20:58:17,047 SliceQueryFilter.java (line 123)
collecting 0 of 2147483647: 76616c7565:false:374@1311147641237918




On Thu, Jul 14, 2011 at 1:36 PM, Jonathan Ellis  wrote:

> That says "I'm collecting data to answer requests."
>
> I don't see anything here that indicates an infinite loop.
>
> I do see that it's saying "N of 2147483647" which looks like you're
> doing slices with a much larger limit than is advisable (good way to
> OOM the way you already did).
>
> On Wed, Jul 13, 2011 at 8:27 PM, Yan Chunlu  wrote:
> > I gave cassandra 8GB heap size and somehow it run out of memory and
> crashed.
> > after I start it, it just runs in to the following infinite loop, the
> last
> > line:
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> > goes for ever
> > I have 3 nodes and RF=2, so I am losing data. is that means I am screwed
> and
> > can't get it back?
> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> > collecting 20 of 2147483647: q74k:false:14@1308886095008943
> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 10fbu:false:1@1310223075340297
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: apbg:false:13@1305641597957086
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 1 of 2147483647: auje:false:13@1305641597957075
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 2 of 2147483647: ayj8:false:13@1305641597957060
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 3 of 2147483647: b4fz:false:13@1305641597957096
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 1 of 2147483647: 1017f:false:14@1310168680375612
> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> > collecting 2 of 2147483647: 1018e:false:14@1310168759614715
> > DEBUG [main] 2011-07-13 22:19:00,587 SliceQueryFilter.java (line 123)
> > collecting 3 of 2147483647: 101dd:false:14@1310169260225339
> >
> > On Thu, Jul 14, 2011 at 11:27 AM, Yan Chunlu 
> wrote:
> >>
> >> DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> >
> >
> > --
> > 闫春路
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>



-- 
闫春路

Re: node repair eat up all disk io and slow down entire cluster(3 nodes)

[Yan Chunlu]

thank you very much for the help, I will try to adjust minor compaction and
also dealing with single CF at a time.

On Thu, Jul 21, 2011 at 7:56 AM, Aaron Morton wrote:

> If you have never run repair also check the section on repair on this page
> http://wiki.apache.org/cassandra/Operations About how frequently it should
> be run.
>
> There is an issue where repair can stream too much data, and this can lead
> to excessive disk use.
>
> My non scientific approach to the never run repair before problem is to
> repair a single CF at a time, starting with the small ones that are less
> likely to have differences as they will stream the smallest amount of data.
>
> If you really want to conserve disk IO during the repair consider disabling
> the minor compaction by setting the min and max thresholds to 0 via node
> tool.
>
> hope that helps.
>
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/07/2011, at 11:46 PM, Yan Chunlu  wrote:
>
> just found this:
> 
> https://issues.apache.org/jira/browse/CASSANDRA-2156
>
> but seems only available to 0.8 and people submitted a patch for 0.6, I am
> using 0.7.4, do I need to dig into the code and make my own patch?
>
> does add compaction throttle solve the io problem?  thanks!
>
> On Wed, Jul 20, 2011 at 4:44 PM, Yan Chunlu < 
> springri...@gmail.com> wrote:
>
>> at the beginning of using cassandra, I have no idea that I should run
>> "node repair" frequently, so basically, I have 3 nodes with RF=3 and have
>> not run node repair for months, the data size is 20G.
>>
>> the problem is when I start running node repair now, it eat up all disk io
>> and the server load became 20+ and increasing, the worst thing is, the
>> entire cluster has slowed down and can not handle request. so I have to stop
>> it immediately because it make my web service unavailable.
>>
>> the server has Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] and 8G
>> memory, with Western Digital WD RE3 WD1002FBYS SATA disk.
>>
>> I really have no idea what to do now, as currently I have already found
>> some data loss, any suggestions would be appreciated.
>>
>
>
>
> --
> 闫春路
>
>


-- 
闫春路

Re: with proof Re: cassandra goes infinite loop and data lost.....

[Jonathan Ellis]

This is not an infinite loop, you can see the column objects being
iterated over are different.

Like I said last time, "I do see that it's saying "N of 2147483647"
which looks like you're
doing slices with a much larger limit than is advisable."

On Wed, Jul 20, 2011 at 9:00 PM, Yan Chunlu  wrote:
> this time it is another node, the node goes down during repair, and come
> back but never up, I change log level to "DEBUG" and found out it print out
> the following message infinitely
> DEBUG [main] 2011-07-20 20:58:16,286 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:6@1311207851757243
> DEBUG [main] 2011-07-20 20:58:16,319 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:98@1306722716288857
> DEBUG [main] 2011-07-20 20:58:16,424 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:95@1311089980134545
> DEBUG [main] 2011-07-20 20:58:16,611 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:85@1311154048866767
> DEBUG [main] 2011-07-20 20:58:16,754 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:366@1311207176880564
> DEBUG [main] 2011-07-20 20:58:16,770 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:80@1310443605930900
> DEBUG [main] 2011-07-20 20:58:16,816 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:486@1311173929610402
> DEBUG [main] 2011-07-20 20:58:16,870 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:101@1310818289021118
> DEBUG [main] 2011-07-20 20:58:17,041 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:677@1311202595772170
> DEBUG [main] 2011-07-20 20:58:17,047 SliceQueryFilter.java (line 123)
> collecting 0 of 2147483647: 76616c7565:false:374@1311147641237918
>
>
>
> On Thu, Jul 14, 2011 at 1:36 PM, Jonathan Ellis  wrote:
>>
>> That says "I'm collecting data to answer requests."
>>
>> I don't see anything here that indicates an infinite loop.
>>
>> I do see that it's saying "N of 2147483647" which looks like you're
>> doing slices with a much larger limit than is advisable (good way to
>> OOM the way you already did).
>>
>> On Wed, Jul 13, 2011 at 8:27 PM, Yan Chunlu  wrote:
>> > I gave cassandra 8GB heap size and somehow it run out of memory and
>> > crashed.
>> > after I start it, it just runs in to the following infinite loop, the
>> > last
>> > line:
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
>> > goes for ever
>> > I have 3 nodes and RF=2, so I am losing data. is that means I am screwed
>> > and
>> > can't get it back?
>> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
>> > collecting 20 of 2147483647: q74k:false:14@1308886095008943
>> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 10fbu:false:1@1310223075340297
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: apbg:false:13@1305641597957086
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 1 of 2147483647: auje:false:13@1305641597957075
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 2 of 2147483647: ayj8:false:13@1305641597957060
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 3 of 2147483647: b4fz:false:13@1305641597957096
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 1 of 2147483647: 1017f:false:14@1310168680375612
>> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> > collecting 2 of 2147483647: 1018e:false:14@1310168759614715
>> > DEBUG [main] 2011-07-13 22:19:00,587 SliceQueryFilter.java (line 123)
>> > collecting 3 of 2147483647: 101dd:false:14@1310169260225339
>> >
>> > On Thu, Jul 14, 2011 at 11:27 AM, Yan Chunlu 
>> > wrote:
>> >>
>> >> DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> >> collecting 0 of 2147483647: 100zs:false:14@1310168625866434
>> >
>> >
>> > --
>> > 闫春路
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
>
> --
> 闫春路
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: with proof Re: cassandra goes infinite loop and data lost.....

[Yan Chunlu]

sorry for the misunderstanding.  I saw many N of 2147483647 which N=0 and
thought it was not doing anything.

my node was very unbalanced and I was intend to rebalance it by "nodetool
move" after a "node repair", does that cause the slices much large?

Address Status State   LoadOwnsToken


 84944475733633104818662955375549269696
10.28.53.2  Down   Normal  71.41 GB81.09%
 52773518586096316348543097376923124102
10.28.53.3 Up Normal  14.72 GB10.48%
 70597222385644499881390884416714081360
10.28.53.4  Up Normal  13.5 GB 8.43%
84944475733633104818662955375549269696


should I do "nodetool move" according to
http://wiki.apache.org/cassandra/Operations#Load_balancing  before doing
repair?

thank you for your help!



On Thu, Jul 21, 2011 at 10:47 AM, Jonathan Ellis  wrote:

> This is not an infinite loop, you can see the column objects being
> iterated over are different.
>
> Like I said last time, "I do see that it's saying "N of 2147483647"
> which looks like you're
> doing slices with a much larger limit than is advisable."
>
> On Wed, Jul 20, 2011 at 9:00 PM, Yan Chunlu  wrote:
> > this time it is another node, the node goes down during repair, and come
> > back but never up, I change log level to "DEBUG" and found out it print
> out
> > the following message infinitely
> > DEBUG [main] 2011-07-20 20:58:16,286 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:6@1311207851757243
> > DEBUG [main] 2011-07-20 20:58:16,319 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:98@1306722716288857
> > DEBUG [main] 2011-07-20 20:58:16,424 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:95@1311089980134545
> > DEBUG [main] 2011-07-20 20:58:16,611 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:85@1311154048866767
> > DEBUG [main] 2011-07-20 20:58:16,754 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:366@1311207176880564
> > DEBUG [main] 2011-07-20 20:58:16,770 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:80@1310443605930900
> > DEBUG [main] 2011-07-20 20:58:16,816 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:486@1311173929610402
> > DEBUG [main] 2011-07-20 20:58:16,870 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:101@1310818289021118
> > DEBUG [main] 2011-07-20 20:58:17,041 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:677@1311202595772170
> > DEBUG [main] 2011-07-20 20:58:17,047 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:374@1311147641237918
> >
> >
> >
> > On Thu, Jul 14, 2011 at 1:36 PM, Jonathan Ellis 
> wrote:
> >>
> >> That says "I'm collecting data to answer requests."
> >>
> >> I don't see anything here that indicates an infinite loop.
> >>
> >> I do see that it's saying "N of 2147483647" which looks like you're
> >> doing slices with a much larger limit than is advisable (good way to
> >> OOM the way you already did).
> >>
> >> On Wed, Jul 13, 2011 at 8:27 PM, Yan Chunlu 
> wrote:
> >> > I gave cassandra 8GB heap size and somehow it run out of memory and
> >> > crashed.
> >> > after I start it, it just runs in to the following infinite loop, the
> >> > last
> >> > line:
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> >> > goes for ever
> >> > I have 3 nodes and RF=2, so I am losing data. is that means I am
> screwed
> >> > and
> >> > can't get it back?
> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> >> > collecting 20 of 2147483647: q74k:false:14@1308886095008943
> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: 10fbu:false:1@1310223075340297
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: apbg:false:13@1305641597957086
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 1 of 2147483647: auje:false:13@1305641597957075
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 2 of 2147483647: ayj8:false:13@1305641597957060
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 3 of 2147483647: b4fz:false:13@1305641597957096
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 1 of 2147483647: 1017f:false:14@1310168680375612
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 2 of 2147483647: 1018e:false:14@13

Re: with proof Re: cassandra goes infinite loop and data lost.....

[aaron morton]

Personally I would do a repair first if you need to do one, just so you are 
confident everything is where is should be. 

Then do the move as described in the wiki. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jul 2011, at 15:14, Yan Chunlu wrote:

> sorry for the misunderstanding.  I saw many N of 2147483647 which N=0 and 
> thought it was not doing anything.
> 
> my node was very unbalanced and I was intend to rebalance it by "nodetool 
> move" after a "node repair", does that cause the slices much large?
> 
> Address Status State   LoadOwnsToken  
>  
>
> 84944475733633104818662955375549269696  
> 10.28.53.2  Down   Normal  71.41 GB81.09%  
> 52773518586096316348543097376923124102  
> 10.28.53.3 Up Normal  14.72 GB10.48%  
> 70597222385644499881390884416714081360  
> 10.28.53.4  Up Normal  13.5 GB 8.43%   
> 84944475733633104818662955375549269696  
> 
> 
> should I do "nodetool move" according to 
> http://wiki.apache.org/cassandra/Operations#Load_balancing  before doing 
> repair?
> 
> thank you for your help!
> 
> 
> 
> On Thu, Jul 21, 2011 at 10:47 AM, Jonathan Ellis  wrote:
> This is not an infinite loop, you can see the column objects being
> iterated over are different.
> 
> Like I said last time, "I do see that it's saying "N of 2147483647"
> which looks like you're
> doing slices with a much larger limit than is advisable."
> 
> On Wed, Jul 20, 2011 at 9:00 PM, Yan Chunlu  wrote:
> > this time it is another node, the node goes down during repair, and come
> > back but never up, I change log level to "DEBUG" and found out it print out
> > the following message infinitely
> > DEBUG [main] 2011-07-20 20:58:16,286 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:6@1311207851757243
> > DEBUG [main] 2011-07-20 20:58:16,319 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:98@1306722716288857
> > DEBUG [main] 2011-07-20 20:58:16,424 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:95@1311089980134545
> > DEBUG [main] 2011-07-20 20:58:16,611 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:85@1311154048866767
> > DEBUG [main] 2011-07-20 20:58:16,754 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:366@1311207176880564
> > DEBUG [main] 2011-07-20 20:58:16,770 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:80@1310443605930900
> > DEBUG [main] 2011-07-20 20:58:16,816 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:486@1311173929610402
> > DEBUG [main] 2011-07-20 20:58:16,870 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:101@1310818289021118
> > DEBUG [main] 2011-07-20 20:58:17,041 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:677@1311202595772170
> > DEBUG [main] 2011-07-20 20:58:17,047 SliceQueryFilter.java (line 123)
> > collecting 0 of 2147483647: 76616c7565:false:374@1311147641237918
> >
> >
> >
> > On Thu, Jul 14, 2011 at 1:36 PM, Jonathan Ellis  wrote:
> >>
> >> That says "I'm collecting data to answer requests."
> >>
> >> I don't see anything here that indicates an infinite loop.
> >>
> >> I do see that it's saying "N of 2147483647" which looks like you're
> >> doing slices with a much larger limit than is advisable (good way to
> >> OOM the way you already did).
> >>
> >> On Wed, Jul 13, 2011 at 8:27 PM, Yan Chunlu  wrote:
> >> > I gave cassandra 8GB heap size and somehow it run out of memory and
> >> > crashed.
> >> > after I start it, it just runs in to the following infinite loop, the
> >> > last
> >> > line:
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
> >> > goes for ever
> >> > I have 3 nodes and RF=2, so I am losing data. is that means I am screwed
> >> > and
> >> > can't get it back?
> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> >> > collecting 20 of 2147483647: q74k:false:14@1308886095008943
> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: 10fbu:false:1@1310223075340297
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 0 of 2147483647: apbg:false:13@1305641597957086
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 1 of 2147483647: auje:false:13@1305641597957075
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
> >> > collecting 2 of 2147483647: ayj8:false:13@1305641597957060
> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java 

Re: with proof Re: cassandra goes infinite loop and data lost.....

[Yan Chunlu]

thans for the reply.

now the problem is how can I get rid of the ""N of 2147483647 ", it seems
never ends, and the node never goes UP
last time it happens I run "node cleanup", turns out some data loss(not sure
if caused by cleanup).

On Thu, Jul 21, 2011 at 11:37 AM, aaron morton wrote:

> Personally I would do a repair first if you need to do one, just so you are
> confident everything is where is should be.
>
> Then do the move as described in the wiki.
>
> Cheers
>
>  -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 21 Jul 2011, at 15:14, Yan Chunlu wrote:
>
> sorry for the misunderstanding.  I saw many N of 2147483647 which N=0 and
> thought it was not doing anything.
>
> my node was very unbalanced and I was intend to rebalance it by "nodetool
> move" after a "node repair", does that cause the slices much large?
>
> Address Status State   LoadOwnsToken
>
>
>  84944475733633104818662955375549269696
> 10.28.53.2  Down   Normal  71.41 GB81.09%
>  52773518586096316348543097376923124102
> 10.28.53.3 Up Normal  14.72 GB10.48%
>  70597222385644499881390884416714081360
> 10.28.53.4  Up Normal  13.5 GB 8.43%
> 84944475733633104818662955375549269696
>
>
> should I do "nodetool move" according to
> http://wiki.apache.org/cassandra/Operations#Load_balancing  before doing
> repair?
>
> thank you for your help!
>
>
>
> On Thu, Jul 21, 2011 at 10:47 AM, Jonathan Ellis wrote:
>
>> This is not an infinite loop, you can see the column objects being
>> iterated over are different.
>>
>> Like I said last time, "I do see that it's saying "N of 2147483647"
>> which looks like you're
>> doing slices with a much larger limit than is advisable."
>>
>> On Wed, Jul 20, 2011 at 9:00 PM, Yan Chunlu 
>> wrote:
>> > this time it is another node, the node goes down during repair, and come
>> > back but never up, I change log level to "DEBUG" and found out it print
>> out
>> > the following message infinitely
>> > DEBUG [main] 2011-07-20 20:58:16,286 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:6@1311207851757243
>> > DEBUG [main] 2011-07-20 20:58:16,319 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:98@1306722716288857
>> > DEBUG [main] 2011-07-20 20:58:16,424 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:95@1311089980134545
>> > DEBUG [main] 2011-07-20 20:58:16,611 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:85@1311154048866767
>> > DEBUG [main] 2011-07-20 20:58:16,754 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:366@1311207176880564
>> > DEBUG [main] 2011-07-20 20:58:16,770 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:80@1310443605930900
>> > DEBUG [main] 2011-07-20 20:58:16,816 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:486@1311173929610402
>> > DEBUG [main] 2011-07-20 20:58:16,870 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:101@1310818289021118
>> > DEBUG [main] 2011-07-20 20:58:17,041 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:677@1311202595772170
>> > DEBUG [main] 2011-07-20 20:58:17,047 SliceQueryFilter.java (line 123)
>> > collecting 0 of 2147483647: 76616c7565:false:374@1311147641237918
>> >
>> >
>> >
>> > On Thu, Jul 14, 2011 at 1:36 PM, Jonathan Ellis 
>> wrote:
>> >>
>> >> That says "I'm collecting data to answer requests."
>> >>
>> >> I don't see anything here that indicates an infinite loop.
>> >>
>> >> I do see that it's saying "N of 2147483647" which looks like you're
>> >> doing slices with a much larger limit than is advisable (good way to
>> >> OOM the way you already did).
>> >>
>> >> On Wed, Jul 13, 2011 at 8:27 PM, Yan Chunlu 
>> wrote:
>> >> > I gave cassandra 8GB heap size and somehow it run out of memory and
>> >> > crashed.
>> >> > after I start it, it just runs in to the following infinite loop, the
>> >> > last
>> >> > line:
>> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> >> > collecting 0 of 2147483647: 100zs:false:14@1310168625866434
>> >> > goes for ever
>> >> > I have 3 nodes and RF=2, so I am losing data. is that means I am
>> screwed
>> >> > and
>> >> > can't get it back?
>> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
>> >> > collecting 20 of 2147483647: q74k:false:14@1308886095008943
>> >> > DEBUG [main] 2011-07-13 22:19:00,585 SliceQueryFilter.java (line 123)
>> >> > collecting 0 of 2147483647: 10fbu:false:1@1310223075340297
>> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> >> > collecting 0 of 2147483647: apbg:false:13@1305641597957086
>> >> > DEBUG [main] 2011-07-13 22:19:00,586 SliceQueryFilter.java (line 123)
>> >> > coll

Cassandra Storage Sizing

[Todd Burruss]

I put together a blog post on Cassandra Storage Sizing so I don't need to keep 
figuring it out again and again.  Hope everyone finds it useful, and give 
feedback if you find errors.

http://btoddb-cass-storage.blogspot.com/

... enjoy

Re: Cassandra training in Bangalore, India

[samal]

As per my knowledge, there is not such expert training available in India as
of now.
As Sameer said there is enough online material available from where you can
learn.I have been playing with Cassandra since beginning. We can plan for
Meetup/learning session near Mumbai/Pune region.

Re: Cassandra training in Bangalore, India

[Sasha Dolgy]

I am quite certain if you find enough people and pony up the fees a few
people on this list would be willing to make the journey...
On Jul 21, 2011 8:02 AM, "samal"  wrote:
> As per my knowledge, there is not such expert training available in India
as
> of now.
> As Sameer said there is enough online material available from where you
can
> learn.I have been playing with Cassandra since beginning. We can plan for
> Meetup/learning session near Mumbai/Pune region.

reset keys_cached

[魏金仙]

Can any one tell how to reset "keys_cached"?
Thanks.

Re: Need help json2sstable

[Nilabja Banerjee]

Thank you...
but  I have already gone through that.. but still not working... I am
getting .. "*You must supply exactly one sstable
* Can you tell me why I am getting this?


On 21 July 2011 02:41, Tyler Hobbs  wrote:

> The sstable2json/json2sstable format is detailed here:
> http://www.datastax.com/docs/0.7/utilities/sstable2json
>
> On Wed, Jul 20, 2011 at 4:58 AM, Nilabja Banerjee
>  wrote:
> >
> >
> >
> >
> > On 20 July 2011 11:33, Nilabja Banerjee 
> wrote:
> >>
> >> Hi All,
> >>
> >> Here Is my Json structure.
> >>
> >>
> >> {"Fetch_CC" :{
> >> "cc":{ "":"1000",
> >>  ":"ICICI",
> >>  "":"",
> >>  "city":{
> >>  "name":"banglore"
> >>};
> >>};
> >> }
> >>
> >> If the structure is incorrect, please give me one small structre to use
> >> below utility.
> >> I am using 0.7.5 version.
> >> Now how can I can use Json2SStable utilities? Please provide me the
> steps.
> >> What are the things I have configure?
> >>
> >> Thank You
> >
> >
>
>
>
> --
> Tyler Hobbs
> Software Engineer, DataStax
> Maintainer of the pycassa Cassandra Python client library
>