RE: Node down

[Rene Kochen]

A restart of node1 fixed the problem.

The only thing I saw in the log of node1 before the problem was the following:

InetAddress /172.27.70.135 is now dead.
InetAddress /172.27.70.135 is now UP

After this, the nodetool ring command showed node 172.27.70.135 as dead.

You mention a "stored ring view". Can it be that this stored ring view was out 
of sync with the actual (gossip) situation?

Thanks!

Rene

From: aaron morton [mailto:aa...@thelastpickle.com]
Sent: woensdag 1 februari 2012 21:03
To: user@cassandra.apache.org
Subject: Re: Node down

Without knowing too much more information I would try this...

* Restart node each node in turn, watch the logs to see what it says about the 
other.
* If that restart did not fix it, try using the  
Dcassandra.load_ring_state=false JVM option when starting the node. That will 
tell it to ignore it's stored ring view and use what gossip is telling it. Add 
it as a new line at the bottom of cassandra-env.sh.

If it's still failing watch the logs and see what it says when it marks the 
other as been down.

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 1/02/2012, at 11:12 PM, Rene Kochen wrote:


I have a cluster with seven nodes.

If I run the node-tool ring command on all nodes, I see the following:

Node1 says that node2 is down.
Node 2 says that node1 is down.
All other nodes say that everyone is up.

Is this normal behavior?

I see no network related problems. Also no problems between node1 and node2.
I use Cassandra 0.7.10

Thanks,

Rene

Re: Restart cassandra every X days?

[R. Verlangen]

Yes, I already did a repair and cleanup. Currently my ring looks like this:

Address DC  RackStatus State   LoadOwns
   Token
***.89datacenter1 rack1   Up Normal  2.44 GB 50.00%  0
***.135datacenter1 rack1   Up Normal  6.99 GB 50.00%
 85070591730234615865843651857942052864

It's not really a problem, but I'm still wondering why this happens.

2012/2/1 aaron morton 

> Do you mean the load in nodetool ring is not even, despite the tokens been
> evenly distributed ?
>
> I would assume this is not the case given the difference, but it may be
> hints given you have just done an upgrade. Check the system using nodetool
> cfstats to see. They will eventually be delivered and deleted.
>
> More likely you will want to:
> 1) nodetool repair to make sure all data is distributed then
> 2) nodetool cleanup if you have changed the tokens at any point finally
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 31/01/2012, at 11:56 PM, R. Verlangen wrote:
>
> After running 3 days on Cassandra 1.0.7 it seems the problem has been
> solved. One weird thing remains, on our 2 nodes (both 50% of the ring), the
> first's usage is just over 25% of the second.
>
> Anyone got an explanation for that?
>
> 2012/1/29 aaron morton 
>
>> Yes but…
>>
>> For every upgrade read the NEWS.TXT it will go through the upgrade
>> procedure in detail. If you want to feel extra smart scan through the
>> CHANGES.txt to get an idea of whats going on.
>>
>> Cheers
>>
>>   -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 29/01/2012, at 4:14 AM, Maxim Potekhin wrote:
>>
>>  Sorry if this has been covered, I was concentrating solely on 0.8x --
>> can I just d/l 1.0.x and continue using same data on same cluster?
>>
>> Maxim
>>
>>
>> On 1/28/2012 7:53 AM, R. Verlangen wrote:
>>
>> Ok, seems that it's clear what I should do next ;-)
>>
>> 2012/1/28 aaron morton 
>>
>>> There are no blockers to upgrading to 1.0.X.
>>>
>>>  A
>>>  -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>>   On 28/01/2012, at 7:48 AM, R. Verlangen wrote:
>>>
>>> Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x
>>> stable enough to upgrade for, or should we wait for a couple of weeks?
>>>
>>> 2012/1/27 Edward Capriolo 
>>>
 I would not say that issuing restart after x days is a good idea. You
 are mostly developing a superstition. You should find the source of the
 problem. It could be jmx or thrift clients not closing connections. We
 don't restart nodes on a regiment they work fine.


 On Thursday, January 26, 2012, Mike Panchenko  wrote:
 > There are two relevant bugs (that I know of), both resolved in
 somewhat recent versions, which make somewhat regular restarts beneficial
 > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in
 GCInspector, fixed in 0.7.9/0.8.5)
 > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap
 fragmentation due to the way memtables used to be allocated, refactored in
 1.0.0)
 > Restarting daily is probably too frequent for either one of those
 problems. We usually notice degraded performance in our ancient cluster
 after ~2 weeks w/o a restart.
 > As Aaron mentioned, if you have plenty of disk space, there's no
 reason to worry about "cruft" sstables. The size of your active set is what
 matters, and you can determine if that's getting too big by watching for
 iowait (due to reads from the data partition) and/or paging activity of the
 java process. When you hit that problem, the solution is to 1. try to tune
 your caches and 2. add more nodes to spread the load. I'll reiterate -
 looking at raw disk space usage should not be your guide for that.
 > "Forcing" a gc generally works, but should not be relied upon (note
 "suggest" in
 http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()).
 It's great news that 1.0 uses a better mechanism for releasing unused
 sstables.
 > nodetool compact triggers a "major" compaction and is no longer a
 recommended by datastax (details here
 http://www.datastax.com/docs/1.0/operations/tuning#tuning-compactionbottom 
 of the page).
 > Hope this helps.
 > Mike.
 > On Wed, Jan 25, 2012 at 5:14 PM, aaron morton <
 aa...@thelastpickle.com> wrote:
 >
 > That disk usage pattern is to be expected in pre 1.0 versions. Disk
 usage is far less interesting than disk free space, if it's using 60 GB and
 there is 200GB thats ok. If it's using 60Gb and there is 6MB free thats a
 problem.
 > In pre 1.0 the compacted files are deleted on disk by waiting for the
 JVM do decide to GC all remaining references. If there is not enough 

Re: Cluster Cloning

[pprun]

the bulk loader , or

the basic and well-known tools: export & import SStable 



-pprun

On 2012年02月02日 15:16, Hefeng Yuan wrote:

Hi,

We need clone the data between 2 clusters. These 2 clusters have 
different number of nodes.


source: 6 nodes, RF5
destination: 3 nodes, RF3

Cassandra version is 0.8.1.

Is there any suggestion on how to do this?

--
Thanks,
Hefeng

batch mode and flushing

[A J]

Hello, when you set 'commitlog_sync: batch' on all the nodes in a
multi-DC cluster and call writes with CL=ALL, does the operation wait
till the write is flushed to all the disks on all the nodes ?

Thanks.

Re: Schedule for CQL 1.1

[Tamil selvan R.S]

Thank you Eric :)

Regards,
Tamil Selvan

On Wed, Feb 1, 2012 at 10:08 PM, Eric Evans  wrote:

> On Wed, Feb 1, 2012 at 5:50 AM, Tamil selvan R.S 
> wrote:
> >  Where to follow the progress of Cassandra CQL development progress and
> > release schedule?
>
> We don't have a formal roadmap for this I'm afraid.  Your best bet is
> to search the issue tracker for CQL tickets, and follow the list.
>
> At the moment there are 2 branches of development, CQLv2 and CQLv3.
>
> CQLv3 (which will debut in the upcoming 1.1 release) is a significant
> departure in a number of important ways, including support for
> wide-rows which use the compound column approach to create indexed,
> tabular structures from rows.  See
> https://issues.apache.org/jira/browse/CASSANDRA-2474 (if you dare) and
> https://issues.apache.org/jira/browse/CASSANDRA-3761 for more.
>
> CQLv2 will remain the default for now, and CQLv3 will be available as
> a run-time option.  When consensus is that CQLv3 is feature-complete
> and ready for production use, it will become the new default, and
> CQLv2 will be available as a run-time option for legacy clients (this
> isn't expected to happen before 1.2).
>
> Hope this helps.
>
> --
> Eric Evans
> Acunu | http://www.acunu.com | @acunu
>

Recommended configuration for good streaming performance?

[Erik Forsberg]

Hi!

We're experimenting with streaming from Hadoop to Cassandra using 
BulkoutputFormat, on cassandra-1.1 branch.


Are there any specific settings we should tune on the Cassandra servers 
in order to get the best streaming performance?


Our Cassandra hardware are 16 core (including HT cores) with 24GiB of 
RAM. They have two disks each. So far we've configured them with 
commitlog on one disk and sstables on the other, but with streaming not 
using commitlog (correct?) maybe it makes sense to have sstables on both 
disks, doubling available I/O?


Thoughts on number of parallel streaming clients?

Thanks,
\EF

Using sstableloader

[Josh Behrends]

I'm new to administering Cassandra so please be kind!

I've been tasked with upgrading a .6 cluster to 1.0.7.  In doing this I
need a rollback plan in case things go sideways since my window for the
upgrade is fairly small.  So we've decided to stand up a brand new cluster
running 1.0.7 and then stop the old cluster, snapshot the data, and then
move it over to the new cluster.

So I need to know, can I use sstableloader to take the snapshot data from
my .6 cluster and stream it into my new 1.0.7 cluster?  I should also note
that the new cluster MAY NOT have the same number of nodes either.

Josh

Re: Node down

[aaron morton]

> You mention a “stored ring view”. Can it be that this stored ring view was 
> out of sync with the actual (gossip) situation?
After checking the code, not as much as I thought it did :)
Stored ring state is just the map from ip address to token, I thought it has a 
little more in there.

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 2/02/2012, at 9:44 PM, Rene Kochen wrote:

> A restart of node1 fixed the problem.
>  
> The only thing I saw in the log of node1 before the problem was the following:
>  
> InetAddress /172.27.70.135 is now dead.
> InetAddress /172.27.70.135 is now UP
>  
> After this, the nodetool ring command showed node 172.27.70.135 as dead.
>  
> You mention a “stored ring view”. Can it be that this stored ring view was 
> out of sync with the actual (gossip) situation?
>  
> Thanks!
>  
> Rene
>  
> From: aaron morton [mailto:aa...@thelastpickle.com] 
> Sent: woensdag 1 februari 2012 21:03
> To: user@cassandra.apache.org
> Subject: Re: Node down
>  
> Without knowing too much more information I would try this…
>  
> * Restart node each node in turn, watch the logs to see what it says about 
> the other. 
> * If that restart did not fix it, try using the  
> Dcassandra.load_ring_state=false JVM option when starting the node. That will 
> tell it to ignore it's stored ring view and use what gossip is telling it. 
> Add it as a new line at the bottom of cassandra-env.sh. 
>  
> If it's still failing watch the logs and see what it says when it marks the 
> other as been down. 
>  
> Cheers
>  
>  
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 1/02/2012, at 11:12 PM, Rene Kochen wrote:
> 
> 
> I have a cluster with seven nodes.
>  
> If I run the node-tool ring command on all nodes, I see the following:
>  
> Node1 says that node2 is down.
> Node 2 says that node1 is down.
> All other nodes say that everyone is up.
>  
> Is this normal behavior?
>  
> I see no network related problems. Also no problems between node1 and node2.
> I use Cassandra 0.7.10
>  
> Thanks,
>  
> Rene

Re: Restart cassandra every X days?

[aaron morton]

Speaking technically, that ain't right.

I would:
* Check if node .135 is holding a lot of hints. 
* Take a look on disk and see what is there.
* Go through a repair and compact on each node.

Cheers

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 2/02/2012, at 9:55 PM, R. Verlangen wrote:

> Yes, I already did a repair and cleanup. Currently my ring looks like this:
> 
> Address DC  RackStatus State   LoadOwns   
>  Token
> ***.89datacenter1 rack1   Up Normal  2.44 GB 50.00%  0
> ***.135datacenter1 rack1   Up Normal  6.99 GB 50.00%  
> 85070591730234615865843651857942052864
> 
> It's not really a problem, but I'm still wondering why this happens.
> 
> 2012/2/1 aaron morton 
> Do you mean the load in nodetool ring is not even, despite the tokens been 
> evenly distributed ? 
> 
> I would assume this is not the case given the difference, but it may be hints 
> given you have just done an upgrade. Check the system using nodetool cfstats 
> to see. They will eventually be delivered and deleted. 
> 
> More likely you will want to:
> 1) nodetool repair to make sure all data is distributed then
> 2) nodetool cleanup if you have changed the tokens at any point finally
> 
> Cheers
> 
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 31/01/2012, at 11:56 PM, R. Verlangen wrote:
> 
>> After running 3 days on Cassandra 1.0.7 it seems the problem has been 
>> solved. One weird thing remains, on our 2 nodes (both 50% of the ring), the 
>> first's usage is just over 25% of the second. 
>> 
>> Anyone got an explanation for that?
>> 
>> 2012/1/29 aaron morton 
>> Yes but…
>> 
>> For every upgrade read the NEWS.TXT it will go through the upgrade procedure 
>> in detail. If you want to feel extra smart scan through the CHANGES.txt to 
>> get an idea of whats going on. 
>> 
>> Cheers
>> 
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 29/01/2012, at 4:14 AM, Maxim Potekhin wrote:
>> 
>>> Sorry if this has been covered, I was concentrating solely on 0.8x --
>>> can I just d/l 1.0.x and continue using same data on same cluster?
>>> 
>>> Maxim
>>> 
>>> 
>>> On 1/28/2012 7:53 AM, R. Verlangen wrote:
 
 Ok, seems that it's clear what I should do next ;-)
 
 2012/1/28 aaron morton 
 There are no blockers to upgrading to 1.0.X.
 
 A 
 -
 Aaron Morton
 Freelance Developer
 @aaronmorton
 http://www.thelastpickle.com
 
 On 28/01/2012, at 7:48 AM, R. Verlangen wrote:
 
> Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x 
> stable enough to upgrade for, or should we wait for a couple of weeks?
> 
> 2012/1/27 Edward Capriolo 
> I would not say that issuing restart after x days is a good idea. You are 
> mostly developing a superstition. You should find the source of the 
> problem. It could be jmx or thrift clients not closing connections. We 
> don't restart nodes on a regiment they work fine.
> 
> 
> On Thursday, January 26, 2012, Mike Panchenko  wrote:
> > There are two relevant bugs (that I know of), both resolved in somewhat 
> > recent versions, which make somewhat regular restarts beneficial
> > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak in 
> > GCInspector, fixed in 0.7.9/0.8.5)
> > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap 
> > fragmentation due to the way memtables used to be allocated, refactored 
> > in 1.0.0)
> > Restarting daily is probably too frequent for either one of those 
> > problems. We usually notice degraded performance in our ancient cluster 
> > after ~2 weeks w/o a restart.
> > As Aaron mentioned, if you have plenty of disk space, there's no reason 
> > to worry about "cruft" sstables. The size of your active set is what 
> > matters, and you can determine if that's getting too big by watching 
> > for iowait (due to reads from the data partition) and/or paging 
> > activity of the java process. When you hit that problem, the solution 
> > is to 1. try to tune your caches and 2. add more nodes to spread the 
> > load. I'll reiterate - looking at raw disk space usage should not be 
> > your guide for that.
> > "Forcing" a gc generally works, but should not be relied upon (note 
> > "suggest" in 
> > http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()). 
> > It's great news that 1.0 uses a better mechanism for releasing unused 
> > sstables.
> > nodetool compact triggers a "major" compaction and is no longer a 
> > recommended by datastax (details here 
> > http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction 
> > bottom of the page).
> >

Re: batch mode and flushing

[aaron morton]

Yes. 
Be aware that the commit log will block and not flush for 
commitlog_sync_batch_window_in_ms 

Cheers
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/02/2012, at 5:44 AM, A J wrote:

> Hello, when you set 'commitlog_sync: batch' on all the nodes in a
> multi-DC cluster and call writes with CL=ALL, does the operation wait
> till the write is flushed to all the disks on all the nodes ?
> 
> Thanks.

Re: Using sstableloader

[aaron morton]

It will make your life *a lot*  easier by doing a 1 to 1 migration from the 0.6 
cluster to the 1.X one. If you want to add nodes do it once you have 1.X happy 
and stable, if you need to reduce nodes threaten to hold your breath until you 
pass out.  

You can then simply:
* drain and shapshot the 0.6 cluster. 
* rsync data to the  1.X cluster and copy the configs (with necessary IP 
changes etc). 
* turn on the 1.X cluster
* scrub

Your roll back is to switch back the 0.6 at any time. 

Last time I did a 0.6 upgrade I did partial migrations to the live system so I 
could then do a final delta migration when the site was down. This was done to 
reduce the down time when we turned the 0.6 off. It also meant we could test 
the app against the 1.X cluster with stale production data *before* the big 
switch over. 

As always read the NEWS.txt for instructions, you will want to go all the way 
back to the 0.7 instructions. 

Cheers
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/02/2012, at 7:24 AM, Josh Behrends wrote:

> I'm new to administering Cassandra so please be kind!
> 
> I've been tasked with upgrading a .6 cluster to 1.0.7.  In doing this I need 
> a rollback plan in case things go sideways since my window for the upgrade is 
> fairly small.  So we've decided to stand up a brand new cluster running 1.0.7 
> and then stop the old cluster, snapshot the data, and then move it over to 
> the new cluster.  
> 
> So I need to know, can I use sstableloader to take the snapshot data from my 
> .6 cluster and stream it into my new 1.0.7 cluster?  I should also note that 
> the new cluster MAY NOT have the same number of nodes either.
> 
> Josh

data model with composite columns

[Deno Vichas]

hey all!

i have a CF that has ~10 columns in it and now i'm finding the need to 
use composite column names.  can you, or should mix and match composite 
and non-composite column names in the same CF?  if you can/should how 
does sorting work with a single comparator?



thanks,
deno

Re: read-repair?

[Guy Incognito]

sorry to be dense, but which is it?  do i get the old version or the new 
version?  or is it indeterminate?


On 02/02/2012 01:42, Peter Schuller wrote:

i have RF=3, my row/column lives on 3 nodes right?  if (for some reason, eg
a timed-out write at quorum) node 1 has a 'new' version of the row/column
(eg clock = 10), but node 2 and 3 have 'old' versions (clock = 5), when i
try to read my row/column at quorum, what do i get back?

You either get back the new version or the old version, depending on
whether node 1 was participated in the read. In your scenario, the
prevoius write at quorum failed (since it only made it to one node),
so this is not a violation of the contract.

Once node 2 and/or 3 return their response, read repair (if it is
active) will cause re-read and re-conciliation followed by a row
mutation being send to the nodes to correct the column.


do i get the clock 5 version because that is what the quorum agrees on, and

No; a quorum of node is waited for, and the newest column wins. This
accomplish the reads-see-write invariant.

Re: read-repair?

[Peter Schuller]

> sorry to be dense, but which is it?  do i get the old version or the new
> version?  or is it indeterminate?

Indeterminate, depending on which nodes happen to be participating in
the read. Eventually you should get the new version, unless the node
that took the new version permanently crashed with data loss prior to
the data making it elsewhere.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: data model with composite columns

[aaron morton]

Short answer is no. The slightly longer answer is nope.

All column names in a CF are compared using the same comparator. You will need 
to create a new CF. 

Cheers.
 
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/02/2012, at 10:25 AM, Deno Vichas wrote:

> hey all!
> 
> i have a CF that has ~10 columns in it and now i'm finding the need to use 
> composite column names.  can you, or should mix and match composite and 
> non-composite column names in the same CF?  if you can/should how does 
> sorting work with a single comparator?
> 
> 
> thanks,
> deno
> 
> 

Re: data model with composite columns

[Deno Vichas]

this is what i thought. thanks for clarifying.


On 2/2/2012 10:44 PM, aaron morton wrote:

Short answer is no. The slightly longer answer is nope.

All column names in a CF are compared using the same comparator. You 
will need to create a new CF.


Cheers.

-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/02/2012, at 10:25 AM, Deno Vichas wrote:


hey all!

i have a CF that has ~10 columns in it and now i'm finding the need 
to use composite column names.  can you, or should mix and match 
composite and non-composite column names in the same CF?  if you 
can/should how does sorting work with a single comparator?



thanks,
deno

Re: Restart cassandra every X days?

[R. Verlangen]

Well, it seems it's balancing itself, 24 hours later the ring looks like
this:

***.89datacenter1 rack1   Up Normal  7.36 GB 50.00%  0
***.135datacenter1 rack1   Up Normal  8.84 GB 50.00%
 85070591730234615865843651857942052864

Looks pretty normal, right?

2012/2/2 aaron morton 

> Speaking technically, that ain't right.
>
> I would:
> * Check if node .135 is holding a lot of hints.
> * Take a look on disk and see what is there.
> * Go through a repair and compact on each node.
>
>
> Cheers
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 2/02/2012, at 9:55 PM, R. Verlangen wrote:
>
> Yes, I already did a repair and cleanup. Currently my ring looks like this:
>
> Address DC  RackStatus State   Load
>  OwnsToken
> ***.89datacenter1 rack1   Up Normal  2.44 GB 50.00%  0
> ***.135datacenter1 rack1   Up Normal  6.99 GB 50.00%
>  85070591730234615865843651857942052864
>
> It's not really a problem, but I'm still wondering why this happens.
>
> 2012/2/1 aaron morton 
>
>> Do you mean the load in nodetool ring is not even, despite the tokens
>> been evenly distributed ?
>>
>> I would assume this is not the case given the difference, but it may be
>> hints given you have just done an upgrade. Check the system using nodetool
>> cfstats to see. They will eventually be delivered and deleted.
>>
>> More likely you will want to:
>> 1) nodetool repair to make sure all data is distributed then
>> 2) nodetool cleanup if you have changed the tokens at any point finally
>>
>> Cheers
>>
>>   -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 31/01/2012, at 11:56 PM, R. Verlangen wrote:
>>
>> After running 3 days on Cassandra 1.0.7 it seems the problem has been
>> solved. One weird thing remains, on our 2 nodes (both 50% of the ring), the
>> first's usage is just over 25% of the second.
>>
>> Anyone got an explanation for that?
>>
>> 2012/1/29 aaron morton 
>>
>>> Yes but…
>>>
>>> For every upgrade read the NEWS.TXT it will go through the upgrade
>>> procedure in detail. If you want to feel extra smart scan through the
>>> CHANGES.txt to get an idea of whats going on.
>>>
>>> Cheers
>>>
>>>   -
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 29/01/2012, at 4:14 AM, Maxim Potekhin wrote:
>>>
>>>  Sorry if this has been covered, I was concentrating solely on 0.8x --
>>> can I just d/l 1.0.x and continue using same data on same cluster?
>>>
>>> Maxim
>>>
>>>
>>> On 1/28/2012 7:53 AM, R. Verlangen wrote:
>>>
>>> Ok, seems that it's clear what I should do next ;-)
>>>
>>> 2012/1/28 aaron morton 
>>>
 There are no blockers to upgrading to 1.0.X.

  A
  -
 Aaron Morton
 Freelance Developer
 @aaronmorton
 http://www.thelastpickle.com

   On 28/01/2012, at 7:48 AM, R. Verlangen wrote:

 Ok. Seems that an upgrade might fix these problems. Is Cassandra 1.x.x
 stable enough to upgrade for, or should we wait for a couple of weeks?

 2012/1/27 Edward Capriolo 

> I would not say that issuing restart after x days is a good idea. You
> are mostly developing a superstition. You should find the source of the
> problem. It could be jmx or thrift clients not closing connections. We
> don't restart nodes on a regiment they work fine.
>
>
> On Thursday, January 26, 2012, Mike Panchenko  wrote:
> > There are two relevant bugs (that I know of), both resolved in
> somewhat recent versions, which make somewhat regular restarts beneficial
> > https://issues.apache.org/jira/browse/CASSANDRA-2868 (memory leak
> in GCInspector, fixed in 0.7.9/0.8.5)
> > https://issues.apache.org/jira/browse/CASSANDRA-2252 (heap
> fragmentation due to the way memtables used to be allocated, refactored in
> 1.0.0)
> > Restarting daily is probably too frequent for either one of those
> problems. We usually notice degraded performance in our ancient cluster
> after ~2 weeks w/o a restart.
> > As Aaron mentioned, if you have plenty of disk space, there's no
> reason to worry about "cruft" sstables. The size of your active set is 
> what
> matters, and you can determine if that's getting too big by watching for
> iowait (due to reads from the data partition) and/or paging activity of 
> the
> java process. When you hit that problem, the solution is to 1. try to tune
> your caches and 2. add more nodes to spread the load. I'll reiterate -
> looking at raw disk space usage should not be your guide for that.
> > "Forcing" a gc generally works, but should not be relied upon (note
> "suggest" in
> http://docs.oracle.com/javase/6/docs/api/java/lang/System.html#gc()).
> It's great news 

Re: Can you query Cassandra while it's doing major compaction

[R. Verlangen]

It will have a performance penalty, so it would be better to spread the
compactions over a period of time. But Cassandra will still take care of
any reads/writes (within the given timeout).

2012/2/3 myreasoner 

> If every node in the cluster is running major compaction, would it be able
> to
> answer any read request?  And is it wise to write anything to a cluster
> while it's doing major compaction?
>
>
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Can-you-query-Cassandra-while-it-s-doing-major-compaction-tp7249985p7249985.html
> Sent from the cassandra-u...@incubator.apache.org mailing list archive at
> Nabble.com.
>

Re: Can you query Cassandra while it's doing major compaction

[Peter Schuller]

> If every node in the cluster is running major compaction, would it be able to
> answer any read request?  And is it wise to write anything to a cluster
> while it's doing major compaction?

Compaction is something that is supposed to be continuously running in
the background. As noted, it will have a performance impact in that it
(1) generates I/O, (2) leads to cache eviction, and (if you're CPU
bound rather than disk bound) (3) adds CPU load.

But there is no intention that clients should have to care about
compaction; it's to be viewed as a background operation continuously
happening. A good rule of thumb is that an individual node should be
able to handle your traffic when doing compaction; you don't want to
be in the position where you're just barely dealing with the traffic,
and a node doing compaction not being able to handle it.

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)