Re: Ran out of space during cleanup.. HELP

[Oleg Anastasyev]

Mark  gmail.com> writes:

> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>  at 
> > 
> On 12/7/10 8:44 PM, Mark wrote:
> > 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2 
> > are now at 100% disk space. What should I do?
> 
> 


Is there files with -tmp n their names ? 
Try to remove them to free up disk space.

Re: fauna/cassandra gem does not work with Cassandra 0.7

[Joshua Partogi]

Hi Ryan,

Thanks for the swift response. I've tested your latest commit and it fixed
the problem.

Kind regards,
Joshua

On Wed, Dec 8, 2010 at 5:23 AM, Ryan King  wrote:

> Please file this on github issues:
> https://github.com/fauna/cassandra/issues. And I'll get to it soon.
>
> -ryan
>
> On Tue, Dec 7, 2010 at 2:21 AM, Joshua Partogi 
> wrote:
> > Hi,
> >
> > I pull out fauna/cassandra gem 0.10.0 from github.
> >
> > I then tried to get a value from cassandra as such.
> >
> > irb(main):002:0> require 'cassandra/0.7'
> > => true
> > irb(main):003:0> client = Cassandra.new('Keyspace1', '127.0.0.1:9160')
> > => # > @servers=["127.0.0.1:9160"]>
> > irb(main):004:0> client.insert(:Standard1, "5", {'screen_name' =>
> > "buttonscat"})
> > => nil
> > irb(main):006:0> client.get(:Standard1, "5", 'screen_name')
> > NoMethodError: undefined method `multiget' for
> > #
> > from
> >
> /home/jpartogi/.rvm/gems/ruby-1.8.7-p302/gems/cassandra-0.10.0/lib/cassandra/0.7/protocol.rb:53:in
> > `_multiget'
> > from
> >
> /home/jpartogi/.rvm/gems/ruby-1.8.7-p302/gems/cassandra-0.10.0/lib/cassandra/cassandra.rb:198:in
> > `multi_get'
> > from
> >
> /home/jpartogi/.rvm/gems/ruby-1.8.7-p302/gems/cassandra-0.10.0/lib/cassandra/cassandra.rb:189:in
> > `get'
> > from (irb):6
> > from :0
> > irb(main):007:0>
> >
> >
> > It turns out that the gem is still relying on thrift < 0.2.5 as can be
> seen
> > on cassandra.rb.
> >
> > My question is, will the gem be updated to use thrift client 0.5?
> >
> > Sorry if this question does not fall in to the right place.
> >
> > Kind regards,
> > Joshua
> >
> > --
> > http://twitter.com/jpartogi
> >
>



-- 
http://twitter.com/jpartogi

How to Tell if Decommission has Completed

[Jake Maizel]

Hello,

Is there a definitive way to tell if a Decommission operation has
completed, such as a log message similar to what happens with a Drain
command?

Thanks.

-- 
Jake Maizel
Network Operations
Soundcloud

Mail & GTalk: j...@soundcloud.com
Skype: jakecloud

Rosenthaler strasse 13, 101 19, Berlin, DE

Re: Ran out of space during cleanup.. HELP

[Jake Maizel]

Also, look for any snapshots that can be cleared with nodetool
clearsnapshot or just run the command to remove any that exist.

On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev  wrote:
>
> Mark  gmail.com> writes:
>
>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush
>>      at
>> >
>> On 12/7/10 8:44 PM, Mark wrote:
>> > 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
>> > are now at 100% disk space. What should I do?
>>
>>
>
>
> Is there files with -tmp n their names ?
> Try to remove them to free up disk space.
>
>
>



-- 
Jake Maizel
Network Operations
Soundcloud

Mail & GTalk: j...@soundcloud.com
Skype: jakecloud

Rosenthaler strasse 13, 101 19, Berlin, DE

Re: GC Exceptions and cluster nodes are dying

[Amin Sakka, Novapost]

Thanks for your answer Aaron,

I'm now on the RC1, I have no longer the ActiveCount error, however my nodes
still dying under bulk insertion.

I have modified my nodes configuration (all of them has now 2GB Heap size).
The nodes still under heavy pressure and they dies after a random timeout
(sometimes after 10 minutes of insertion
and sometimes after 50 minutes).
I want to point that I'm inserting rows in 4 different columns families at
the same time and that the rows size is too little (few KiloBytes).
I've attached here my cassandra.yaml configuration file.
Can you help me please to solve this issue?

Thanks!

Here is some of my log output:

DEBUG [MutationStage:27] 2010-12-08 15:31:19,214 RowMutationVerbHandler.java
(line 78) RowMutation(keyspace='SAE',
key='6163636f756e743936353a726566393230',
modifications=[ColumnFamily(Document
[6465736372697074696f6e:false:4...@1291818633293000
,646f63756d656e744964:false:3...@1291818633293000
,7265666572656e6365:false:1...@1291818633293000,])]) applied.  Sending
response to 714546@/10.0.100.94
DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 RowMutationVerbHandler.java
(line 54) Applying RowMutation(keyspace='SAE',
key='6163636f756e743938373a726566313835',
modifications=[ColumnFamily(Document
[6465736372697074696f6e:false:4...@1291818633524000
,646f63756d656e744964:false:3...@1291818633524000
,7265666572656e6365:false:1...@1291818633524000,])])
DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 Table.java (line 378)
applying mutation of row 6163636f756e743938373a726566313835
DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 RowMutationVerbHandler.java
(line 78) RowMutation(keyspace='SAE',
key='6163636f756e743938373a726566313835',
modifications=[ColumnFamily(Document
[6465736372697074696f6e:false:4...@1291818633524000
,646f63756d656e744964:false:3...@1291818633524000
,7265666572656e6365:false:1...@1291818633524000,])]) applied.  Sending
response to 714547@/10.0.100.94
*DEBUG [ScheduledTasks:1] 2010-12-08 15:31:19,373 GCInspector.java (line
135) GC for ParNew: 13 ms, 34456296 reclaimed leaving 1465917704 used; max
is 2256404480*





2010/12/1 Aaron Morton 

> Running nodes with different JVM heap sizes would not be recommended
> practice, for many reasons. Nor would I recommend running them with all the
> memory the machine has, it will just lead to the OS swapping the JVM out to
> disk and considerable slow things down.
>
> I would suggest a heap size of 1.5 or 2.0 GB for each node, and have a read
> of the JVM Heap Size section here
> http://wiki.apache.org/cassandra/MemtableThresholds . AFAIK the logs are
> showing your cluster was under heavy GC pressure.
>
> Finally, the ActiveCount error message was a known issue in beta 2. Treat
> yourself and try RC1 :)
> http://www.mail-archive.com/user@cassandra.apache.org/msg06298.html
>
> Aaron
>
>
>
> On 02 Dec, 2010,at 12:33 AM, asakka  wrote:
>
>
> Hello,
>
> I'm making some tests on a data model with 3 CF and 1 SCF, I want to start
> by inserting 1 million rows (my target is to have 1billion rows) .
> I have three nodes cluster (I'm using the same machines with 3GB of RAM
> each , intel core2 duo 1,6GHZ), RF = 2, CL = 1, HEAPSIZE of the seed = 3GO
> (it was 1.5GO, I've doubled it to avoid the heap size exception I had) ,
> the
> other two nodes are 1.5GO.
>
> I am using cassandra (V0.7.0-beta2) and Hector (V0.7.0.18) . I'm making
> insertion in batch mode using hector Mutator.
> My disk_access_mode is standard.
> I reduced also my memtable_throughput_in_mb to 64, but the problem persists
> and I have the following exception :
> I want to know if it is a configuration or hardware problem ?
>
> INFO [Timer-0] 2010-12-01 10:34:42,124 Gossiper.java (line 196) InetAddress
> /10.0.100.215 is now dead.
> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,188 Gossiper.java (line 594) Node
> /10.0.100.215 has restarted, now UP again
> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,189 StorageService.java (line
> 643) Node /10.0.100.215 state jump to normal
> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,189 StorageService.java (line
> 650) Will not change my token ownership to /10.0.100.215
> INFO [HINTED-HANDOFF-POOL:1] 2010-12-01 10:34:44,189
> HintedHandOffManager.java (line 196) Started hinted handoff for endpoint
> /10.0.100.215
> INFO [HINTED-HANDOFF-POOL:1] 2010-12-01 10:34:44,189
> HintedHandOffManager.java (line 252) Finished hinted handoff of 0 rows to
> endpoint /10.0.100.215
> INFO [GC inspection] 2010-12-01 10:40:29,141 GCInspector.java (line 129) GC
> for ParNew: 750 ms, 14693208 reclaimed leaving 2140055192 used; max is
> 3355312128
> INFO [GC inspection] 2010-12-01 10:40:30,280 GCInspector.java (line 129) GC
> for ParNew: 445 ms, 17042288 reclaimed leaving 2178211008 used; max is
> 3355312128
> INFO [WRITE-/10.0.100.214] 2010-12-01 10:40:31,552
> OutboundTcpConnection.java (line 115) error writing to /10.0.100.214
> INFO [GC inspection] 2010-12-01 10:40:32,280 GCInspector.java (line 129) GC
> for ParNew: 211 ms, 25550568 reclaimed l

Re: GC Exceptions and cluster nodes are dying

[Jonathan Ellis]

Cut your memtable thresholds (throughput and ops) in half.  See "describe
keyspaces" and "update keyspace" in the cli.

On Wed, Dec 8, 2010 at 9:18 AM, Amin Sakka, Novapost  wrote:

> Thanks for your answer Aaron,
>
> I'm now on the RC1, I have no longer the ActiveCount error, however my
> nodes still dying under bulk insertion.
>
> I have modified my nodes configuration (all of them has now 2GB Heap size).
> The nodes still under heavy pressure and they dies after a random timeout
> (sometimes after 10 minutes of insertion
> and sometimes after 50 minutes).
> I want to point that I'm inserting rows in 4 different columns families at
> the same time and that the rows size is too little (few KiloBytes).
> I've attached here my cassandra.yaml configuration file.
> Can you help me please to solve this issue?
>
> Thanks!
>
> Here is some of my log output:
>
> DEBUG [MutationStage:27] 2010-12-08 15:31:19,214
> RowMutationVerbHandler.java (line 78) RowMutation(keyspace='SAE',
> key='6163636f756e743936353a726566393230',
> modifications=[ColumnFamily(Document
> [6465736372697074696f6e:false:4...@1291818633293000
> ,646f63756d656e744964:false:3...@1291818633293000
> ,7265666572656e6365:false:1...@1291818633293000,])]) applied.  Sending
> response to 714546@/10.0.100.94
> DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 RowMutationVerbHandler.java
> (line 54) Applying RowMutation(keyspace='SAE',
> key='6163636f756e743938373a726566313835',
> modifications=[ColumnFamily(Document
> [6465736372697074696f6e:false:4...@1291818633524000
> ,646f63756d656e744964:false:3...@1291818633524000
> ,7265666572656e6365:false:1...@1291818633524000,])])
> DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 Table.java (line 378)
> applying mutation of row 6163636f756e743938373a726566313835
> DEBUG [MutationStage:7] 2010-12-08 15:31:19,214 RowMutationVerbHandler.java
> (line 78) RowMutation(keyspace='SAE',
> key='6163636f756e743938373a726566313835',
> modifications=[ColumnFamily(Document
> [6465736372697074696f6e:false:4...@1291818633524000
> ,646f63756d656e744964:false:3...@1291818633524000
> ,7265666572656e6365:false:1...@1291818633524000,])]) applied.  Sending
> response to 714547@/10.0.100.94
> *DEBUG [ScheduledTasks:1] 2010-12-08 15:31:19,373 GCInspector.java (line
> 135) GC for ParNew: 13 ms, 34456296 reclaimed leaving 1465917704 used; max
> is 2256404480*
>
>
>
>
>
> 2010/12/1 Aaron Morton 
>
> Running nodes with different JVM heap sizes would not be recommended
>> practice, for many reasons. Nor would I recommend running them with all the
>> memory the machine has, it will just lead to the OS swapping the JVM out to
>> disk and considerable slow things down.
>>
>> I would suggest a heap size of 1.5 or 2.0 GB for each node, and have a
>> read of the JVM Heap Size section here
>> http://wiki.apache.org/cassandra/MemtableThresholds . AFAIK the logs are
>> showing your cluster was under heavy GC pressure.
>>
>> Finally, the ActiveCount error message was a known issue in beta 2. Treat
>> yourself and try RC1 :)
>> http://www.mail-archive.com/user@cassandra.apache.org/msg06298.html
>>
>> Aaron
>>
>>
>>
>> On 02 Dec, 2010,at 12:33 AM, asakka  wrote:
>>
>>
>> Hello,
>>
>> I'm making some tests on a data model with 3 CF and 1 SCF, I want to start
>> by inserting 1 million rows (my target is to have 1billion rows) .
>> I have three nodes cluster (I'm using the same machines with 3GB of RAM
>> each , intel core2 duo 1,6GHZ), RF = 2, CL = 1, HEAPSIZE of the seed = 3GO
>> (it was 1.5GO, I've doubled it to avoid the heap size exception I had) ,
>> the
>> other two nodes are 1.5GO.
>>
>> I am using cassandra (V0.7.0-beta2) and Hector (V0.7.0.18) . I'm making
>> insertion in batch mode using hector Mutator.
>> My disk_access_mode is standard.
>> I reduced also my memtable_throughput_in_mb to 64, but the problem
>> persists
>> and I have the following exception :
>> I want to know if it is a configuration or hardware problem ?
>>
>> INFO [Timer-0] 2010-12-01 10:34:42,124 Gossiper.java (line 196)
>> InetAddress
>> /10.0.100.215 is now dead.
>> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,188 Gossiper.java (line 594)
>> Node
>> /10.0.100.215 has restarted, now UP again
>> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,189 StorageService.java (line
>> 643) Node /10.0.100.215 state jump to normal
>> INFO [GOSSIP_STAGE:1] 2010-12-01 10:34:44,189 StorageService.java (line
>> 650) Will not change my token ownership to /10.0.100.215
>> INFO [HINTED-HANDOFF-POOL:1] 2010-12-01 10:34:44,189
>> HintedHandOffManager.java (line 196) Started hinted handoff for endpoint
>> /10.0.100.215
>> INFO [HINTED-HANDOFF-POOL:1] 2010-12-01 10:34:44,189
>> HintedHandOffManager.java (line 252) Finished hinted handoff of 0 rows to
>> endpoint /10.0.100.215
>> INFO [GC inspection] 2010-12-01 10:40:29,141 GCInspector.java (line 129)
>> GC
>> for ParNew: 750 ms, 14693208 reclaimed leaving 2140055192 used; max is
>> 3355312128
>> INFO [GC inspection] 2010-12-01 10:40:30,280 GCInspector.java 

Re: Node goes AWOL briefly; failed replication does not report error to client, though consistency=ALL

[Jonathan Ellis]

On Tue, Dec 7, 2010 at 4:00 PM, Reverend Chip  wrote:
> On 12/7/2010 1:10 PM, Jonathan Ellis wrote:
>> I'm inclined to think there's a bug in your client, then.
>
> That doesn't pass the smell test.  The very same client has logged
> timeout and unavailable exceptions on other occasions, e.g. when there
> are too many clients or (in a previous configuration) when the JVMs had
> insufficient memory.  It's too much of a coincidence to believe that the
> client's exception reporting happens to fail only at the same time that
> a server experiences unexplained and problematic gossip failures.

You're probably right.

>>   DEBUG-level
>> logs could confirm or refute this by logging for each insert how many
>> replicas are being blocked for, which nodes it got responses from, and
>> whether a TimedOutException from not getting ALL replies was returned
>> to the client.
>
> Full DEBUG level logs would be a space problem; I'm loading at least 1T
> per node (after 3x replication), and these events are rare.  Can the
> DEBUG logs be limited to the specific modules helpful for this diagnosis
> of the gossip problem and, secondarily, the failure to report
> replication failure?

The gossip problem is almost certainly due to a GC pause.  You can
check that by enabling verbose GC logging (uncomment the lines in
cassandra-env.sh).

The replication failure is what we want DEBUG logs for, and
restricting it to the right modules isn't going to help since when
you're stress-testing writes, the write modules are going to be 99% of
the log volume anyway.

Maybe a script to constantly throw away all but the most recent log
file until you see the WARN line would be sufficient workaround?

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: How to Tell if Decommission has Completed

[Nick Bailey]

I believe the decommission call is blocking in both .6 and .7, so once it
returns it should have completed.

On Wed, Dec 8, 2010 at 3:10 AM, Jake Maizel  wrote:

> Hello,
>
> Is there a definitive way to tell if a Decommission operation has
> completed, such as a log message similar to what happens with a Drain
> command?
>
> Thanks.
>
> --
> Jake Maizel
> Network Operations
> Soundcloud
>
> Mail & GTalk: j...@soundcloud.com
> Skype: jakecloud
>
> Rosenthaler strasse 13, 101 19, Berlin, DE
>

Re: Ran out of space during cleanup.. HELP

[Mark]

Did both but didn't seem to help. I have another drive on that machine 
with some free space. If I add another directory to the 
DataFileDirectory config and restart, will it start using that directory?


Anything else I can do?

This actually leads me to an important question. Should I always make 
sure that Cassandra doesn't get past 50% of the drives free space, 
otherwise an anticompaction like this can just destroy the machine?


On 12/8/10 1:12 AM, Jake Maizel wrote:

Also, look for any snapshots that can be cleared with nodetool
clearsnapshot or just run the command to remove any that exist.

On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev  wrote:

Mark  gmail.com>  writes:


Caused by: java.lang.RuntimeException: Insufficient disk space to flush
  at
On 12/7/10 8:44 PM, Mark wrote:

3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
are now at 100% disk space. What should I do?




Is there files with -tmp n their names ?
Try to remove them to free up disk space.

Re: If one seed node crash, how can I add one seed node?

[lei liu]

Thanks all,
I have three questions:
1. Must seed list be identical on all nodes ?
2. If one seed node crash, all node will communication with the failure seed
node, I think this is harm for all nodes, isn't it？if this yes， how I can
replace the failure seed node in all node?



2010/12/8 Jonathan Ellis 

> "Two seeds per datacenter" is my rule of thumb.
>
> On Tue, Dec 7, 2010 at 3:07 PM, Eric Gilmore  wrote:
>
>> What would comprise a sane and reasonably balanced list?  Should there be
>> a certain proportion of seeds per total nodes?  Any other considerations
>> besides a) list must be identical on all nodes and b) you can't
>> auto-bootstrap a seed node?
>>
>> I'm new to thinking about this setting, but it sounds like this discussion
>> may be approaching some best-practice guidelines.
>>
>>
>> On Tue, Dec 7, 2010 at 1:01 PM, Jonathan Ellis  wrote:
>>
>>> The gossip-to-seed each round is to prevent cluster partitions, so if
>>> you're following correct procedure and making every node's seed list
>>> identical, then any potential new nodes gossiping to one of the old seeds
>>> means it is still harmless for old nodes not to gossip to the new one until
>>> the next restart.
>>>
>>>
>>> On Tue, Dec 7, 2010 at 2:10 PM, Aaron Morton wrote:
>>>
 Ryan,
 I've not checked with the code but the wiki docs for the Gossip Protocol
 say it makes use of the seed list.
 http://wiki.apache.org/cassandra/ArchitectureGossip

 During each gossip round a node will try to gossip to one seed node.

 Which made me think keeping the list sane and reasonably balanced was a
 good idea. Obviously would not matter too much on a small cluster though.

 Aaron


 On 08 Dec, 2010,at 07:16 AM, Ryan King  wrote:

 Note that there's not really anything special about the seed node and
 its all relative�C the cluster doesn't necessarily have to agreed on who 
 the
 seeds are.

 So, to bring up a new node to replace the old seed, just set the new
 node's seed to any existing node in the system. After that you can go back
 and make the setting consistent across the cluster.

 -ryan

 On Tue, Dec 7, 2010 at 7:01 AM, Nick Bailey  wrote:

> Yes, cassandra only reads the configuration when it starts up. However
> seed nodes are only used when a node starts. After that they aren't 
> needed.
> There should be no reason to restart your cluster after adding a seed node
> to  your cluster.
>
>
>
> On Tue, Dec 7, 2010 at 2:09 AM, aaron morton 
> wrote:
>
>> You will need to restart the nodes for them to pickup changes in
>> cassandra.yaml
>>
>>
>> Aaron
>>
>>
>> On 7 Dec 2010, at 16:32, lei liu wrote:
>>
>> Thanks Nick.
>>
>> After I add the new node as seed node in the configuration for all of
>> my nodes, do I need to restart all of my nodes?
>>
>> 2010/12/7 Nick Bailey 
>>
>>> The node can be set as a seed node at any time. It does not need to
>>> be a seed node when it joins the cluster. You should remove it as a seed
>>> node, set autobootstrap to true and let it join the cluster. Once it has
>>> joined the cluster you should add it as a seed node in the 
>>> configuration for
>>> all of your nodes.
>>>
>>>
>>>
>>> On Mon, Dec 6, 2010 at 9:59 AM, lei liu  wrote:
>>>
 Thank Jonathan for your reply.

 How  can I bootstrap the node into cluster, I know if the node is
 seed node, I can't set AutoBootstrap to true.

 2010/12/6 Jonathan Ellis 

> set it as a seed _after_ bootstrapping it into the cluster.
>
>
> On Mon, Dec 6, 2010 at t5:01 AM, lei liu 
> wrote:
> > After one seed node crash, I want to add one node as seed node, I
> set
> > auto_bootstrap to true, but the new node don't migrate data from
> other
> > node s.
> >
> > How can I add one new seed node and let the node to migrate data
> from other
> > nodes?
> >
> >
> >
> > Thanks,
> >
> > LiuLei
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra
> support
> http://riptano.com
>


>>>
>>
>>
>

>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>>
>>
>>
>>
>> --
>> *Eric Gilmore
>> *
>> Consulting Technical Writer
>> Riptano, Inc.
>> Ph: 510 684 9786  (cell)
>>
>>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.c

Re: How to Tell if Decommission has Completed

[Jake Maizel]

Indeed, it is.  Also, the node being decommissioned drops out of the
ring when it is completed.  Trail and error.  Thanks for following up.

On Wed, Dec 8, 2010 at 4:39 PM, Nick Bailey  wrote:
> I believe the decommission call is blocking in both .6 and .7, so once it
> returns it should have completed.
>
> On Wed, Dec 8, 2010 at 3:10 AM, Jake Maizel  wrote:
>>
>> Hello,
>>
>> Is there a definitive way to tell if a Decommission operation has
>> completed, such as a log message similar to what happens with a Drain
>> command?
>>
>> Thanks.
>>
>> --
>> Jake Maizel
>> Network Operations
>> Soundcloud
>>
>> Mail & GTalk: j...@soundcloud.com
>> Skype: jakecloud
>>
>> Rosenthaler strasse 13, 101 19, Berlin, DE
>
>



-- 
Jake Maizel
Network Operations
Soundcloud

Mail & GTalk: j...@soundcloud.com
Skype: jakecloud

Rosenthaler strasse 13, 101 19, Berlin, DE

Re: Ran out of space during cleanup.. HELP

[Jake Maizel]

I was in a similar situation and luckily had snapshots to clear and
gain space but you are correct.  I would be careful of using the disk
more than 50% as the anit-compaction during cleanup could fail.

I don't have any experience with adding a data directory on the fly.

On Wed, Dec 8, 2010 at 4:51 PM, Mark  wrote:
> Did both but didn't seem to help. I have another drive on that machine with
> some free space. If I add another directory to the DataFileDirectory config
> and restart, will it start using that directory?
>
> Anything else I can do?
>
> This actually leads me to an important question. Should I always make sure
> that Cassandra doesn't get past 50% of the drives free space, otherwise an
> anticompaction like this can just destroy the machine?
>
> On 12/8/10 1:12 AM, Jake Maizel wrote:
>>
>> Also, look for any snapshots that can be cleared with nodetool
>> clearsnapshot or just run the command to remove any that exist.
>>
>> On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev
>>  wrote:
>>>
>>> Mark  gmail.com>  writes:
>>>
 Caused by: java.lang.RuntimeException: Insufficient disk space to flush
      at
 On 12/7/10 8:44 PM, Mark wrote:
>
> 3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
> are now at 100% disk space. What should I do?

>>>
>>> Is there files with -tmp n their names ?
>>> Try to remove them to free up disk space.
>>>
>>>
>>>
>>
>>
>



-- 
Jake Maizel
Network Operations
Soundcloud

Mail & GTalk: j...@soundcloud.com
Skype: jakecloud

Rosenthaler strasse 13, 101 19, Berlin, DE

Re: Ran out of space during cleanup.. HELP

[Mark]

Does cassandra suffer from this same issue in 0.7? You would think it 
would at least warn if not prevent anticompaction if it knows there is a 
good chance of running out of space.


On 12/8/10 8:05 AM, Jake Maizel wrote:

I was in a similar situation and luckily had snapshots to clear and
gain space but you are correct.  I would be careful of using the disk
more than 50% as the anit-compaction during cleanup could fail.

I don't have any experience with adding a data directory on the fly.

On Wed, Dec 8, 2010 at 4:51 PM, Mark  wrote:

Did both but didn't seem to help. I have another drive on that machine with
some free space. If I add another directory to the DataFileDirectory config
and restart, will it start using that directory?

Anything else I can do?

This actually leads me to an important question. Should I always make sure
that Cassandra doesn't get past 50% of the drives free space, otherwise an
anticompaction like this can just destroy the machine?

On 12/8/10 1:12 AM, Jake Maizel wrote:

Also, look for any snapshots that can be cleared with nodetool
clearsnapshot or just run the command to remove any that exist.

On Wed, Dec 8, 2010 at 9:04 AM, Oleg Anastasyev
  wrote:

Markgmail.com>writes:


Caused by: java.lang.RuntimeException: Insufficient disk space to flush
  at
On 12/7/10 8:44 PM, Mark wrote:

3 Node cluster and I just ran a nodetool cleanup on node #3. 1 and 2
are now at 100% disk space. What should I do?

Is there files with -tmp n their names ?
Try to remove them to free up disk space.

Re: Running multiple instances on a single server --micrandra ??

[Anand Somani]

Interesting idea, .

If it is like dividing the entire load on the system by 6, so if the
effective load is still the same and used SSD's for commit volume we could
get away with 1 commitlog SSD. Even if these 6 instances can handle 80% of
the load (compared to 1 on this machine), that might be acceptable. Could
that help?

I mean the benefits of smaller cassandra nodes does sound very enticing.
Sure we would probably have to throw more memory/CPU at it to get comparable
to 1 instance on that box (or reduce the load), but it does look better than
6 boxes.

On Tue, Dec 7, 2010 at 10:00 PM, Jonathan Ellis  wrote:

> The major downside is you're going to want to let each instance have
> its own dedicated commitlog spindle too, unless you just don't have
> many updates.
>
> On Tue, Dec 7, 2010 at 8:25 PM, Edward Capriolo 
> wrote:
> > I am quite ready to be stoned for this thread but I have been thinking
> > about this for a while and I just wanted to bounce these ideas of some
> > guru's.
> >
> > Cassandra does allow multiple data directories, but as far as I can
> > tell no one runs in this configuration. This is something that is very
> > different between the hbase architecture and the Cassandra
> > architecture. HBase borrows the concept from hadoop of JBOD
> > configurations. HBase has many small ish (~256 MB) regions managed
> > with Zookeeper. Cassandra has a few (1 per node) large node sized
> > Token Ranges managed by Gossip consensus.
> >
> > Lets say a node has 6 300 GB disks. You have the options of RAID5,
> > RAID6, RAID10, or RAID0. The problem I have found with these
> > configurations are major compactions (of even large minor ones) can
> > take a long time. Even if your disk is not heavily utilized this is a
> > lot of data to move through. Thus node joins take a long time. Node
> > moves take a long time.
> >
> > The idea behind "micrandra" is for a 6 disk system run 6 instances of
> > Cassandra, one per disk. Use the RackAwareSnitch to make sure no
> > replicas live on the same node.
> >
> > The downsides
> > 1) we would have to manage 6x the instances of cassandra
> > 2) we would have some overhead for each JVM.
> >
> > The upsides ?
> > 1) Since disk/instance failure only degrades the overall performance
> > 1/6th (RAID0 you lost the entire node) (RAID5 still takes a hit when
> > down a disk)
> > 2) Moves and joins have less work to do
> > 3) Can scale up a single node by adding a single disk to an existing
> > system (assuming the ram and cpu is light)
> > 4) OPP would be "easier" to balance out hot spots (maybe not on this
> > one in not an OPP)
> >
> > What does everyone thing? Does it ever make sense to run this way?
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Re: About a drastic change in performance

[Nick Bailey]

On Wed, Dec 8, 2010 at 1:19 AM, Arijit Mukherjee  wrote:

>  So how do you iterate over all records


You can iterate over your records with RandomPartitioner, they will just be
in the order of their hash, not the order of the keys.


> or try to find a list of all records matching a certain criteria?


It sounds like you want a secondary index on a specific column.
http://www.riptano.com/blog/whats-new-cassandra-07-secondary-indexes



>
> Arijit
>
> On 7 December 2010 15:41, Arijit Mukherjee  wrote:
> > Hi All
> >
> > I was building an application which stores some telecom call records
> > in a Cassandra store and later performs some analysis on them. I
> > created two versions, (1) - where the key is of the form "A|B" where A
> > and B are two mobile numbers and A calls B, and (2) - where the key is
> > of the form "A|B|TS" where A and B are same as before and TS is a time
> > stamp (the time when the call was made). In both cases, the record
> > structure is as follows: [key, DURATION_S1, DURATION_S2], where S1 and
> > S2 are two different sources for the same information (two network
> > elements).
> >
> > Basically I have two files from two network elements, and I parse the
> > files and store the records in Cassandra. In both versions, there is a
> > column family ()
> > which is used to store the records. In the first version, as (A|B) may
> > occur multiple times within the files, the DURATION_S* fields are
> > updated every time a duplicate key is encountered. In the second case,
> > (A|B|TS) is unique - so there is no need for updating the DURATION_S*
> > fields. Thus, in the second case, the number of records in the Event
> > CF is slightly more - 9590, compared to 8378 in the 1st case.
> >
> > In both versions, the records are processed and sent to Cassandra
> > within a reasonable period of time. The problem is in a range_slice
> > query. I am trying to find all the records for which DURATION_S1 !=
> > DURATION_S2. And in fact, the code to do this is almost the same in
> > both versions:
> >
> > /**
> > * Selects all entries from the Event CF and finds out all those
> > entries where
> > * DUR_SRC1 != DUR_SRC2
> > */
> >public void findDurationMismatches() {
> >boolean iterate = true;
> >long count = 0;
> >long totalCount = 0;
> >int rowCount = 500;
> >try {
> >KeyRange keyRange = new KeyRange();
> >keyRange.setStart_key("");
> >keyRange.setEnd_key("");
> >// use a keyCount of 500 - means iterate over 500 records
> > until all keys are considered
> >keyRange.setCount(rowCount);
> >List columns = new ArrayList();
> >columns.add("DUR_SRC1".getBytes(ENCODING));
> >columns.add("DUR_SRC2".getBytes(ENCODING));
> >
> >SlicePredicate slicePredicate = new SlicePredicate();
> >slicePredicate.setColumn_names(columns);
> >ColumnParent columnParent = new ColumnParent(EVENT_CF);
> >List keySlices =
> > client.get_range_slices(KEYSPACE, columnParent,
> >slicePredicate, keyRange, ConsistencyLevel.ONE);
> >
> >while (iterate) {
> >//logger.debug("Number of rows retrieved: " +
> keySlices.size());
> >totalCount = totalCount + keySlices.size();
> >if (keySlices.size() < rowCount) {
> >// this is the last set
> >iterate = false;
> >}
> >for (KeySlice keySlice : keySlices) {
> >List result =
> keySlice.getColumns();
> >if (result.size() == 2) {
> >String count_src1 = new
> > String(result.get(0).getColumn().getValue(), ENCODING);
> >String count_src2 = new
> > String(result.get(1).getColumn().getValue(), ENCODING);
> >if (!count_src1.equals(count_src2)) {
> >count++;
> >//printToConsole(keySlice.getKey(),
> > keySlice.getColumns());
> >}
> >keyRange.setStart_key(keySlice.getKey());
> >}
> >keySlices = client.get_range_slices(KEYSPACE,
> > columnParent, slicePredicate,
> >keyRange, ConsistencyLevel.ONE);
> >}
> >}
> >logger.debug("Found " + count + " records with mismatched
> > duration fields.");
> >logger.debug("Total number of records processed: " +
> totalCount);
> >
> >} catch (Exception exception) {
> >exception.printStackTrace();
> >logger.error("Exception: " + exception.getMessage());
> >}
> >}
> >
> > The trouble is - the same code, takes more than 5 mins to iterate over
> > 9590 records in the 2nd version, whereas it takes about 2-3 seconds to
> > iterate over 8300 records in the 1st ve

Re: issue for cassandra-cli on 0.7-rc1

[Ned Wolpert]

Just a note that the README.txt file doesn't show using the ';' in the
command.

On Thu, Dec 2, 2010 at 10:46 AM, Yikuo Chan  wrote:

> Hi Norman :
>
> it's work and  thanks for your help ..
>
> Kevin
>
>
> On Fri, Dec 3, 2010 at 1:43 AM, Norman Maurer  wrote:
>
>> You need to terminate the command with a ";".
>>
>> Try:
>> help;
>>
>> Bye,
>> Norman
>>
>> 2010/12/2 Yikuo Chan :
>> > Hi there ,
>> >
>> > After install cassandra 0.7rc1 , I get no respond problem after execute
>> > cassandra-cli , please reference below log and help
>> > me to fix this issue :
>> >
>> > [r...@xxx cassandra]# bin/cassandra-cli --host 10.31.23.22
>> > Connected to: "xxx Test Cluster" on 10.31.23.22/9160
>> > Welcome to cassandra CLI.
>> >
>> > Type 'help' or '?' for help. Type 'quit' or 'exit' to quit.
>> > [defa...@unknown] help (enter)
>> > help (enter)
>> >(blank , CLI no respond )
>> > [r...@sjdc-ers-zb1 cassandra]#
>> >
>> >
>> > thanks
>> >
>> > Kevin
>> >
>>
>
>


-- 
Virtually, Ned Wolpert

"Settle thy studies, Faustus, and begin..."   --Marlowe

Re: issue for cassandra-cli on 0.7-rc1

[Brandon Williams]

On Wed, Dec 8, 2010 at 12:09 PM, Ned Wolpert wrote:

> Just a note that the README.txt file doesn't show using the ';' in the
> command.


Fixed in RC2 (pending vote completion right now.)

-Brandon

Fwd: NullPointerException in Beta3 and rc1

[Aaron Morton]

Please send this to the list rather than me personally. AaronBegin forwarded message:From: Wenjun Che Date: 08 December 2010 4:35:10 PMTo: aa...@thelastpickle.comSubject: Re: NullPointerException in Beta3 and rc1I created the CF on beta3 with:create column family RecipientChat with gc_grace=5 and comparator = 'AsciiType' and column_metadata=[{column_name:recipient,validation_class:BytesType,index_type:0}]After I added about 5000 rows, I got the error when querying the CF with recipient='somevalue' and anotherColumn='anotherValue'.
I tried truncating the CF and it was still getting the same error.The last thing I tried is upgrading to rc1 and saw the same error.Thanks

Re: NullPointerException in Beta3 and rc1

[Aaron Morton]

Jonathan suggested your cluster has multiple schemas, caused by https://issues.apache.org/jira/browse/CASSANDRA-1824Can you run this API command describe_schema_versions() , it's not listed on the wiki yet but it will tell you how many schema versions are out there. pycassa supports it. AaronOn 09 Dec, 2010,at 08:19 AM, Aaron Morton  wrote:Please send this to the list rather than me personally. AaronBegin forwarded message:From: Wenjun Che Date: 08 December 2010 4:35:10 PMTo: aa...@thelastpickle.comSubject: Re: NullPointerException in Beta3 and rc1I created the CF on beta3 with:create column family RecipientChat with gc_grace=5 and comparator = 'AsciiType' and column_metadata=[{column_name:recipient,validation_class:BytesType,index_type:0}]After I added about 5000 rows, I got the error when querying the CF with recipient='somevalue' and anotherColumn='anotherValue'.
I tried truncating the CF and it was still getting the same error.The last thing I tried is upgrading to rc1 and saw the same error.Thanks

Re: If one seed node crash, how can I add one seed node?

[Aaron Morton]

1. Ideally yes, but the system will work if they are not. 2. No. Once the node is down they will stop sending requests to it, and gossip is designed to test down nodes to see if they are back up.AaronOn 09 Dec, 2010,at 04:54 AM, lei liu  wrote:Thanks all,I have three questions:1. Must seed list be identical on all nodes ?2. If one seed node crash, all node will communication with the failure seed node, I think this is harm for all nodes, isn't it？if this yes， how I can replace the failure seed node in all node? 
 2010/12/8 Jonathan Ellis 
"Two seeds per datacenter" is my rule of thumb.On Tue, Dec 7, 2010 at 3:07 PM, Eric Gilmore  wrote:

What would comprise a sane and reasonably balanced list?  Should there be a certain proportion of seeds per total nodes?  Any other considerations besides a) list must be identical on all nodes and b) you can't auto-bootstrap a seed node?


I'm new to thinking about this setting, but it sounds like this discussion may be approaching some best-practice guidelines.On Tue, Dec 7, 2010 at 1:01 PM, Jonathan Ellis  wrote:


The gossip-to-seed each round is to prevent cluster partitions, so if you're following correct procedure and making every node's seed list identical, then any potential new nodes gossiping to one of the old seeds means it is still harmless for old nodes not to gossip to the new one until the next restart



On Tue, Dec 7, 2010 at 2:10 PM, Aaron Morton  wrote:



Ryan, I've not checked with the code but the wiki docs for the Gossip Protocol say it makes use of the seed list. http://wiki.apache.org/cassandra/ArchitectureGossip



During each gossip round a node will try to gossip to one seed node. Which made me think keeping the list sane and reasonably balanced was a good idea. Obviously would not matter too much on a small cluster though. 



AaronOn 08 Dec, 2010,at 07:16 AM, Ryan King  wrote:



Note that there's not really anything special about the seed node and its all relative– the cluster doesn't necessarily have to agreed on who the seeds are.



So, to bring up a new node to replace the old seed, just set the new node's seed to any existing node in the system. After that you can go back and make the setting consistent across the cluster.
-ryanOn Tue, Dec 7, 2010 at 7:01 AM, Nick Bailey  wrote:




Yes, cassandra only reads the configuration when it starts up. However seed nodes are only used when a node starts. After that they aren't needed. There should be no reason to restart your cluster after adding a seed node to  your cluster.






On Tue, Dec 7, 2010 at 2:09 AM, aaron morton  wrote:






You will need to restart the nodes for them to pickup changes in cassandra.yamlAaron

On 7 Dec 2010, at 16:32, lei liu wrote:Thanks Nick.After I add the new node as seed node in the configuration for all of my nodes, do I need to restart all of my nodes?







2010/12/7 Nick Bailey 







The node can be set as a seed node at any time. It does not need to be a seed node when it joins the cluster. You should remove it as a seed node, set autobootstrap to true and let it join the cluster. Once it has joined the cluster you should add it as a seed node in the configuration for all of your nodes.









On Mon, Dec 6, 2010 at 9:59 AM, lei liu  wrote:









Thank Jonathan for your reply.How  can I bootstrap the node into cluster, I know if the node is seed node, I can't set AutoBootstrap to true.2010/12/6 Jonathan Ellis 










set it as a seed _after_ bootstrapping it into the cluster.

On Mon, Dec 6, 2010 at t5:01 AM, lei liu  wrote:
> After one seed node crash, I want to add one node as seed node, I set
> auto_bootstrap to true, but the new node don't migrate data from other
> node s.
>
> How can I add one new seed node and let the node to migrate data from other
> nodes?
>
>
>
> Thanks,
>
> LiuLei
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com





-- Jonathan EllisProject Chair, Apache Cassandraco-founder of Riptano, the source for professional Cassandra support


http://riptano.com

-- Eric GilmoreConsulting Technical Writer

Riptano, Inc.
Ph: 510 684 9786  (cell)
-- Jonathan EllisProject Chair, Apache Cassandraco-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: Node goes AWOL briefly; failed replication does not report error to client, though consistency=ALL

[Reverend Chip]

On 12/8/2010 7:30 AM, Jonathan Ellis wrote:
> On Tue, Dec 7, 2010 at 4:00 PM, Reverend Chip  wrote:
>> Full DEBUG level logs would be a space problem; I'm loading at least 1T
>> per node (after 3x replication), and these events are rare.  Can the
>> DEBUG logs be limited to the specific modules helpful for this diagnosis
>> of the gossip problem and, secondarily, the failure to report
>> replication failure?
> The gossip problem is almost certainly due to a GC pause.  You can
> check that by enabling verbose GC logging (uncomment the lines in
> cassandra-env.sh).

Makes sense.  I thought full GC events were already logged; I'll see to
the change.  Meanwhile, what's the possible remediation if we're up
against GC?  Maybe just making gossip more forgiving?  Perhaps farming
gossip to a separate process, so frozen JVM can be distinguished from
dead JVM or offline machine.

I wonder whether we're alone in seeing this problem, or if no one else
is noticing, curious, or concerned.

> The replication failure is what we want DEBUG logs for, and
> restricting it to the right modules isn't going to help since when
> you're stress-testing writes, the write modules are going to be 99% of
> the log volume anyway.
>
> Maybe a script to constantly throw away all but the most recent log
> file until you see the WARN line would be sufficient workaround?

OK, that should be workable.

Unable to Query data(get) in cassandra-cli (0.6.6)....

[rambabu pakala]

Hi,
 
Using EmbeddedCassandra Serivce inside Junit Tests(BEFORECLASS) and tests are 
running fine and no issues. Code to start the cassandra is something like the 
following:
 
BUT the issue is when i try to get the data using cassandra-cli, i am not 
getting any results. the data cleanup happens only in BEFORECLASS.
 
So any thing I am missing here and what is the reason for unable to get the 
data?
 
try {
CassandraServiceDataCleaner cleaner = new CassandraServiceDataCleaner();
cleaner.prepare();
EmbeddedCassandraService cassandra = new EmbeddedCassandraService();
cassandra.init();
Thread t = new Thread(cassandra);
t.setDaemon(true);
t.start();

}
catch (Exception e) {
e.printStackTrace();
}


  

fauna cassandra client 0.9.0

[Ryan King]

I just pushed a 0.9.0 release of the fauna-cassandra ruby client. This
is our first release that includes support for Cassandra 0.7
(currently supporting RC1 and not earlier 0.7 releases).

code/download: https://rubygems.org/gems/cassandra
git: http://github.com/fauna/cassandra

File any bugs on github, please

-ryan

Re: fauna cassandra client 0.9.0

[Joshua Partogi]

Nice. Thanks for the hardwork Ryan. Will try this out tonight.

Cheers,
Joshua.

On Thu, Dec 9, 2010 at 8:44 AM, Ryan King  wrote:

> I just pushed a 0.9.0 release of the fauna-cassandra ruby client. This
> is our first release that includes support for Cassandra 0.7
> (currently supporting RC1 and not earlier 0.7 releases).
>
> code/download: https://rubygems.org/gems/cassandra
> git: http://github.com/fauna/cassandra
>
> File any bugs on github, please
>
> -ryan
>



-- 
http://twitter.com/jpartogi 

temp/data

[Mark]

What is this directory used for and how was it created?

Re: temp/data

[Aaron Morton]

You'll need to provide some more information. Is it under / or under something else? What was in it?These are the yaml settings that control where cassandra stores data...# directories where Cassandra should store data on disk.data_file_directories:    - /var/lib/cassandra/data# commit logcommitlog_directory: /var/lib/cassandra/commitlog# saved cachessaved_caches_directory: /var/lib/cassandra/saved_cachesAaronOn 09 Dec, 2010,at 01:09 PM, Mark  wrote:What is this directory used for and how was it created?

Re: temp/data

[Robert Coli]

On Wed, Dec 8, 2010 at 4:09 PM, Mark  wrote:
> What is this directory used for and how was it created?

I believe you may be referring to the temp directory used for, for
example, a place to put SSTable files which are created as a part of
streaming?

I presume, that like other directories used by Cassandra, it was
created by a best-faith effort from Cassandra to ensure that all
relevant directories exist.

=Rob