OutboundTcpConnection in 1.0.0 may stop working

[liangfeng]

In cassandra1.0.0, I found that OutboundTcpConnection will throw
RuntimeException when it encounters an IOException(in write()) .In this case,
OutboundTcpConnection as a thread will stop working--do not send any message to
other nodes.
This is not reasonable,isn't it?

Re: OutboundTcpConnection in 1.0.0 may stop working

[Sylvain Lebresne]

On Wed, Sep 21, 2011 at 9:59 AM, liangfeng  wrote:
> In cassandra1.0.0, I found that OutboundTcpConnection will throw
> RuntimeException when it encounters an IOException(in write()) .In this case,
> OutboundTcpConnection as a thread will stop working--do not send any message 
> to
> other nodes.
> This is not reasonable,isn't it?

I don't think it is. This was introduced by CASSANDRA-1788 so maybe there was a
good reason for it but I don't see it right away if there is one. This
is likely a mistake.

Do you mind opening a ticket on JIRA for this ?

--
Sylvain

delete counter via CLI

[Radim Kolar]

I cant get counter from CLI deleted:

[default@whois] get ipbans['10.0.0.7'];
=> (counter=hits, value=18)
Returned 1 results.
[default@whois] del ipbans['10.0.0.7'];
null
[default@whois] get ipbans['10.0.0.7'];
=> (counter=hits, value=18)
Returned 1 results.
[default@whois]

Re: -DCassandra.join_ring=false;

[aaron morton]

If you have node 1 with token 0 and a total token range to 10 then…

* add node 2 with token 3 and it will only take ownership of the range 0 to 3. 
Node 1 will own 3 to 0
* add node 3 with token 6 and it will take ownership of the range 3 to 6.  Node 
1 will now own 6 to 0

Unless you have an urgent need I would one node at a time and let the bootstrap 
finish.

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21/09/2011, at 6:38 PM, Yang wrote:

> CASSANDRA-526 provides this ability,  I just want to make sure,
> 
> let's say I have one node, with token 0, and now I want to add 2 new
> ones, with initial_token set at 1/3 and 2/3 of the full range.
> 
> now I start nodes 2 and 3 with the -DCassandra.join_ring=false option,
> and later use JMX joinRing() to let 2 and 3 join.
> 
> if there is a slight lag between the time 2 and 3 talk to 1, would it
> be possible that 2/3 of the keyspace is shipped out to node 2, and
> then when node 3 joins, it's shuffled out to node 3 again
> (unnecessarily) ?  basically I want everyone to come together, agree
> on
> the current global membership, then start carving out the existing load.
> 
> 
> Thanks
> Yang

Read failure when adding node + move; Or: What is the right way to add a node?

[David Boxenhorn]

Initial state: 3 nodes, RF=3, version = 0.7.8, some queries are with
CL=QUORUM

1. Add node with with correct token for 4 nodes, repair
2. Move first node to balance 4 nodes, repair
3. Move second node

===> Start getting timeouts, Hector warning: WARNING - Error:
me.prettyprint.hector.api.exceptions.HUnavailableException: : May not be
enough replicas present to handle consistency level.

What is going on? My traffic isn't high. None of my nodes' logs show
ANYTHING during the move

4. When the node finishes moving, the timeouts stop happening

Is there some state in the above scenario that I don't have the required
replication of at least 2?

Re: OutboundTcpConnection in 1.0.0 may stop working

[liangfeng]

> Do you mind opening a ticket on JIRA for this ?

 Yes,if it is a mistake. Thanks!

deleted counters keeps their value?

[Radim Kolar]

Another problem with counters: Counter delete + incr does not set value 
to 1 but to old value before deletion + 1


[default@whois] list ipbans;
Using default limit of 100
---
RowKey: 10.0.0.7

1 Row Returned.
[default@whois] incr ipbans['10.0.0.7']['hits'];
Value incremented.
[default@whois] list ipbans;
Using default limit of 100
---
RowKey: 10.0.0.7
=> (counter=hits, value=23)

1 Row Returned.
[default@whois]

Re: deleted counters keeps their value?

[aaron morton]

see technical limitations for deleting counters 
http://wiki.apache.org/cassandra/Counters

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21/09/2011, at 9:42 PM, Radim Kolar wrote:

> Another problem with counters: Counter delete + incr does not set value to 1 
> but to old value before deletion + 1
> 
> [default@whois] list ipbans;
> Using default limit of 100
> ---
> RowKey: 10.0.0.7
> 
> 1 Row Returned.
> [default@whois] incr ipbans['10.0.0.7']['hits'];
> Value incremented.
> [default@whois] list ipbans;
> Using default limit of 100
> ---
> RowKey: 10.0.0.7
> => (counter=hits, value=23)
> 
> 1 Row Returned.
> [default@whois]
> 

Re: deleted counters keeps their value?

[Radim Kolar]

Dne 21.9.2011 12:07, aaron morton napsal(a):
see technical limitations for deleting counters 
http://wiki.apache.org/cassandra/Counters
For instance, if you issue very quickly the sequence "increment, remove, 
increment" it is possible for the removal to be lost (if for some reason 
the remove happens to be the last received messages).


But i do not remove then very quickly. it does that even with 60 seconds 
between delete and increment. I do not understand what means: "remove 
happens to be the last received messages".

Re: deleted counters keeps their value?

[David Boxenhorn]

The reason why counters work is that addition is commutative, i.e.

x + y = y + x

but deletes are not commutative, i.e.

x + delete ≠ delete + x

so the result depends on the order in which the messages arrive.


2011/9/21 Radim Kolar 

> Dne 21.9.2011 12:07, aaron morton napsal(a):
>
>  see technical limitations for deleting counters http://wiki.apache.org/**
>> cassandra/Counters 
>>
> For instance, if you issue very quickly the sequence "increment, remove,
> increment" it is possible for the removal to be lost (if for some reason the
> remove happens to be the last received messages).
>
> But i do not remove then very quickly. it does that even with 60 seconds
> between delete and increment. I do not understand what means: "remove
> happens to be the last received messages".
>

Re: Setting java heap size for Cassandra process

[Evgeniy Ryabitskiy]

Looks like I have same problem as here:
https://issues.apache.org/jira/browse/CASSANDRA-2868

But, it's been fixed in 0.8.5 and I'm using 0.8.5 ...


Evgeny.

Re: OutboundTcpConnection in 1.0.0 may stop working

[Jonathan Ellis]

Yeah, looks like it should just pass through to where writeConnected
can handle it more gracefully like it did pre-1788.

On Wed, Sep 21, 2011 at 3:48 AM, Sylvain Lebresne  wrote:
> On Wed, Sep 21, 2011 at 9:59 AM, liangfeng  
> wrote:
>> In cassandra1.0.0, I found that OutboundTcpConnection will throw
>> RuntimeException when it encounters an IOException(in write()) .In this case,
>> OutboundTcpConnection as a thread will stop working--do not send any message 
>> to
>> other nodes.
>> This is not reasonable,isn't it?
>
> I don't think it is. This was introduced by CASSANDRA-1788 so maybe there was 
> a
> good reason for it but I don't see it right away if there is one. This
> is likely a mistake.
>
> Do you mind opening a ticket on JIRA for this ?
>
> --
> Sylvain
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Snapshots V. SSTables2Json

[David McNelis]

When planning a DR strategy, which option is going to, most consistently,
take the least amount of disk space, be fastest to recover from, least
complicated recovery, ect?

I've read through the Operations documents and my take is this so far.  If I
have specific column families I want to snapshot across the cluster, then
sstables2json would make the most sense.  However, if I want to back  up an
individual node(s), so that I can better and more quickly recover from a
node failure then snapshots would make more sense?

Regularly backing up the data on a large cluster with a high replication
factor is redundant, but in a situation where you have an RF <= 2, and are
located in a single rack / datacenter, then it might make sense to implement
something like this to backup and store data offsite, and I'm trying to
figure out  what a good, viable, and storage efficient plan would look like.

-- 
*David McNelis*
Lead Software Engineer
Agentis Energy
www.agentisenergy.com
o: 630.359.6395
c: 219.384.5143

*A Smart Grid technology company focused on helping consumers of energy
control an often under-managed resource.*

Re: OutboundTcpConnection in 1.0.0 may stop working

[Jonathan Ellis]

https://issues.apache.org/jira/browse/CASSANDRA-3235

On Wed, Sep 21, 2011 at 4:34 AM, liangfeng  wrote:
>> Do you mind opening a ticket on JIRA for this ?
>
>  Yes,if it is a mistake. Thanks!
>
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: Snapshots V. SSTables2Json

[Jonathan Ellis]

you can snapshot individual CFs.  sstable2json is primarily for debugging.

On Wed, Sep 21, 2011 at 9:17 AM, David McNelis
 wrote:
> When planning a DR strategy, which option is going to, most consistently,
> take the least amount of disk space, be fastest to recover from, least
> complicated recovery, ect?
> I've read through the Operations documents and my take is this so far.  If I
> have specific column families I want to snapshot across the cluster, then
> sstables2json would make the most sense.  However, if I want to back  up an
> individual node(s), so that I can better and more quickly recover from a
> node failure then snapshots would make more sense?
> Regularly backing up the data on a large cluster with a high replication
> factor is redundant, but in a situation where you have an RF <= 2, and are
> located in a single rack / datacenter, then it might make sense to implement
> something like this to backup and store data offsite, and I'm trying to
> figure out  what a good, viable, and storage efficient plan would look like.
> --
> David McNelis
> Lead Software Engineer
> Agentis Energy
> www.agentisenergy.com
> o: 630.359.6395
> c: 219.384.5143
> A Smart Grid technology company focused on helping consumers of energy
> control an often under-managed resource.
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: deleted counters keeps their value?

[Radim Kolar]

Dne 21.9.2011 14:44, David Boxenhorn napsal(a):
The reason why counters work is that addition is commutative, but 
deletes are not commutative

This is not my case. if you look at my 2 posts.
1st post seems to be CLI bug because same operation from program 
works fine.
 In 2nd post i already see that counter is deleted and it can not 
be caused by message ordering because it is single node cluster


in 1.0 Beta 1. case reported in my 1st post is fixed, but 2nd problem 
still exists;


on single node cluster: -> increment -> delete -> increment with 1 sec 
delays should work. How it can be fine to return stale value by counter 
design?

Re: deleted counters keeps their value?

[Sam Overton]

Hi Radim,

This is the current behaviour you will see if you are inserting a row
tombstone in a counter CF, as the row tombstone does not remove the old
counter value from the memtable. When you increment the counter after
deleting the row, the counter now has a timestamp later than the row
tombstone, and the new value is resolved with the old value which is still
in the memtable.

The behaviour described on the wiki and by others in this thread is what you
will see if inserting a column tombstone for your counter.

Sam

-- 
Sam Overton
Acunu | http://www.acunu.com | @acunu


On 21 September 2011 16:49, Radim Kolar  wrote:

> Dne 21.9.2011 14:44, David Boxenhorn napsal(a):
>
>> The reason why counters work is that addition is commutative, but deletes
>> are not commutative
>>
> This is not my case. if you look at my 2 posts.
>1st post seems to be CLI bug because same operation from program works
> fine.
> In 2nd post i already see that counter is deleted and it can not be
> caused by message ordering because it is single node cluster
>
> in 1.0 Beta 1. case reported in my 1st post is fixed, but 2nd problem still
> exists;
>
> on single node cluster: -> increment -> delete -> increment with 1 sec
> delays should work. How it can be fine to return stale value by counter
> design?
>

Re: deleted counters keeps their value?

[Sylvain Lebresne]

On Wed, Sep 21, 2011 at 5:49 PM, Radim Kolar  wrote:
> Dne 21.9.2011 14:44, David Boxenhorn napsal(a):
>>
>> The reason why counters work is that addition is commutative, but deletes
>> are not commutative
>
> This is not my case. if you look at my 2 posts.
>    1st post seems to be CLI bug because same operation from program works
> fine.
>     In 2nd post i already see that counter is deleted and it can not be
> caused by message ordering because it is single node cluster
>
> in 1.0 Beta 1. case reported in my 1st post is fixed, but 2nd problem still
> exists;
>
> on single node cluster: -> increment -> delete -> increment with 1 sec
> delays should work. How it can be fine to return stale value by counter
> design?

Incrementing after a deletion don't work. It's a current limitation and will
probably won't be "fixed" soon (and yes, the underlying reason is that
deletion and incrementation are not commutative, which is a problem
because there is no guarantee in Cassandra for a given node of the order
it will receive messages -- independently of the order of the operation the
client does).

If you need to reset a counter, you need to insert -value.

--
Sylvain

Re: deleted counters keeps their value?

[Radim Kolar]

Sam,

thank you for your detailed problem description. What is reason why 
delete cant remove old counter value from memtable? Because currently we 
need to code workaround in our applications.


It would be nice to copy your description of this problem to:
http://wiki.apache.org/cassandra/Counters
because it is more clear than existing text.

Re: deleted counters keeps their value?

[Sylvain Lebresne]

On Wed, Sep 21, 2011 at 6:19 PM, Radim Kolar  wrote:
> Sam,
>
> thank you for your detailed problem description. What is reason why delete
> cant remove old counter value from memtable?

The reason is: even if it was doing this, incrementing after a deletion would
still not work correctly. So we've avoided adding special casing in the code
since it wouldn't fix anything.

--
Sylvain


> Because currently we need to
> code workaround in our applications.
>
> It would be nice to copy your description of this problem to:
> http://wiki.apache.org/cassandra/Counters
> because it is more clear than existing text.
>

Cassandra reconfiguration

[Hiren Shah]

Hi,

We have cassandra in production in three data centers with five nodes each. We 
are getting rid of one of the data centers, so I want to reconfig my cluster to 
two data centers, seven nodes each. I first need to shut down DC3. I have 
already shut down traffic to DC3 and am running repairs. My priority #1 is to 
maintain the stability of overall cluster (the two live data centers). Some 
data loss is tolerable, but it will be ideal if I can avoid that.

What is the best way to decommission entire data center? Should I issue 
decommission to the nodes one by one? Will that put additional load on 
remaining nodes in DC3? Or should I just shut them down and then run 
removetoken for these nodes? Or is there a better way?

Thanks in advance,
Hiren Shah

Unable to create compaction marker

[Daning]

I got this exception in system.log several times a day, do you have idea 
what caused this problem and what is the consequence?



ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578 
AbstractCassandraDaemon.java (line 139) Fatal exception in thread 
Thread[CompactionExecutor:12,1,main]

java.io.IOError: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
at 
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
at 
org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
at 
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
at 
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
at 
org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
at 
org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
at 
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)

at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unable to create compaction marker
at 
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)

... 13 more


Thank you in advance.

Daning

Re: Unable to create compaction marker

[Jonathan Ellis]

Means Cassandra couldn't create an empty file in the data directory
designating a sstable as compacted.  I'd look for permissions
problems.

Short term there is no dire consequence, although it will keep
re-compacting that sstable.  Longer term you'll run out of disk space
since nothing gets deleted.

On Wed, Sep 21, 2011 at 1:44 PM, Daning  wrote:
> I got this exception in system.log several times a day, do you have idea
> what caused this problem and what is the consequence?
>
>
> ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
> AbstractCassandraDaemon.java (line 139) Fatal exception in thread
> Thread[CompactionExecutor:12,1,main]
> java.io.IOError: java.io.IOException: Unable to create compaction marker
>        at
> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
>        at
> org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
>        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
>        at
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
>        at
> org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
>        at
> org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
>        at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
>        at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
>        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>        at java.util.concurrent.FutureTask.run(Unknown Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>        at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Unable to create compaction marker
>        at
> org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
>        ... 13 more
>
>
> Thank you in advance.
>
> Daning
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Re: [VOTE] Release Mojo's Cassandra Maven Plugin 0.8.6-1

[Colin Taylor]

+1 (non binding but lgtm)

On Wed, Sep 21, 2011 at 2:27 AM, Stephen Connolly
 wrote:
> Hi,
>
> I'd like to release version 0.8.6-1 of Mojo's Cassandra Maven Plugin
> to sync up with the recent 0.8.6 release of Apache Cassandra.
>
>
> We solved 2 issues:
> http://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=12121&version=17425
>
>
> Staging Repository:
> https://nexus.codehaus.org/content/repositories/orgcodehausmojo-010/
>
> Site:
> http://mojo.codehaus.org/cassandra-maven-plugin/index.html
>
> SCM Tag:
> https://svn.codehaus.org/mojo/tags/cassandra-maven-plugin-0.8.6-1@14748
>
>  [ ] +1 Yeah! fire ahead oh and the blind man on the galloping horse
> says it looks fine too.
>  [ ] 0 Mehhh! like I care, I don't have any opinions either, I'd
> follow somebody else if only I could decide who
>  [ ] -1 No! wait up there I have issues (in general like, ya know,
> and being a trouble-maker is only one of them)
>
> The vote is open for 72 hours and will succeed by lazy consensus.
>
> Cheers
>
> -Stephen
>
> P.S.
>  In the interest of ensuring (more is) better testing, this vote is
> also open to subscribers of the dev and user@cassandra.apache.org
> mailing lists
>

Re: TransportException when storing large values

[Pete Warden]

Thanks guys, that was precisely the problem. I also upped
the thrift_max_message_length_in_mb setting, since I guessed the default of
16MB would also block larger messages.

On Tue, Sep 20, 2011 at 6:47 PM, Jim Ancona  wrote:

> Pete,
>
> See this thread
>
> http://groups.google.com/group/hector-users/browse_thread/thread/cb3e72c85dbdd398/82b18ffca0e3940a?#82b18ffca0e3940a
> for a bit more info.
>
> Jim
>
> On Tue, Sep 20, 2011 at 9:02 PM, Tyler Hobbs  wrote:
> > From cassandra.yaml:
> >
> > # Frame size for thrift (maximum field length).
> > # 0 disables TFramedTransport in favor of TSocket. This option
> > # is deprecated; we strongly recommend using Framed mode.
> > thrift_framed_transport_size_in_mb: 15
> >
> > So you can either increase that limit, or split your write into multiple
> > operations.
> >
> > On Tue, Sep 20, 2011 at 4:26 PM, Pete Warden 
> wrote:
> >>
> >> I'm running into consistent problems when storing values larger than
> 15MB
> >> into Cassandra, and I was hoping for some advice on tracking down what's
> >> going wrong. From the FAQ it seems like what I'm trying to do is
> possible,
> >> so I assume I'm messing something up with my configuration. I have a
> minimal
> >> set of code to reproduce the issue below, which I've run on the DataStax
> >> 0.8.1 AMI I'm using in production (ami-9996c4dc)
> >> # To set up the test data structure on Cassandra:
> >> cassandra-cli
> >> connect localhost/9160;
> >> create keyspace TestKeyspace with
> >>   placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and
> >>   strategy_options = [{replication_factor:3}];
> >> use TestKeyspace;
> >> create column family TestFamily with
> >>   comparator = UTF8Type and
> >>   column_metadata =
> >>   [
> >> {column_name: test_column, validation_class: UTF8Type},
> >>   ];
> >> # From bash on the same machine, with Ruby and the Cassandra gem
> >> installed:
> >> irb
> >> require 'rubygems'
> >> require 'cassandra/0.8'
> >> client = Cassandra.new('TestKeyspace', 'localhost:9160', :retries => 5,
> >> :connect_timeout => 5, :timeout => 10)
> >> # With data this size, the call works
> >> column_value = 'a' * (14*1024*1024)
> >> row_value = { 'column_name' => column_value }
> >> client.insert(:TestFamily, 'SomeKey', row_value)
> >>
> >> # With data this size, the call fails with the exception below
> >> column_value = 'a' * (15*1024*1024)
> >> row_value = { 'column_name' => column_value }
> >> client.insert(:TestFamily, 'SomeKey', row_value)
> >> # Results:
> >> This first call with a 14MB chunk of data succeeds, but the second one
> >> fails with this exception:
> >> CassandraThrift::Cassandra::Client::TransportException:
> >> CassandraThrift::Cassandra::Client::TransportException
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:53:in
> >> `open'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:37:in
> >> `open'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/connection/socket.rb:11:in
> >> `connect!'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:105:in
> >> `connect!'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:144:in
> >> `handled_proxy'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in
> >> `batch_mutate'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:7:in
> >> `_mutate'
> >> from
> >>
> /usr/lib/ruby/gems/1.8/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:459:in
> >> `insert'
> >> from (irb):6
> >> from :0
> >> Any suggestions on how to dig deeper? I'll be reaching out to the
> >> Cassandra gem folks, etc too of course.
> >> cheers,
> >>Pete
> >
> >
> > --
> > Tyler Hobbs
> > Software Engineer, DataStax
> > Maintainer of the pycassa Cassandra Python client library
> >
> >
>

Moving to a new cluster

[Philippe]

Hello,
We're currently running on a 3-node RF=3 cluster. Now that we have a better
grip on things, we want to replace it with a 12-node RF=3 cluster of
"smaller" servers. So I wonder what the best way to move the data to the new
cluster would be. I can afford to stop writing to the current cluster for
whatever time is necessary. Has anyone written up something on this subject
?

My plan is the following (nodes in cluster 1 are node1.1->1.3, nodes in
cluster 2 are node2.1->2.12)

   - stop writing to current cluster & drain it
   - get a snapshot on each node
   - Since it's RF=3, each node should have all the data, so assuming I set
   the tokens correctly I would move the snapshot from node1.1 to node2.1, 2.2,
   2.3 and 2.4 then node1.2->node2.5,2.6,2.,2.8, etc. This is because the range
   for node1.1 is now spread across 2.1->2.4
   - Run repair & clean & scrub on each node (more or less in //)

What do you think ?
Thanks

Re: Snapshots V. SSTables2Json

[aaron morton]

It all depends on what sort of disasters you are planning for and how valuable 
your data is.

The cheap and cheerful approach is to snapshot and then rsync / copy off the 
node. Or you can do something like https://github.com/synack/tablesnap . 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 22/09/2011, at 2:27 AM, Jonathan Ellis wrote:

> you can snapshot individual CFs.  sstable2json is primarily for debugging.
> 
> On Wed, Sep 21, 2011 at 9:17 AM, David McNelis
>  wrote:
>> When planning a DR strategy, which option is going to, most consistently,
>> take the least amount of disk space, be fastest to recover from, least
>> complicated recovery, ect?
>> I've read through the Operations documents and my take is this so far.  If I
>> have specific column families I want to snapshot across the cluster, then
>> sstables2json would make the most sense.  However, if I want to back  up an
>> individual node(s), so that I can better and more quickly recover from a
>> node failure then snapshots would make more sense?
>> Regularly backing up the data on a large cluster with a high replication
>> factor is redundant, but in a situation where you have an RF <= 2, and are
>> located in a single rack / datacenter, then it might make sense to implement
>> something like this to backup and store data offsite, and I'm trying to
>> figure out  what a good, viable, and storage efficient plan would look like.
>> --
>> David McNelis
>> Lead Software Engineer
>> Agentis Energy
>> www.agentisenergy.com
>> o: 630.359.6395
>> c: 219.384.5143
>> A Smart Grid technology company focused on helping consumers of energy
>> control an often under-managed resource.
>> 
>> 
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Re: Cassandra reconfiguration

[aaron morton]

What are the token assignments and what is the RF ? 

Without knowing the details I would guess…

Make the RF changes you want for DC 1 and 2 and repair. 
decomission the nodes in DC3 one at a time. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 22/09/2011, at 5:29 AM, Hiren Shah wrote:

> Hi,
>  
> We have cassandra in production in three data centers with five nodes each. 
> We are getting rid of one of the data centers, so I want to reconfig my 
> cluster to two data centers, seven nodes each. I first need to shut down DC3. 
> I have already shut down traffic to DC3 and am running repairs. My priority 
> #1 is to maintain the stability of overall cluster (the two live data 
> centers). Some data loss is tolerable, but it will be ideal if I can avoid 
> that.
>  
> What is the best way to decommission entire data center? Should I issue 
> decommission to the nodes one by one? Will that put additional load on 
> remaining nodes in DC3? Or should I just shut them down and then run 
> removetoken for these nodes? Or is there a better way?
>  
> Thanks in advance,
> Hiren Shah
>  

Re: Moving to a new cluster

[aaron morton]

How much data is on the nodes in cluster 1 and how much disk space on cluster 2 
? Be aware that Cassandra 0.8 has an issue where repair can go crazy and use a 
lot of space. 

If you are not regularly running repair I would also repair before the move.

The repair after the copy is a good idea but should technically not be 
necessary. If you can practice the move watch the repair to see if much is 
transferred (check the logs). There is always a small transfer, but if you see 
data been transferred for several minutes I would investigate. 

When you start a repair it will repair will the other nodes it replicates data 
with. So you only need to run it every RF nodes. Start it one one, watch the 
logs to see who it talks to and then start it on the first node it does not 
talk to. And so on. 

Add a snapshot before the clean (repair will also snapshot before it runs)

Scrub is not needed unless you are migrating or you have file errors.

If your cluster is online, consider running the clean every RFth node rather 
than all at once (e.g. 1,4, 7, 10 then 2,5,8,11). It will have less impact on 
clients. 

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 22/09/2011, at 10:27 AM, Philippe wrote:

> Hello,
> We're currently running on a 3-node RF=3 cluster. Now that we have a better 
> grip on things, we want to replace it with a 12-node RF=3 cluster of 
> "smaller" servers. So I wonder what the best way to move the data to the new 
> cluster would be. I can afford to stop writing to the current cluster for 
> whatever time is necessary. Has anyone written up something on this subject ?
> 
> My plan is the following (nodes in cluster 1 are node1.1->1.3, nodes in 
> cluster 2 are node2.1->2.12)
> stop writing to current cluster & drain it
> get a snapshot on each node
> Since it's RF=3, each node should have all the data, so assuming I set the 
> tokens correctly I would move the snapshot from node1.1 to node2.1, 2.2, 2.3 
> and 2.4 then node1.2->node2.5,2.6,2.,2.8, etc. This is because the range for 
> node1.1 is now spread across 2.1->2.4
> Run repair & clean & scrub on each node (more or less in //)
> What do you think ?
> Thanks

Using different versions of Cassandra in production

[Anthony Ikeda]

Okay, this is leaning more towards getting Brisk into our environment and
making sure we can get it all working.

We plan on deploying to production Cassandra 0.8.5/6, however, Brisk only
works on 0.8.1 (in the 0.8.x release)

Can we have a Brisk node operating as part of the ring to still do our data
warehousing somewhat in a non-accessed state?

For our configuration, we have a ring of 2 DC's 3 nodes per DC. Our
applications will be configured using Hector to connect to all 3 nodes in
the local DC

Could we have:
* 3 Cassandra 0.8.5 Nodes
* 1 Brisk node
in one DC with the applications still connecting only to the 0.8.5 nodes?
The other DC will have only the 3 Cassandra 0.8.5 nodes.

Are there any risks to setting up this way?

My understanding is that DataStax will be contributing the Hive Cassandra
Driver to the Apache project, however, my concern is that with Brisk we had
very little config involved in getting started. I don't want to have to
install Hadoop and any dependencies separately when it can come prepackaged
and configured out of the box - yes I know this is the goal with DataStax
Enterprise.

Anthony

Re: Unable to create compaction marker

[Daning]

Thanks Jonathan. where is the data directory configured? since I did not 
any permission problem.


Daning

On 09/21/2011 11:01 AM, Jonathan Ellis wrote:

Means Cassandra couldn't create an empty file in the data directory
designating a sstable as compacted.  I'd look for permissions
problems.

Short term there is no dire consequence, although it will keep
re-compacting that sstable.  Longer term you'll run out of disk space
since nothing gets deleted.

On Wed, Sep 21, 2011 at 1:44 PM, Daning  wrote:

I got this exception in system.log several times a day, do you have idea
what caused this problem and what is the consequence?


ERROR [CompactionExecutor:12] 2011-09-15 14:29:19,578
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[CompactionExecutor:12,1,main]
java.io.IOError: java.io.IOException: Unable to create compaction marker
at
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:642)
at
org.apache.cassandra.db.DataTracker.removeOldSSTablesSize(DataTracker.java:280)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:253)
at
org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:214)
at
org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1094)
at
org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:599)
at
org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unable to create compaction marker
at
org.apache.cassandra.io.sstable.SSTableReader.markCompacted(SSTableReader.java:638)
... 13 more


Thank you in advance.

Daning

Cassandra yaml configuration

[Sajith Kariyawasam]

Hi all,

Im refering to the book authored by Eben Hewitt, named Cassandra The
Definitive Guide

There, in the Sample Application chapter (Chapter 4), example 4.1, a sample
schema is given in a file named "cassandra.yaml". (I have mentioned it
below)

I'm using Cassandra 0.8.6 version.

My question is that, whether the given format of the file is correct? if so,
where i have to mention in the cassandra configuration to load this yaml
file?
because the format of the sample schema  comes with Cassandra
(conf/schema-sample.txt) is bit different from the sample given in the book.


*Schema definition in cassandra.yaml*

*keyspaces:*
*- name: Hotelier*
*replica_placement_strategy:
org.apache.cassandra.locator.RackUnawareStrategy*
*replication_factor: 1*
*column_families:*
*- name: Hotel*
*compare_with: UTF8Type*
*- name: HotelByCity*
*compare_with: UTF8Type*
*- name: Guest*
*compare_with: BytesType*
*- name: Reservation*
*compare_with: TimeUUIDType*
*- name: PointOfInterest*
*column_type: Super*
*compare_with: UTF8Type*
*compare_subcolumns_with: UTF8Type*
*- name: Room*
*compare_with: BytesType*
*compare_subcolumns_with: BytesType*
*- name: RoomAvailability*
*column_type: Super*
*compare_with: BytesType*
*compare_subcolumns_with: BytesType*



-- 
Best Regards
Sajith

Re: Thrift 7

[Suman Ghosh]

Any suggestions on this, folks?


On Tue, Sep 6, 2011 at 12:59 AM, tushar pal  wrote:

> Hi,
> I am facing some problem using Thrift 7.
> I downloaded the tar file.I downloaded the windows exe too for . Created a
> thrift jar from the lib java path and then generated the java class from
> tutorial.thrift file.Now while I use run the ant file inside example I got
> some error that some of the package are used in that generated java files do
> not exists in that generated thrift jar.
> Can anybody please help ?
>
> --
> TUSHAR KANTI PAL
>
>
>


-- 
Get me at GMail--> sumanthewhiz[at]gmail[dot]com
... or there's Yahoo --> sumanthewhiz[at]yahoo[dot]com

Re: Cassandra yaml configuration

[Maki Watanabe]

The book is a bit out dated now.
You should better to use cassandra-cli to define your application schema.
Please refer to conf/schema-sample.txt and help in cassandra-cli.

% cassandra-cli
[default@unknown] help;
[default@unknown] help create keyspace;
[default@unknown] help create column family;

You can load schema defined in text file by cassandra-cli:
% cassandra-cli -h host -p port --file your-schema-definition.txt

maki



2011/9/22 Sajith Kariyawasam :
> Hi all,
> Im refering to the book authored by Eben Hewitt, named Cassandra The
> Definitive Guide
>
> There, in the Sample Application chapter (Chapter 4), example 4.1, a sample
> schema is given in a file named "cassandra.yaml". (I have mentioned it
> below)
> I'm using Cassandra 0.8.6 version.
> My question is that, whether the given format of the file is correct? if so,
> where i have to mention in the cassandra configuration to load this yaml
> file?
> because the format of the sample schema  comes with Cassandra
> (conf/schema-sample.txt) is bit different from the sample given in the book.
>
> Schema definition in cassandra.yaml
> 
> keyspaces:
> - name: Hotelier
> replica_placement_strategy: org.apache.cassandra.locator.RackUnawareStrategy
> replication_factor: 1
> column_families:
> - name: Hotel
> compare_with: UTF8Type
> - name: HotelByCity
> compare_with: UTF8Type
> - name: Guest
> compare_with: BytesType
> - name: Reservation
> compare_with: TimeUUIDType
> - name: PointOfInterest
> column_type: Super
> compare_with: UTF8Type
> compare_subcolumns_with: UTF8Type
> - name: Room
> compare_with: BytesType
> compare_subcolumns_with: BytesType
> - name: RoomAvailability
> column_type: Super
> compare_with: BytesType
> compare_subcolumns_with: BytesType
>
> --
> Best Regards
> Sajith
>



-- 
w3m

Re: Cassandra yaml configuration

[Sajith Kariyawasam]

Thanks Maki.

If you came across with any other book supporting latest Cassandara
versions, pls let me know.


On Thu, Sep 22, 2011 at 12:03 PM, Maki Watanabe wrote:

> The book is a bit out dated now.
> You should better to use cassandra-cli to define your application schema.
> Please refer to conf/schema-sample.txt and help in cassandra-cli.
>
> % cassandra-cli
> [default@unknown] help;
> [default@unknown] help create keyspace;
> [default@unknown] help create column family;
>
> You can load schema defined in text file by cassandra-cli:
> % cassandra-cli -h host -p port --file your-schema-definition.txt
>
> maki
>
>
>
> 2011/9/22 Sajith Kariyawasam :
> > Hi all,
> > Im refering to the book authored by Eben Hewitt, named Cassandra The
> > Definitive Guide
> >
> > There, in the Sample Application chapter (Chapter 4), example 4.1, a
> sample
> > schema is given in a file named "cassandra.yaml". (I have mentioned it
> > below)
> > I'm using Cassandra 0.8.6 version.
> > My question is that, whether the given format of the file is correct? if
> so,
> > where i have to mention in the cassandra configuration to load this yaml
> > file?
> > because the format of the sample schema  comes with Cassandra
> > (conf/schema-sample.txt) is bit different from the sample given in the
> book.
> >
> > Schema definition in cassandra.yaml
> > 
> > keyspaces:
> > - name: Hotelier
> > replica_placement_strategy:
> org.apache.cassandra.locator.RackUnawareStrategy
> > replication_factor: 1
> > column_families:
> > - name: Hotel
> > compare_with: UTF8Type
> > - name: HotelByCity
> > compare_with: UTF8Type
> > - name: Guest
> > compare_with: BytesType
> > - name: Reservation
> > compare_with: TimeUUIDType
> > - name: PointOfInterest
> > column_type: Super
> > compare_with: UTF8Type
> > compare_subcolumns_with: UTF8Type
> > - name: Room
> > compare_with: BytesType
> > compare_subcolumns_with: BytesType
> > - name: RoomAvailability
> > column_type: Super
> > compare_with: BytesType
> > compare_subcolumns_with: BytesType
> >
> > --
> > Best Regards
> > Sajith
> >
>
>
>
> --
> w3m
>



-- 
Best Regards
Sajith