Start key must sort before (or equal to) finish key in your partitioner
Hello! I am using an OrderPreservingPartitioner and sequential UUID's as keys. When I use get_range I frequently get an error indicating that my start key is not before or equal to my finish key, which seems impossible. The code to request slices looks like this: sometime = Time.now a = UUIDTools::UUID.timestamp_create(sometime) b = UUIDTools::UUID.timestamp_create(sometime + 1) slice = @cassandra.get_range(:ReportData, :start => a.to_s, :finish => b.to_s) puts "SORT ERROR!" if a >= b# this never displays This is the error I get: /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/../vendor/gen-rb/cassandra.rb:152:in `recv_get_range_slices': start key must sort before (or equal to) finish key in your partitioner! (CassandraThrift::InvalidRequestException) from /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/../vendor/gen-rb/cassandra.rb:142:in `get_range_slices' from /usr/lib/ruby/gems/1.8/gems/thrift_client-0.4.0/lib/thrift_client.rb:140:in `send' from /usr/lib/ruby/gems/1.8/gems/thrift_client-0.4.0/lib/thrift_client.rb:140:in `proxy' from (eval):1:in `get_range_slices' from /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/cassandra/protocol.rb:78:in `_get_range' from /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/cassandra/cassandra.rb:230:in `get_range' from ./read.rb:29:in `get_range' from ./read.rb:62 from ./read.rb:58:in `loop' from ./read.rb:58
Re: Creating a Cassandra cluster under Windows XP
Just to clarify things: I'm trying to get my first cluster up and running and it's not working. I have Cassandra working on one machine just fine, with all my data, but when I put Cassandra on a 2nd machine, and point it to my first machine, nothing happens. There seems to be some communication between the two machines, because when I stop Cassandra on the 1st machine, the 2nd machine tells me that the connection is lost. On Mon, May 31, 2010 at 9:09 AM, David Boxenhorn wrote: > We have a VPN with PC’s running Windows XP SP3. > > I want to define a Cassandra cluster on two PC’s in the VPN: 192.168.80.234 > and 192.168.80.12. > > I can ping from 192.168.80.12 to 192.168.80..234 and vice-versa. > > I download and install Cassandra 0.6.2 > > Why doesn’t the cluster cluster? > > I configure 192.168.80.12 to be the seed, as follows: > > > > ** > > * Lookin2* > > * false* > > * * > > * 192.168.80.12* > > * * > > * 192.168.80.12* > > * * > > * 7000* > > * 7001* > > * 192.168.80.12* > > ** > > * 9160 * > > * false* > > ** > > > > > > I configure 192.168.80.234 to be a non-seed node, as follows: > > ** > > * Lookin2* > > * true* > > * * > > * 192.168.80.12* > > * * > > * 192.168.80.234* > > * * > > * 7000* > > * 7001* > > * 192.168.80.234* > > ** > > * 9160 * > > * false* > > ** > > > > (All other values in the two storage-conf.xml files are default values from > the 0.6.2 installation.) > > > > > > I start up Cassandra on 192.168.80.12, then start up Cassandra on > 192.168.80.234. > > > > On 192.168.80.234, I see the following in the DOS console: > > > > > > *C:\apache-cassandra-0.6.2\bin>cassandra -f* > > *Starting Cassandra Server* > > *Listening for transport dt_socket at address: * > > * INFO 19:01:49,687 Auto DiskAccessMode determined to be standard* > > * INFO 19:01:49,984 Replaying > \var\lib\cassandra\commitlog\CommitLog-127523519217* > > *1.log* > > * INFO 19:01:50,031 Creating new commitlog segment > /var/lib/cassandra/commitlog\C* > > *ommitLog-1275235310031.log* > > * INFO 19:01:50,093 LocationInfo has reached its threshold; switching in a > fresh* > > *Memtable at > CommitLogContext(file='/var/lib/cassandra/commitlog\CommitLog-127523* > > *5310031.log', position=173)* > > * INFO 19:01:50,093 Enqueuing flush of Memtable(LocationInfo)@700852* > > * INFO 19:01:50,093 Writing Memtable(LocationInfo)@700852* > > * INFO 19:01:50,296 Completed flushing > C:\var\lib\cassandra\data\system\LocationI* > > *nfo-1-Data.db* > > * INFO 19:01:50,328 Log replay complete* > > * INFO 19:01:50,375 Saved Token found: > 51560289722671854279455339870443755117* > > * INFO 19:01:50,375 Saved ClusterName found: Lookin2* > > * INFO 19:01:50,390 Starting up server gossip* > > * INFO 19:01:50,421 Joining: getting load information* > > * INFO 19:01:50,421 Sleeping 9 ms to wait for load information...* > > * INFO 19:03:20,421 Joining: getting bootstrap token* > > *ERROR 19:03:20,421 Exception encountered during startup.* > > *java.lang.RuntimeException: No other nodes seen! Unable to bootstrap* > > *at > org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper* > > *.java:120)* > > *at > org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.j* > > *ava:102)* > > *at > org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.* > > *java:97)* > > *at > org.apache.cassandra.service.StorageService.initServer(StorageService* > > *.java:356)* > > *at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.jav* > > *a:99)* > > *at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java* > > *:177)* > > *Exception encountered during startup.* > > *java.lang.RuntimeException: No other nodes seen! Unable to bootstrap* > > *at > org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper* > > *.java:120)* > > *at > org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.j* > > *ava:102)* > > *at > org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.* > > *java:97)* > > *at > org.apache.cassandra.service.StorageService.initServer(StorageService* > > *.java:356)* > > *at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.jav* > > *a:99)* > > *at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java* > > *:177)* > > * * > > *C:\apache-cassandra-0.6.2\bin>* > > > > > > > > > > In system.log, I see the following: > > > > *INFO [main] 2010-05-30 19:01:49,687 DatabaseDescriptor.java (line 232) > Auto DiskAccessMode determined to be standard* > > * INFO [main] 2010-05-30 19:01:49,984 CommitLog.java (line 172) Replaying > \var\lib\cassandra\commitlog\CommitLog-1275235192171.log* > > * INFO [main] 2010-05-30 19:01:50,031 CommitLogSegment.java (line 50) > Creating new commitlog segment > /var/lib/cassandra/commitlog\CommitLog-1275235310031.log* > > * INFO [main] 201
How to drop a column family (Cassandra 6.1)
Hi In the API I see : system_drop_column_family *Requires Cassandra 0.7 * How to drop a column family with Cassandra 6.1 ? * Delete *the files MYCOLUMNFAMILY-XXX-Data.db, MYCOLUMNFAMILY-XXX-Filter.db , MYCOLUMNFAMILY-XXX-Index.db is a good ways ? Can I recreate this columnfamily after this delete ? Thanks.
Re: How to drop a column family (Cassandra 6.1)
1. nodetool flush 2. stop server 3. delete all files (data, index, filter) 4. start server note that this will delete the data, not the CF definition (not like "drop table" in sql-ish). system_drop_column_family will drop the CF definition truncate (available from 0.7) will delete the data, which is equivalent to the steps 1-4 above On Mon, May 31, 2010 at 12:33 PM, xavier manach wrote: > Hi > > In the API I see : system_drop_column_family *Requires Cassandra 0.7 > * > How to drop a column family with Cassandra 6.1 ? > * > Delete *the files MYCOLUMNFAMILY-XXX-Data.db, > MYCOLUMNFAMILY-XXX-Filter.db , MYCOLUMNFAMILY-XXX-Index.db > is a good ways ? > > Can I recreate this columnfamily after this delete ? > > > Thanks. > >
Re: Creating a Cassandra cluster under Windows XP
try this: ** * 192.168.80.12* * **192.168.80.234* * *
How can I get a right 16-bytes UUIDs TException: UUIDs must be exactly 16 bytes Error:
When I using Thrift with PHP-api batch_insert , It Throw Exception " TException: UUIDs must be exactly 16 bytes Error:" PHP Codes: $column_data[0] = new cassandra_Column(); $column_data[0]->name = 'status_id'; $column_data[0]->value = '1'; $column_data[0]->timestamp = time(); $super_column = new cassandra_SuperColumn(); $b = new SimpleCassieUuid(); $super_column->name = $b->__toString(); //15a599c0-6c98-11df-8f93-e9f1c3a0e6e8 $super_column->columns = $column_data; $c_or_sc = new cassandra_ColumnOrSuperColumn(); $c_or_sc->super_column = $super_column; $mutation['StatusRelationships'] = array($c_or_sc); $status=$client->batch_insert($keyspace, $keyUserId, $mutation, $consistency_level); How can I get a right 16-bytes UUIDs? 15a599c0-6c98-11df-8f93-e9f1c3a0e6e8 Thanks! -- 执著而努力着 david.liu
searching keys of the form substring*
Hi folks,
I want to fetch all those records from my column family such that the key
starts with a specified string...
e.g. Suppose I have a CF keyed on full names(first name + last name) of
persons...
now I want to fetch all those records whose first name is 'John'
Right now, I am using OPP and KeyRange in the following way:
KeyRange keyRange = new KeyRange();
keyRange.setStart_key("John");
keyRange.setEnd_key("Joho");
but this is sort of hard coding can anyone suggest a better way to
achieve this?
I would be really grateful... thank you.
HintedHandoffEnabled
In 0.6.2 I disabled hinted handoff, however tpstats and cfstats report seems odd. On all servers in the cluster I have: false tpstats reports 5 completed handoffs. $ nodetool -h cass25 -p 9004 tpstats Pool NameActive Pending Completed FILEUTILS-DELETE-POOL 0 0 2 STREAM-STAGE 0 0 0 RESPONSE-STAGE0 05903099 ROW-READ-STAGE0 0 669093 LB-OPERATIONS 0 0 0 MESSAGE-DESERIALIZER-POOL 1 06595504 GMFD 0 0 35947 LB-TARGET 0 0 0 CONSISTENCY-MANAGER 0 0 669095 ROW-MUTATION-STAGE0 0 644360 MESSAGE-STREAMING-POOL0 0 0 LOAD-BALANCER-STAGE 0 0 0 FLUSH-SORTER-POOL 0 0 0 MEMTABLE-POST-FLUSHER 0 0 7 FLUSH-WRITER-POOL 0 0 7 AE-SERVICE-STAGE 0 0 1 HINTED-HANDOFF-POOL 0 0 5 In data/system/* there are only LocationInfo files, so looks like hinted handoff is indeed disabled and cfstats does indicate there are 0 bytes, however it also indicates of 32 reads which I didn't expect (cluster has been up for a few hours). $ nodetool -h cass25 -p 9004 tpstats ... Column Family: HintsColumnFamily SSTable count: 0 Space used (live): 0 Space used (total): 0 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 32 Read Latency: 0.062 ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 1 Key cache size: 0 Key cache hit rate: NaN Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Any idea why is this happening?
Re: searching keys of the form substring*
Hi Sagar
You can use variable substitution.
___
Vineet Daniel
___
Let your email find you
On Mon, May 31, 2010 at 3:44 PM, Sagar Agrawal wrote:
> Hi folks,
>
> I want to fetch all those records from my column family such that the key
> starts with a specified string...
>
> e.g. Suppose I have a CF keyed on full names(first name + last name) of
> persons...
> now I want to fetch all those records whose first name is 'John'
>
> Right now, I am using OPP and KeyRange in the following way:
>
> KeyRange keyRange = new KeyRange();
> keyRange.setStart_key("John");
> keyRange.setEnd_key("Joho");
>
> but this is sort of hard coding can anyone suggest a better way to
> achieve this?
>
> I would be really grateful... thank you.
>
>
>
Re: Ruby get_range with :reversed?
On Sat, May 29, 2010 at 11:55 PM, Jonathan Ellis wrote: > Cassandra supports slicing columns within a row in reversed order, but > not iterating rows in reverse. > Thanks Jonathan
Re: Creating a Cassandra cluster under Windows XP
Are required ports (http://wiki.apache.org/cassandra/FAQ#ports) opened at Windows Firewall? David Boxenhorn wrote: > Just to clarify things: I'm trying to get my first cluster up and > running and it's not working. >
nodetool cleanup isn't cleaning up?
I hope I understand nodetool cleanup correctly - it should clean up all data that does not (currently) belong to this node. If so, I think it might not be working correctly. Look at nodes 192.168.252.124 and 192.168.252.99 below 192.168.252.99Up 279.35 MB 3544607988759775661076818827414252202 |<--| 192.168.252.124Up 167.23 MB 56713727820156410577229101238628035242 | ^ 192.168.252.125Up 82.91 MB 85070591730234615865843651857942052863 v | 192.168.254.57Up 366.6 MB 113427455640312821154458202477256070485| ^ 192.168.254.58Up 88.44 MB 141784319550391026443072753096570088106v | 192.168.254.59Up 88.45 MB 170141183460469231731687303715884105727|-->| I wanted 124 to take all the load from 99. So I issued a move command. $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 This command tells 99 to take the space b/w (56713727820156410577229101238628035242, 56713727820156410577229101238628035243] which is basically just one item in the token space, almost nothing... I wanted it to be very slim (just playing around). So, next I get this: 192.168.252.124Up 803.33 MB 56713727820156410577229101238628035242 |<--| 192.168.252.99Up 352.85 MB 56713727820156410577229101238628035243 | ^ 192.168.252.125Up 134.24 MB 85070591730234615865843651857942052863 v | 192.168.254.57Up 676.41 MB 113427455640312821154458202477256070485| ^ 192.168.254.58Up 99.74 MB 141784319550391026443072753096570088106v | 192.168.254.59Up 99.94 MB 170141183460469231731687303715884105727|-->| The tokens are correct, but it seems that 99 still has a lot of data. Why? OK, that might be b/c it didn't delete its moved data. So next I issued a nodetool cleanup, which should have taken care of that. Only that it didn't, the node 99 still has 352 MB of data. Why? So, you know what, I waited for 1h. Still no good, data wasn't cleaned up. I restarted the server. Still, data wasn't cleaned up... I issued a cleanup again... still no good... what's up with this node?
Re: nodetool cleanup isn't cleaning up?
Hello! You likely need wait for GCGraceSeconds seconds or modify this param. http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html === Thus, a delete operation can't just wipe out all traces of the data being removed immediately: if we did, and a replica did not receive the delete operation, when it becomes available again it will treat the replicas that did receive the delete as having missed a write update, and repair them! So, instead of wiping out data on delete, Cassandra replaces it with a special value called a tombstone. The tombstone can then be propagated to replicas that missed the initial remove request. ... Here, we defined a constant, GCGraceSeconds, and had each node track tombstone age locally. Once it has aged past the constant, it can be GC'd. === On 31.05.2010 16:23, Ran Tavory wrote: I hope I understand nodetool cleanup correctly - it should clean up all data that does not (currently) belong to this node. If so, I think it might not be working correctly. Look at nodes 192.168.252.124 and 192.168.252.99 below 192.168.252.99Up 279.35 MB 3544607988759775661076818827414252202 |<--| 192.168.252.124Up 167.23 MB 56713727820156410577229101238628035242 | ^ 192.168.252.125Up 82.91 MB 85070591730234615865843651857942052863 v | 192.168.254.57Up 366.6 MB 113427455640312821154458202477256070485| ^ 192.168.254.58Up 88.44 MB 141784319550391026443072753096570088106v | 192.168.254.59Up 88.45 MB 170141183460469231731687303715884105727|-->| I wanted 124 to take all the load from 99. So I issued a move command. $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 This command tells 99 to take the space b/w (56713727820156410577229101238628035242, 56713727820156410577229101238628035243] which is basically just one item in the token space, almost nothing... I wanted it to be very slim (just playing around). So, next I get this: 192.168.252.124Up 803.33 MB 56713727820156410577229101238628035242 |<--| 192.168.252.99Up 352.85 MB 56713727820156410577229101238628035243 | ^ 192.168.252.125Up 134.24 MB 85070591730234615865843651857942052863 v | 192.168.254.57Up 676.41 MB 113427455640312821154458202477256070485| ^ 192.168.254.58Up 99.74 MB 141784319550391026443072753096570088106v | 192.168.254.59Up 99.94 MB 170141183460469231731687303715884105727|-->| The tokens are correct, but it seems that 99 still has a lot of data. Why? OK, that might be b/c it didn't delete its moved data. So next I issued a nodetool cleanup, which should have taken care of that. Only that it didn't, the node 99 still has 352 MB of data. Why? So, you know what, I waited for 1h. Still no good, data wasn't cleaned up. I restarted the server. Still, data wasn't cleaned up... I issued a cleanup again... still no good... what's up with this node? -- Best regards, Maximmailto:maxi...@trackstudio.com LinkedIn Profile: http://www.linkedin.com/in/maximkr Google Talk/Jabber: maxi...@gmail.com ICQ number: 307863079 Skype Chat: maxim.kramarenko Yahoo! Messenger: maxim_kramarenko
Re: Creating a Cassandra cluster under Windows XP
Thank you! That's the problem!!! On Mon, May 31, 2010 at 2:57 PM, Mishail wrote: > Are required ports (http://wiki.apache.org/cassandra/FAQ#ports) opened > at Windows Firewall? > > David Boxenhorn wrote: > > Just to clarify things: I'm trying to get my first cluster up and > > running and it's not working. > > > > >
Re: Creating a Cassandra cluster under Windows XP
Thanks to huajun qi too. On Mon, May 31, 2010 at 3:33 PM, David Boxenhorn wrote: > Thank you! That's the problem!!! > > On Mon, May 31, 2010 at 2:57 PM, Mishail wrote: > >> Are required ports (http://wiki.apache.org/cassandra/FAQ#ports) opened >> at Windows Firewall? >> >> David Boxenhorn wrote: >> > Just to clarify things: I'm trying to get my first cluster up and >> > running and it's not working. >> > >> >> >> >
Re: nodetool cleanup isn't cleaning up?
Do you think it's the tombstones that take up the disk space? Shouldn't the tombstones be moved along with the data? On Mon, May 31, 2010 at 3:29 PM, Maxim Kramarenko wrote: > Hello! > > You likely need wait for GCGraceSeconds seconds or modify this param. > > http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html > === > Thus, a delete operation can't just wipe out all traces of the data being > removed immediately: if we did, and a replica did not receive the delete > operation, when it becomes available again it will treat the replicas that > did receive the delete as having missed a write update, and repair them! So, > instead of wiping out data on delete, Cassandra replaces it with a special > value called a tombstone. The tombstone can then be propagated to replicas > that missed the initial remove request. > ... > Here, we defined a constant, GCGraceSeconds, and had each node track > tombstone age locally. Once it has aged past the constant, it can be GC'd. > === > > > > On 31.05.2010 16:23, Ran Tavory wrote: > >> I hope I understand nodetool cleanup correctly - it should clean up all >> data that does not (currently) belong to this node. If so, I think it >> might not be working correctly. >> >> Look at nodes 192.168.252.124 and 192.168.252.99 below >> >> 192.168.252.99Up 279.35 MB >> 3544607988759775661076818827414252202 |<--| >> 192.168.252.124Up 167.23 MB >> 56713727820156410577229101238628035242 | ^ >> 192.168.252.125Up 82.91 MB >> 85070591730234615865843651857942052863 v | >> 192.168.254.57Up 366.6 MB >> 113427455640312821154458202477256070485| ^ >> 192.168.254.58Up 88.44 MB >> 141784319550391026443072753096570088106v | >> 192.168.254.59Up 88.45 MB >> 170141183460469231731687303715884105727|-->| >> >> I wanted 124 to take all the load from 99. So I issued a move command. >> >> $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 >> >> This command tells 99 to take the space b/w >> (56713727820156410577229101238628035242, >> 56713727820156410577229101238628035243] >> which is basically just one item in the token space, almost nothing... I >> wanted it to be very slim (just playing around). >> >> So, next I get this: >> >> 192.168.252.124Up 803.33 MB >> 56713727820156410577229101238628035242 |<--| >> 192.168.252.99Up 352.85 MB >> 56713727820156410577229101238628035243 | ^ >> 192.168.252.125Up 134.24 MB >> 85070591730234615865843651857942052863 v | >> 192.168.254.57Up 676.41 MB >> 113427455640312821154458202477256070485| ^ >> 192.168.254.58Up 99.74 MB >> 141784319550391026443072753096570088106v | >> 192.168.254.59Up 99.94 MB >> 170141183460469231731687303715884105727|-->| >> >> The tokens are correct, but it seems that 99 still has a lot of data. >> Why? OK, that might be b/c it didn't delete its moved data. >> So next I issued a nodetool cleanup, which should have taken care of >> that. Only that it didn't, the node 99 still has 352 MB of data. Why? >> So, you know what, I waited for 1h. Still no good, data wasn't cleaned up. >> I restarted the server. Still, data wasn't cleaned up... I issued a >> cleanup again... still no good... what's up with this node? >> >> >> > -- > Best regards, > Maximmailto:maxi...@trackstudio.com > > LinkedIn Profile: http://www.linkedin.com/in/maximkr > Google Talk/Jabber: maxi...@gmail.com > ICQ number: 307863079 > Skype Chat: maxim.kramarenko > Yahoo! Messenger: maxim_kramarenko >
Re: nodetool cleanup isn't cleaning up?
Hello! I think (but not sure, please correct me if required), that after you change token, nodes just receive new data, but don't immediate deletes old one. It seems like "clean" will mark them as tombstone and it will be deleted when you run "compact" after GCGraceSeconds seconds. On 31.05.2010 17:00, Ran Tavory wrote: Do you think it's the tombstones that take up the disk space? Shouldn't the tombstones be moved along with the data? On Mon, May 31, 2010 at 3:29 PM, Maxim Kramarenko mailto:maxi...@trackstudio.com>> wrote: Hello! You likely need wait for GCGraceSeconds seconds or modify this param. http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html === Thus, a delete operation can't just wipe out all traces of the data being removed immediately: if we did, and a replica did not receive the delete operation, when it becomes available again it will treat the replicas that did receive the delete as having missed a write update, and repair them! So, instead of wiping out data on delete, Cassandra replaces it with a special value called a tombstone. The tombstone can then be propagated to replicas that missed the initial remove request. ... Here, we defined a constant, GCGraceSeconds, and had each node track tombstone age locally. Once it has aged past the constant, it can be GC'd. ===
Re: How to drop a column family (Cassandra 6.1)
Perfect :) Thanks Ran. 2010/5/31 Ran Tavory > 1. nodetool flush > 2. stop server > 3. delete all files (data, index, filter) > 4. start server > > note that this will delete the data, not the CF definition (not like "drop > table" in sql-ish). > > system_drop_column_family will drop the CF definition > > truncate (available from 0.7) will delete the data, which is equivalent to > the steps 1-4 above > > > On Mon, May 31, 2010 at 12:33 PM, xavier manach wrote: > >> Hi >> >> In the API I see : system_drop_column_family *Requires Cassandra 0.7 >> * >> How to drop a column family with Cassandra 6.1 ? >> * >> Delete *the files MYCOLUMNFAMILY-XXX-Data.db, >> MYCOLUMNFAMILY-XXX-Filter.db , MYCOLUMNFAMILY-XXX-Index.db >> is a good ways ? >> >> Can I recreate this columnfamily after this delete ? >> >> >> Thanks. >> >> >
cluster throwing errors when new or existing node joins
Hi I have a setup of 4 nodes, whenever I am restarting any of the nodes, even after deleting the data directories and commit log I get the following error ERROR 18:46:41,296 Fatal exception in thread Thread[COMMIT-LOG-WRITER,5,main] java.lang.RuntimeException: java.lang.NullPointerException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.lang.Thread.run(Thread.java:636) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.Table$TableMetadata.getColumnFamilyId(Table.java:131) at org.apache.cassandra.db.Table.getColumnFamilyId(Table.java:364) at org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:103) at org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:475) at org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 1 more ERROR 18:46:41,297 Error in ThreadPoolExecutor java.lang.NullPointerException at org.apache.cassandra.db.Table.apply(Table.java:407) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR 18:46:41,299 Fatal exception in thread Thread[ROW-MUTATION-STAGE:5,5,main] java.lang.NullPointerException at org.apache.cassandra.db.Table.apply(Table.java:407) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR 18:46:51,309 Error in ThreadPoolExecutor java.lang.NullPointerException at org.apache.cassandra.db.Table.apply(Table.java:407) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR 18:46:51,310 Fatal exception in thread Thread[ROW-MUTATION-STAGE:6,5,main] Kindly suggest what can be the reason for this error. ___ VD ___ Let your email find you
Administration Memory for Noobs. (GC for ConcurrentMarkSweep ?)
Hi. I search informations for basic tunning of memory in Cassandra. My situation : I started to test larges imports of data in Cassandra 6.1. My first import worked fine : 100 Millions row in 2 hours ~ around 1 insert row by seconds My second is slower with the same script in another column family : ~ around 500 insert row by seconds... I didn't understand why I have a lot of GC for ConcurrentMarkSweep. [ GC for ConcurrentMarkSweep: 3437 ms, 104971488 reclaimed leaving 986519328 used; max is 1211170816. ] ( The max did'n't move, what is this value 1211170816 ? ) I think the GC process appear when the insert is slow. The inserts didn't work when the GC works ? My machine has 66M of RAM, and the processor java only use around 1.8 % How Can I optimise the use of memory ? There is a the guideline for best performances ? Thanks.
Can't get data after building cluster
I am trying to get a cluster up and working for the first time. I got one server up and running, with lots of data on it, which I can see with the CLI. I added my 2nd server, they seem to recognize each other. Now I can't see my data with the CLI. I do a get and it returns null. The data files seem to be intact. What happened??? How can I fix it?
sorting by column value
Is it possible to have columns in a super column sorted by value rather than name? I assume not but I thought I ask anyway. What I would love to do is something along the lines of /user//country/DE += 1 and then get the sorted result of "/user//country" cheers -- Torsten
Re: Can't get data after building cluster
OK. Got it working. I had some data in the 2nd server from previous failed attempts at hooking up to the cluster. When I deleted that data and tried again, it said "bootstrapping" and my 1st server started working again. On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn wrote: > I am trying to get a cluster up and working for the first time. > > I got one server up and running, with lots of data on it, which I can see > with the CLI. > > I added my 2nd server, they seem to recognize each other. > > Now I can't see my data with the CLI. I do a get and it returns null. The > data files seem to be intact. > > What happened??? How can I fix it? > >
Re: Can't get data after building cluster
So this means that I can take my entire cluster off line if I make a mistake adding a new server??? Yikes! On Mon, May 31, 2010 at 6:41 PM, David Boxenhorn wrote: > OK. Got it working. > > I had some data in the 2nd server from previous failed attempts at hooking > up to the cluster. When I deleted that data and tried again, it said > "bootstrapping" and my 1st server started working again. > > On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn wrote: > >> I am trying to get a cluster up and working for the first time. >> >> I got one server up and running, with lots of data on it, which I can see >> with the CLI. >> >> I added my 2nd server, they seem to recognize each other. >> >> Now I can't see my data with the CLI. I do a get and it returns null. The >> data files seem to be intact. >> >> What happened??? How can I fix it? >> >> >
RE: Inconsistency even after nodetool repair?
Did you watch in the logs to confirm that repair had actually finished? The `nodetool repair` call is not blocking before 0.6.3 (unreleased): see https://issues.apache.org/jira/browse/CASSANDRA-1090 -Original Message- From: "James Golick" Sent: Sunday, May 30, 2010 3:43am To: cassandra-u...@incubator.apache.org Subject: Inconsistency even after nodetool repair? I was getting a ton of these in my logs: INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line 499) DigestMismatchException: Mismatch for key 142667/everything (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17) The nodes were all getting really overloaded, presumably from the read-repair, since everything else looked healthy. So, I ran nodetool repair on both of our nodes, but even after running the repairs, I'm still seeing the same thing in the logs and high load on the nodes. Does that sound right?
Re: Error during startup
What version of Cassandra was this? On Sun, May 30, 2010 at 8:49 AM, David Boxenhorn wrote: > I deleted the system/LocationInfo files, and now everything works. > > Yay! (...what happened?) > > On Sun, May 30, 2010 at 4:18 PM, David Boxenhorn wrote: >> >> I'm getting an "Expected both token and generation columns; found >> ColumnFamily" error during startup can anyone tell me what it is? Details >> below. >> >> Starting Cassandra Server >> Listening for transport dt_socket at address: >> INFO 16:14:33,459 Auto DiskAccessMode determined to be standard >> INFO 16:14:33,615 Sampling index for >> C:\var\lib\cassandra\data\system\LocationInfo-1-Data.db >> INFO 16:14:33,631 Removing orphan >> C:\var\lib\cassandra\data\Lookin2\Users-tmp-27-Index.db >> INFO 16:14:33,631 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db >> INFO 16:14:33,662 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-18-Data.db >> INFO 16:14:33,818 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db >> INFO 16:14:33,850 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db >> INFO 16:14:33,865 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db >> INFO 16:14:33,881 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-580-Data.db >> INFO 16:14:33,896 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-672-Data.db >> INFO 16:14:33,912 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-681-Data.db >> INFO 16:14:33,912 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-691-Data.db >> INFO 16:14:33,928 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-696-Data.db >> INFO 16:14:33,943 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Attractions-17-Data.db >> INFO 16:14:34,006 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-5-Data.db >> INFO 16:14:34,006 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-6-Data.db >> INFO 16:14:34,021 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-29-Data.db >> INFO 16:14:34,350 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-51-Data.db >> INFO 16:14:34,693 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-72-Data.db >> INFO 16:14:35,021 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-77-Data.db >> INFO 16:14:35,225 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-78-Data.db >> INFO 16:14:35,350 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-79-Data.db >> INFO 16:14:35,459 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-80-Data.db >> INFO 16:14:35,459 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-1-Data.db >> INFO 16:14:35,475 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-2-Data.db >> INFO 16:14:35,475 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-30-Data.db >> INFO 16:14:35,631 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-35-Data.db >> INFO 16:14:35,771 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-40-Data.db >> INFO 16:14:35,959 Compacting >> [org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db')] >> ERROR 16:14:35,975 Exception encountered during startup. >> java.lang.RuntimeException: Expected both token and generation columns; >> found ColumnFamily(LocationInfo [Generation:false:4...@4,]) >> at >> org.apache.cassandra.db.SystemTable.initMetadata(SystemTable.java:159) >> at >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:305) >> at >> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) >> at >> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) >> Exception encountered during startup. >> > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Algorithm for distributing key of Cassandra
Doesn't ring a bell. Maybe if you included the link to which you refer? On Sun, May 30, 2010 at 10:37 AM, JKnight JKnight wrote: > Dear all, > At the blog of JONATHAN ELLIS, he said that Cassandra use a hack (trick) for > distributing key for each node. But the link point to that document is not > available. > Could you tell me more about that algorithm? > Thank > > -- > Best regards, > JKnight > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Start key must sort before (or equal to) finish key in your partitioner
OPP uses lexical ordering on the keys, which isn't going to be the same as the natural order for a time-based uuid. On Mon, May 31, 2010 at 2:57 AM, Leslie Viljoen wrote: > Hello! > > I am using an OrderPreservingPartitioner and sequential UUID's as > keys. When I use get_range I frequently get an error indicating that > my start key is not before or equal to my finish key, which seems > impossible. The code to request slices looks like this: > > sometime = Time.now > a = UUIDTools::UUID.timestamp_create(sometime) > b = UUIDTools::UUID.timestamp_create(sometime + 1) > slice = @cassandra.get_range(:ReportData, :start => a.to_s, :finish => b.to_s) > puts "SORT ERROR!" if a >= b # this never displays > > > > This is the error I get: > > /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/../vendor/gen-rb/cassandra.rb:152:in > `recv_get_range_slices': start key must sort before (or equal to) > finish key in your partitioner! > (CassandraThrift::InvalidRequestException) > from > /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/../vendor/gen-rb/cassandra.rb:142:in > `get_range_slices' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.4.0/lib/thrift_client.rb:140:in > `send' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.4.0/lib/thrift_client.rb:140:in > `proxy' > from (eval):1:in `get_range_slices' > from > /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/cassandra/protocol.rb:78:in > `_get_range' > from > /usr/lib/ruby/gems/1.8/gems/cassandra-0.8.2/lib/cassandra/cassandra.rb:230:in > `get_range' > from ./read.rb:29:in `get_range' > from ./read.rb:62 > from ./read.rb:58:in `loop' > from ./read.rb:58 > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: HintedHandoffEnabled
disabling hinted handoff will prevent creation of new hints, but it will continue to attempt to deliver existing ones unless you manually delete the hint files in the system/ dir. On Mon, May 31, 2010 at 6:04 AM, Ran Tavory wrote: > In 0.6.2 I disabled hinted handoff, however tpstats and cfstats report seems > odd. > On all servers in the cluster I have: > false > tpstats reports 5 completed handoffs. > $ nodetool -h cass25 -p 9004 tpstats > Pool Name Active Pending Completed > FILEUTILS-DELETE-POOL 0 0 2 > STREAM-STAGE 0 0 0 > RESPONSE-STAGE 0 0 5903099 > ROW-READ-STAGE 0 0 669093 > LB-OPERATIONS 0 0 0 > MESSAGE-DESERIALIZER-POOL 1 0 6595504 > GMFD 0 0 35947 > LB-TARGET 0 0 0 > CONSISTENCY-MANAGER 0 0 669095 > ROW-MUTATION-STAGE 0 0 644360 > MESSAGE-STREAMING-POOL 0 0 0 > LOAD-BALANCER-STAGE 0 0 0 > FLUSH-SORTER-POOL 0 0 0 > MEMTABLE-POST-FLUSHER 0 0 7 > FLUSH-WRITER-POOL 0 0 7 > AE-SERVICE-STAGE 0 0 1 > HINTED-HANDOFF-POOL 0 0 5 > In data/system/* there are only LocationInfo files, so looks like hinted > handoff is indeed disabled and cfstats does indicate there are 0 bytes, > however it also indicates of 32 reads which I didn't expect (cluster has > been up for a few hours). > $ nodetool -h cass25 -p 9004 tpstats > ... > Column Family: HintsColumnFamily > SSTable count: 0 > Space used (live): 0 > Space used (total): 0 > Memtable Columns Count: 0 > Memtable Data Size: 0 > Memtable Switch Count: 0 > Read Count: 32 > Read Latency: 0.062 ms. > Write Count: 0 > Write Latency: NaN ms. > Pending Tasks: 0 > Key cache capacity: 1 > Key cache size: 0 > Key cache hit rate: NaN > Row cache: disabled > Compacted row minimum size: 0 > Compacted row maximum size: 0 > Compacted row mean size: 0 > Any idea why is this happening? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: HintedHandoffEnabled
I deleted all files in the system dir, including hints before starting the servers. It's not a big deal. but can it be something else? On Mon, May 31, 2010 at 9:53 PM, Jonathan Ellis wrote: > disabling hinted handoff will prevent creation of new hints, but it > will continue to attempt to deliver existing ones unless you manually > delete the hint files in the system/ dir. > > On Mon, May 31, 2010 at 6:04 AM, Ran Tavory wrote: > > In 0.6.2 I disabled hinted handoff, however tpstats and cfstats report > seems > > odd. > > On all servers in the cluster I have: > > false > > tpstats reports 5 completed handoffs. > > $ nodetool -h cass25 -p 9004 tpstats > > Pool NameActive Pending Completed > > FILEUTILS-DELETE-POOL 0 0 2 > > STREAM-STAGE 0 0 0 > > RESPONSE-STAGE0 05903099 > > ROW-READ-STAGE0 0 669093 > > LB-OPERATIONS 0 0 0 > > MESSAGE-DESERIALIZER-POOL 1 06595504 > > GMFD 0 0 35947 > > LB-TARGET 0 0 0 > > CONSISTENCY-MANAGER 0 0 669095 > > ROW-MUTATION-STAGE0 0 644360 > > MESSAGE-STREAMING-POOL0 0 0 > > LOAD-BALANCER-STAGE 0 0 0 > > FLUSH-SORTER-POOL 0 0 0 > > MEMTABLE-POST-FLUSHER 0 0 7 > > FLUSH-WRITER-POOL 0 0 7 > > AE-SERVICE-STAGE 0 0 1 > > HINTED-HANDOFF-POOL 0 0 5 > > In data/system/* there are only LocationInfo files, so looks like hinted > > handoff is indeed disabled and cfstats does indicate there are 0 bytes, > > however it also indicates of 32 reads which I didn't expect (cluster has > > been up for a few hours). > > $ nodetool -h cass25 -p 9004 tpstats > > ... > > Column Family: HintsColumnFamily > > SSTable count: 0 > > Space used (live): 0 > > Space used (total): 0 > > Memtable Columns Count: 0 > > Memtable Data Size: 0 > > Memtable Switch Count: 0 > > Read Count: 32 > > Read Latency: 0.062 ms. > > Write Count: 0 > > Write Latency: NaN ms. > > Pending Tasks: 0 > > Key cache capacity: 1 > > Key cache size: 0 > > Key cache hit rate: NaN > > Row cache: disabled > > Compacted row minimum size: 0 > > Compacted row maximum size: 0 > > Compacted row mean size: 0 > > Any idea why is this happening? > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: nodetool cleanup isn't cleaning up?
you have replication factor > 1 ? On Mon, May 31, 2010 at 7:23 AM, Ran Tavory wrote: > I hope I understand nodetool cleanup correctly - it should clean up all data > that does not (currently) belong to this node. If so, I think it might not > be working correctly. > Look at nodes 192.168.252.124 and 192.168.252.99 below > 192.168.252.99Up 279.35 MB 3544607988759775661076818827414252202 > |<--| > 192.168.252.124Up 167.23 MB > 56713727820156410577229101238628035242 | ^ > 192.168.252.125Up 82.91 MB > 85070591730234615865843651857942052863 v | > 192.168.254.57Up 366.6 MB > 113427455640312821154458202477256070485 | ^ > 192.168.254.58Up 88.44 MB > 141784319550391026443072753096570088106 v | > 192.168.254.59Up 88.45 MB > 170141183460469231731687303715884105727 |-->| > I wanted 124 to take all the load from 99. So I issued a move command. > $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 > > This command tells 99 to take the space b/w > (56713727820156410577229101238628035242, 56713727820156410577229101238628035243] > which is basically just one item in the token space, almost nothing... I > wanted it to be very slim (just playing around). > So, next I get this: > 192.168.252.124Up 803.33 MB > 56713727820156410577229101238628035242 |<--| > 192.168.252.99Up 352.85 MB > 56713727820156410577229101238628035243 | ^ > 192.168.252.125Up 134.24 MB > 85070591730234615865843651857942052863 v | > 192.168.254.57Up 676.41 MB > 113427455640312821154458202477256070485 | ^ > 192.168.254.58Up 99.74 MB > 141784319550391026443072753096570088106 v | > 192.168.254.59Up 99.94 MB > 170141183460469231731687303715884105727 |-->| > The tokens are correct, but it seems that 99 still has a lot of data. Why? > OK, that might be b/c it didn't delete its moved data. > So next I issued a nodetool cleanup, which should have taken care of that. > Only that it didn't, the node 99 still has 352 MB of data. Why? > So, you know what, I waited for 1h. Still no good, data wasn't cleaned up. > I restarted the server. Still, data wasn't cleaned up... I issued a cleanup > again... still no good... what's up with this node? > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: sorting by column value
I'm afraid not. (We have to sort by name to allow fetching individual columns from large rows w/o pulling the whole row into memory.) On Mon, May 31, 2010 at 9:47 AM, Torsten Curdt wrote: > Is it possible to have columns in a super column sorted by value > rather than name? > I assume not but I thought I ask anyway. > > What I would love to do is something along the lines of > > /user//country/DE += 1 > > and then get the sorted result of "/user//country" > > cheers > -- > Torsten > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Can't get data after building cluster
No. On Mon, May 31, 2010 at 10:47 AM, David Boxenhorn wrote: > So this means that I can take my entire cluster off line if I make a > mistake adding a new server??? Yikes! > > On Mon, May 31, 2010 at 6:41 PM, David Boxenhorn wrote: >> >> OK. Got it working. >> >> I had some data in the 2nd server from previous failed attempts at hooking >> up to the cluster. When I deleted that data and tried again, it said >> "bootstrapping" and my 1st server started working again. >> >> On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn >> wrote: >>> >>> I am trying to get a cluster up and working for the first time. >>> >>> I got one server up and running, with lots of data on it, which I can see >>> with the CLI. >>> >>> I added my 2nd server, they seem to recognize each other. >>> >>> Now I can't see my data with the CLI. I do a get and it returns null. The >>> data files seem to be intact. >>> >>> What happened??? How can I fix it? >>> >> > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: HintedHandoffEnabled
Ah... no, that's just its "is there anything to deliver" check. Basically a no-op if there are no data files left. On Mon, May 31, 2010 at 2:05 PM, Ran Tavory wrote: > I deleted all files in the system dir, including hints before starting the > servers. > It's not a big deal. but can it be something else? > > On Mon, May 31, 2010 at 9:53 PM, Jonathan Ellis wrote: >> >> disabling hinted handoff will prevent creation of new hints, but it >> will continue to attempt to deliver existing ones unless you manually >> delete the hint files in the system/ dir. >> >> On Mon, May 31, 2010 at 6:04 AM, Ran Tavory wrote: >> > In 0.6.2 I disabled hinted handoff, however tpstats and cfstats report >> > seems >> > odd. >> > On all servers in the cluster I have: >> > false >> > tpstats reports 5 completed handoffs. >> > $ nodetool -h cass25 -p 9004 tpstats >> > Pool Name Active Pending Completed >> > FILEUTILS-DELETE-POOL 0 0 2 >> > STREAM-STAGE 0 0 0 >> > RESPONSE-STAGE 0 0 5903099 >> > ROW-READ-STAGE 0 0 669093 >> > LB-OPERATIONS 0 0 0 >> > MESSAGE-DESERIALIZER-POOL 1 0 6595504 >> > GMFD 0 0 35947 >> > LB-TARGET 0 0 0 >> > CONSISTENCY-MANAGER 0 0 669095 >> > ROW-MUTATION-STAGE 0 0 644360 >> > MESSAGE-STREAMING-POOL 0 0 0 >> > LOAD-BALANCER-STAGE 0 0 0 >> > FLUSH-SORTER-POOL 0 0 0 >> > MEMTABLE-POST-FLUSHER 0 0 7 >> > FLUSH-WRITER-POOL 0 0 7 >> > AE-SERVICE-STAGE 0 0 1 >> > HINTED-HANDOFF-POOL 0 0 5 >> > In data/system/* there are only LocationInfo files, so looks like hinted >> > handoff is indeed disabled and cfstats does indicate there are 0 bytes, >> > however it also indicates of 32 reads which I didn't expect (cluster has >> > been up for a few hours). >> > $ nodetool -h cass25 -p 9004 tpstats >> > ... >> > Column Family: HintsColumnFamily >> > SSTable count: 0 >> > Space used (live): 0 >> > Space used (total): 0 >> > Memtable Columns Count: 0 >> > Memtable Data Size: 0 >> > Memtable Switch Count: 0 >> > Read Count: 32 >> > Read Latency: 0.062 ms. >> > Write Count: 0 >> > Write Latency: NaN ms. >> > Pending Tasks: 0 >> > Key cache capacity: 1 >> > Key cache size: 0 >> > Key cache hit rate: NaN >> > Row cache: disabled >> > Compacted row minimum size: 0 >> > Compacted row maximum size: 0 >> > Compacted row mean size: 0 >> > Any idea why is this happening? >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: nodetool cleanup isn't cleaning up?
yes, replication factor = 2 On Mon, May 31, 2010 at 10:07 PM, Jonathan Ellis wrote: > you have replication factor > 1 ? > > On Mon, May 31, 2010 at 7:23 AM, Ran Tavory wrote: > > I hope I understand nodetool cleanup correctly - it should clean up all > data > > that does not (currently) belong to this node. If so, I think it might > not > > be working correctly. > > Look at nodes 192.168.252.124 and 192.168.252.99 below > > 192.168.252.99Up 279.35 MB > 3544607988759775661076818827414252202 > > |<--| > > 192.168.252.124Up 167.23 MB > > 56713727820156410577229101238628035242 | ^ > > 192.168.252.125Up 82.91 MB > > 85070591730234615865843651857942052863 v | > > 192.168.254.57Up 366.6 MB > > 113427455640312821154458202477256070485| ^ > > 192.168.254.58Up 88.44 MB > > 141784319550391026443072753096570088106v | > > 192.168.254.59Up 88.45 MB > > 170141183460469231731687303715884105727|-->| > > I wanted 124 to take all the load from 99. So I issued a move command. > > $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 > > > > This command tells 99 to take the space b/w > > > (56713727820156410577229101238628035242, > 56713727820156410577229101238628035243] > > which is basically just one item in the token space, almost nothing... I > > wanted it to be very slim (just playing around). > > So, next I get this: > > 192.168.252.124Up 803.33 MB > > 56713727820156410577229101238628035242 |<--| > > 192.168.252.99Up 352.85 MB > > 56713727820156410577229101238628035243 | ^ > > 192.168.252.125Up 134.24 MB > > 85070591730234615865843651857942052863 v | > > 192.168.254.57Up 676.41 MB > > 113427455640312821154458202477256070485| ^ > > 192.168.254.58Up 99.74 MB > > 141784319550391026443072753096570088106v | > > 192.168.254.59Up 99.94 MB > > 170141183460469231731687303715884105727|-->| > > The tokens are correct, but it seems that 99 still has a lot of data. > Why? > > OK, that might be b/c it didn't delete its moved data. > > So next I issued a nodetool cleanup, which should have taken care of > that. > > Only that it didn't, the node 99 still has 352 MB of data. Why? > > So, you know what, I waited for 1h. Still no good, data wasn't cleaned > up. > > I restarted the server. Still, data wasn't cleaned up... I issued a > cleanup > > again... still no good... what's up with this node? > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Inconsistency even after nodetool repair?
It may not have actually finished at that point. Though, according to JMX, both compactions of each CF had completed, so I assumed it was done. On Mon, May 31, 2010 at 11:29 AM, Stu Hood wrote: > Did you watch in the logs to confirm that repair had actually finished? The > `nodetool repair` call is not blocking before 0.6.3 (unreleased): see > https://issues.apache.org/jira/browse/CASSANDRA-1090 > > -Original Message- > From: "James Golick" > Sent: Sunday, May 30, 2010 3:43am > To: cassandra-u...@incubator.apache.org > Subject: Inconsistency even after nodetool repair? > > I was getting a ton of these in my logs: > > INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line > 499) DigestMismatchException: Mismatch for key 142667/everything > (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17) > > The nodes were all getting really overloaded, presumably from the > read-repair, since everything else looked healthy. So, I ran nodetool > repair > on both of our nodes, but even after running the repairs, I'm still seeing > the same thing in the logs and high load on the nodes. > > Does that sound right? > > >
Ruby and Cassandra blog post
hey everyone, I've been using Cassandra with Ruby for about a month now and am finding it very helpful. I wrote up a blog post about my experiences, with the goal of adding more Ruby-specific examples to the blogosphere. Hope this is helpful! http://www.subelsky.com/2010/05/real-world-ruby-and-cassandra.html -Mike
Re: Continuously increasing RAM usage
Hmm...heap usage slowly growing over time...the picture doesnt look like the problems we were running into, and I dont think they should be related to mmap-ing your data (since that is not counted in heap mem usage). Kyusik Chung On May 29, 2010, at 10:16 AM, James Golick wrote: > Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks > to be about the same: > > http://skitch.com/jamesgolick/df46f/munin-fetlife.com-cassandra0.fetlife.com-cassandra-memory > > There's only one cache turned on, and it's a row cache, but the sizes of the > rows are identical and it's been full since an hour after I rebooted the > nodes, so it's not that. > > On Thu, May 27, 2010 at 1:00 PM, Kyusik Chung wrote: > > I tried setting the IO mode to standard, but it seemed to be a little slower > > and couldn't get the machine to come back online with adequate read > > performance, so I set it back. I'll have to write a solid cache warming > > script if I'm going to try that again. > > What cache are you talking about? Did you turn on row caching? > > When we turned on row caching, repeat hits to the same rows was fast, of > course, but we didnt (given our data access patterns) see significant > differences compared to mmap-ing the data. And once we hit the limit of our > row cache, out-of-cache hits were pretty costly (dont have hard numbers, but > I recall it being worse than having mmap page in/out). > > Is your client making random reads of more rows than will fit in RAM on your > box? We found that in that scenario, after cassandra has used up all of the > free memory on the box, using mmap was slightly worse than using standard > data access. > > We happened to be lucky that our real world data access is limited to a small > subset of rows in any given time period, so mmap works great for us. I guess > the best thing to do is to try to figure out how to make a cassandra node > only need to service requests for data that can fit into memory in a given > time period. More nodes, a lower replication factor, more memory, I guess... > > Im definitely waiting to hear how things change with 0.6.2. > > Kyusik Chung >
Re: nodetool cleanup isn't cleaning up?
well, there you are then. On Mon, May 31, 2010 at 2:34 PM, Ran Tavory wrote: > yes, replication factor = 2 > > On Mon, May 31, 2010 at 10:07 PM, Jonathan Ellis wrote: >> >> you have replication factor > 1 ? >> >> On Mon, May 31, 2010 at 7:23 AM, Ran Tavory wrote: >> > I hope I understand nodetool cleanup correctly - it should clean up all >> > data >> > that does not (currently) belong to this node. If so, I think it might >> > not >> > be working correctly. >> > Look at nodes 192.168.252.124 and 192.168.252.99 below >> > 192.168.252.99Up 279.35 MB >> > 3544607988759775661076818827414252202 >> > |<--| >> > 192.168.252.124Up 167.23 MB >> > 56713727820156410577229101238628035242 | ^ >> > 192.168.252.125Up 82.91 MB >> > 85070591730234615865843651857942052863 v | >> > 192.168.254.57Up 366.6 MB >> > 113427455640312821154458202477256070485 | ^ >> > 192.168.254.58Up 88.44 MB >> > 141784319550391026443072753096570088106 v | >> > 192.168.254.59Up 88.45 MB >> > 170141183460469231731687303715884105727 |-->| >> > I wanted 124 to take all the load from 99. So I issued a move command. >> > $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 >> > >> > This command tells 99 to take the space b/w >> > >> > (56713727820156410577229101238628035242, 56713727820156410577229101238628035243] >> > which is basically just one item in the token space, almost nothing... I >> > wanted it to be very slim (just playing around). >> > So, next I get this: >> > 192.168.252.124Up 803.33 MB >> > 56713727820156410577229101238628035242 |<--| >> > 192.168.252.99Up 352.85 MB >> > 56713727820156410577229101238628035243 | ^ >> > 192.168.252.125Up 134.24 MB >> > 85070591730234615865843651857942052863 v | >> > 192.168.254.57Up 676.41 MB >> > 113427455640312821154458202477256070485 | ^ >> > 192.168.254.58Up 99.74 MB >> > 141784319550391026443072753096570088106 v | >> > 192.168.254.59Up 99.94 MB >> > 170141183460469231731687303715884105727 |-->| >> > The tokens are correct, but it seems that 99 still has a lot of data. >> > Why? >> > OK, that might be b/c it didn't delete its moved data. >> > So next I issued a nodetool cleanup, which should have taken care of >> > that. >> > Only that it didn't, the node 99 still has 352 MB of data. Why? >> > So, you know what, I waited for 1h. Still no good, data wasn't cleaned >> > up. >> > I restarted the server. Still, data wasn't cleaned up... I issued a >> > cleanup >> > again... still no good... what's up with this node? >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Ruby and Cassandra blog post
Added to http://wiki.apache.org/cassandra/ArticlesAndPresentations. Thanks! On Mon, May 31, 2010 at 3:52 PM, Mike Subelsky wrote: > hey everyone, > > I've been using Cassandra with Ruby for about a month now and am > finding it very helpful. I wrote up a blog post about my experiences, > with the goal of adding more Ruby-specific examples to the > blogosphere. Hope this is helpful! > > http://www.subelsky.com/2010/05/real-world-ruby-and-cassandra.html > > -Mike > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: Inconsistency even after nodetool repair?
The nodes only begin sending the data after the compactions have finished... https://issues.apache.org/jira/browse/CASSANDRA-579 -Original Message- From: "James Golick" Sent: Monday, May 31, 2010 3:33pm To: user@cassandra.apache.org Subject: Re: Inconsistency even after nodetool repair? It may not have actually finished at that point. Though, according to JMX, both compactions of each CF had completed, so I assumed it was done. On Mon, May 31, 2010 at 11:29 AM, Stu Hood wrote: > Did you watch in the logs to confirm that repair had actually finished? The > `nodetool repair` call is not blocking before 0.6.3 (unreleased): see > https://issues.apache.org/jira/browse/CASSANDRA-1090 > > -Original Message- > From: "James Golick" > Sent: Sunday, May 30, 2010 3:43am > To: cassandra-u...@incubator.apache.org > Subject: Inconsistency even after nodetool repair? > > I was getting a ton of these in my logs: > > INFO [pool-1-thread-77] 2010-05-30 03:39:18,263 StorageProxy.java (line > 499) DigestMismatchException: Mismatch for key 142667/everything > (5546784193bbe4a4e066dc7fc142f589 vs 03edb87cd131a614d8256ddfdc50dd17) > > The nodes were all getting really overloaded, presumably from the > read-repair, since everything else looked healthy. So, I ran nodetool > repair > on both of our nodes, but even after running the repairs, I'm still seeing > the same thing in the logs and high load on the nodes. > > Does that sound right? > > >
Re: How to insert a row with a TimeUUIDType column in C++
How can I get 16 bytes timeUUID ?
string(36) "4698cc00-6d2f-11df-8c7f-9f342400a648" TException: UUIDs must be
exactly 16 bytes Error:
On Fri, Apr 23, 2010 at 5:59 PM, Olivier Rosello wrote:
> Here is my test code :
>
> ColumnPath new_col;
> new_col.__isset.column = true; /* this is required! */
> new_col.column_family.assign("Incoming");
> new_col.column.assign("1968ec4a-2a73-11df-9aca-00012e27a270");
> client.insert("MyKeyspace", "somekey", new_col, "Random Value", time(NULL),
> ONE);
>
> I didn't found in the C++ Cassandra/Thrift API how to specify TimeUUID
> bytes (16) to the column name. The ColumnPath type get only a string field
> for the name "column".
>
> With a String like this example shows, the TimeUUID is a 36 chars String
> and this code throws a Exception "UUIDs must be exactly 16 bytes".
>
> I didn't found a function like "client.insert_timeuuid_column" which
> convert the column name to an uint8_t[16]... or anything else which could
> help me.
>
> Cheers,
>
> Olivier
>
>
>
> --
> Olivier
>
>
>
--
执著而努力着 david.liu
Re: Can't get data after building cluster
You say "no", but that is exactly what I just observed. Can I have some more explanation? To recap: I added a server to my cluster. It had some junk in the system/LocationInfo files from previous, unsuccessful attempts to add the server to the cluster. (They were unsuccessful because I hadn't opened the port on that computer.) When I finally succeeded in adding the 2nd server, the 1st server started returning null when I tried to get data using the CLI. I stopped the 2nd server, deleted the files in system, restarted, and everything worked. I'm afraid that this, or some similar scenario will do the same, after I go live. How can I protect myself? On Mon, May 31, 2010 at 10:10 PM, Jonathan Ellis wrote: > No. > > On Mon, May 31, 2010 at 10:47 AM, David Boxenhorn > wrote: > > So this means that I can take my entire cluster off line if I make a > > mistake adding a new server??? Yikes! > > > > On Mon, May 31, 2010 at 6:41 PM, David Boxenhorn > wrote: > >> > >> OK. Got it working. > >> > >> I had some data in the 2nd server from previous failed attempts at > hooking > >> up to the cluster. When I deleted that data and tried again, it said > >> "bootstrapping" and my 1st server started working again. > >> > >> On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn > >> wrote: > >>> > >>> I am trying to get a cluster up and working for the first time. > >>> > >>> I got one server up and running, with lots of data on it, which I can > see > >>> with the CLI. > >>> > >>> I added my 2nd server, they seem to recognize each other. > >>> > >>> Now I can't see my data with the CLI. I do a get and it returns null. > The > >>> data files seem to be intact. > >>> > >>> What happened??? How can I fix it? > >>> > >> > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Error during startup
0.6.2 On Mon, May 31, 2010 at 9:50 PM, Jonathan Ellis wrote: > What version of Cassandra was this? > > On Sun, May 30, 2010 at 8:49 AM, David Boxenhorn > wrote: > > I deleted the system/LocationInfo files, and now everything works. > > > > Yay! (...what happened?) > > > > On Sun, May 30, 2010 at 4:18 PM, David Boxenhorn > wrote: > >> > >> I'm getting an "Expected both token and generation columns; found > >> ColumnFamily" error during startup can anyone tell me what it is? > Details > >> below. > >> > >> Starting Cassandra Server > >> Listening for transport dt_socket at address: > >> INFO 16:14:33,459 Auto DiskAccessMode determined to be standard > >> INFO 16:14:33,615 Sampling index for > >> C:\var\lib\cassandra\data\system\LocationInfo-1-Data.db > >> INFO 16:14:33,631 Removing orphan > >> C:\var\lib\cassandra\data\Lookin2\Users-tmp-27-Index.db > >> INFO 16:14:33,631 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db > >> INFO 16:14:33,662 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Users-18-Data.db > >> INFO 16:14:33,818 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db > >> INFO 16:14:33,850 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db > >> INFO 16:14:33,865 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db > >> INFO 16:14:33,881 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-580-Data.db > >> INFO 16:14:33,896 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-672-Data.db > >> INFO 16:14:33,912 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-681-Data.db > >> INFO 16:14:33,912 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-691-Data.db > >> INFO 16:14:33,928 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-696-Data.db > >> INFO 16:14:33,943 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Attractions-17-Data.db > >> INFO 16:14:34,006 Sampling index for > >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-5-Data.db > >> INFO 16:14:34,006 Sampling index for > >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-6-Data.db > >> INFO 16:14:34,021 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-29-Data.db > >> INFO 16:14:34,350 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-51-Data.db > >> INFO 16:14:34,693 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-72-Data.db > >> INFO 16:14:35,021 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-77-Data.db > >> INFO 16:14:35,225 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-78-Data.db > >> INFO 16:14:35,350 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-79-Data.db > >> INFO 16:14:35,459 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-80-Data.db > >> INFO 16:14:35,459 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-1-Data.db > >> INFO 16:14:35,475 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-2-Data.db > >> INFO 16:14:35,475 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Content-30-Data.db > >> INFO 16:14:35,631 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Content-35-Data.db > >> INFO 16:14:35,771 Sampling index for > >> C:\var\lib\cassandra\data\Lookin2\Content-40-Data.db > >> INFO 16:14:35,959 Compacting > >> > [org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db')] > >> ERROR 16:14:35,975 Exception encountered during startup. > >> java.lang.RuntimeException: Expected both token and generation columns; > >> found ColumnFamily(LocationInfo [Generation:false:4...@4,]) > >> at > >> org.apache.cassandra.db.SystemTable.initMetadata(SystemTable.java:159) > >> at > >> > org.apache.cassandra.service.StorageService.initServer(StorageService.java:305) > >> at > >> > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) > >> at > >> > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) > >> Exception encountered during startup. > >> > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: How to insert a row with a TimeUUIDType column in C++
http://php.net/manual/en/function.pack.php
2010/5/31 刘大伟 :
> How can I get 16 bytes timeUUID ?
>
>
> string(36) "4698cc00-6d2f-11df-8c7f-9f342400a648" TException: UUIDs must be
> exactly 16 bytes Error:
>
>
> On Fri, Apr 23, 2010 at 5:59 PM, Olivier Rosello
> wrote:
>>
>> Here is my test code :
>>
>> ColumnPath new_col;
>> new_col.__isset.column = true; /* this is required! */
>> new_col.column_family.assign("Incoming");
>> new_col.column.assign("1968ec4a-2a73-11df-9aca-00012e27a270");
>> client.insert("MyKeyspace", "somekey", new_col, "Random Value",
>> time(NULL), ONE);
>>
>> I didn't found in the C++ Cassandra/Thrift API how to specify TimeUUID
>> bytes (16) to the column name. The ColumnPath type get only a string field
>> for the name "column".
>>
>> With a String like this example shows, the TimeUUID is a 36 chars String
>> and this code throws a Exception "UUIDs must be exactly 16 bytes".
>>
>> I didn't found a function like "client.insert_timeuuid_column" which
>> convert the column name to an uint8_t[16]... or anything else which could
>> help me.
>>
>> Cheers,
>>
>> Olivier
>>
>>
>>
>> --
>> Olivier
>>
>>
>
>
>
> --
> 执著而努力着 david.liu
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Error during startup
Created https://issues.apache.org/jira/browse/CASSANDRA-1146 On Tue, Jun 1, 2010 at 12:46 AM, David Boxenhorn wrote: > 0.6.2 > > On Mon, May 31, 2010 at 9:50 PM, Jonathan Ellis wrote: >> >> What version of Cassandra was this? >> >> On Sun, May 30, 2010 at 8:49 AM, David Boxenhorn >> wrote: >> > I deleted the system/LocationInfo files, and now everything works. >> > >> > Yay! (...what happened?) >> > >> > On Sun, May 30, 2010 at 4:18 PM, David Boxenhorn >> > wrote: >> >> >> >> I'm getting an "Expected both token and generation columns; found >> >> ColumnFamily" error during startup can anyone tell me what it is? >> >> Details >> >> below. >> >> >> >> Starting Cassandra Server >> >> Listening for transport dt_socket at address: >> >> INFO 16:14:33,459 Auto DiskAccessMode determined to be standard >> >> INFO 16:14:33,615 Sampling index for >> >> C:\var\lib\cassandra\data\system\LocationInfo-1-Data.db >> >> INFO 16:14:33,631 Removing orphan >> >> C:\var\lib\cassandra\data\Lookin2\Users-tmp-27-Index.db >> >> INFO 16:14:33,631 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db >> >> INFO 16:14:33,662 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Users-18-Data.db >> >> INFO 16:14:33,818 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db >> >> INFO 16:14:33,850 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db >> >> INFO 16:14:33,865 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db >> >> INFO 16:14:33,881 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-580-Data.db >> >> INFO 16:14:33,896 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-672-Data.db >> >> INFO 16:14:33,912 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-681-Data.db >> >> INFO 16:14:33,912 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-691-Data.db >> >> INFO 16:14:33,928 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-696-Data.db >> >> INFO 16:14:33,943 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Attractions-17-Data.db >> >> INFO 16:14:34,006 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-5-Data.db >> >> INFO 16:14:34,006 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-6-Data.db >> >> INFO 16:14:34,021 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-29-Data.db >> >> INFO 16:14:34,350 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-51-Data.db >> >> INFO 16:14:34,693 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-72-Data.db >> >> INFO 16:14:35,021 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-77-Data.db >> >> INFO 16:14:35,225 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-78-Data.db >> >> INFO 16:14:35,350 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-79-Data.db >> >> INFO 16:14:35,459 Sampling index for >> >> >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-80-Data.db >> >> INFO 16:14:35,459 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-1-Data.db >> >> INFO 16:14:35,475 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-2-Data.db >> >> INFO 16:14:35,475 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Content-30-Data.db >> >> INFO 16:14:35,631 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Content-35-Data.db >> >> INFO 16:14:35,771 Sampling index for >> >> C:\var\lib\cassandra\data\Lookin2\Content-40-Data.db >> >> INFO 16:14:35,959 Compacting >> >> >> >> [org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db')] >> >> ERROR 16:14:35,975 Exception encountered during startup. >> >> java.lang.RuntimeException: Expected both token and generation columns; >> >> found ColumnFamily(LocationInfo [Generation:false:4...@4,]) >> >> at >> >> org.apache.cassandra.db.SystemTable.initMetadata(SystemTable.java:159) >> >> at >> >> >> >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:305) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) >> >> at >> >> >> >> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) >> >> Exception encountered during startup. >> >> >> > >> > >> >> >>
Re: Can't get data after building cluster
To elaborate: If you manage to screw things up to where it thinks a node has data, but it does not (adding a node without bootstrap would do this, for instance, which is probably what you did), at most data in the token range assigned to that node will be affected. On Tue, Jun 1, 2010 at 12:45 AM, David Boxenhorn wrote: > You say "no", but that is exactly what I just observed. Can I have some more > explanation? > > To recap: I added a server to my cluster. It had some junk in the > system/LocationInfo files from previous, unsuccessful attempts to add the > server to the cluster. (They were unsuccessful because I hadn't opened the > port on that computer.) When I finally succeeded in adding the 2nd server, > the 1st server started returning null when I tried to get data using the > CLI. I stopped the 2nd server, deleted the files in system, restarted, and > everything worked. > > I'm afraid that this, or some similar scenario will do the same, after I go > live. How can I protect myself? > > On Mon, May 31, 2010 at 10:10 PM, Jonathan Ellis wrote: >> >> No. >> >> On Mon, May 31, 2010 at 10:47 AM, David Boxenhorn >> wrote: >> > So this means that I can take my entire cluster off line if I make a >> > mistake adding a new server??? Yikes! >> > >> > On Mon, May 31, 2010 at 6:41 PM, David Boxenhorn >> > wrote: >> >> >> >> OK. Got it working. >> >> >> >> I had some data in the 2nd server from previous failed attempts at >> >> hooking >> >> up to the cluster. When I deleted that data and tried again, it said >> >> "bootstrapping" and my 1st server started working again. >> >> >> >> On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn >> >> wrote: >> >>> >> >>> I am trying to get a cluster up and working for the first time. >> >>> >> >>> I got one server up and running, with lots of data on it, which I can >> >>> see >> >>> with the CLI. >> >>> >> >>> I added my 2nd server, they seem to recognize each other. >> >>> >> >>> Now I can't see my data with the CLI. I do a get and it returns null. >> >>> The >> >>> data files seem to be intact. >> >>> >> >>> What happened??? How can I fix it? >> >>> >> >> >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: searching keys of the form substring*
Thanks Vineet for replying, but I am not able to understand how can we use
variable substitution in it.
On Mon, May 31, 2010 at 4:42 PM, vd wrote:
> Hi Sagar
>
> You can use variable substitution.
> ___
> Vineet Daniel
> ___
>
> Let your email find you
>
>
>
> On Mon, May 31, 2010 at 3:44 PM, Sagar Agrawal wrote:
>
>> Hi folks,
>>
>> I want to fetch all those records from my column family such that the key
>> starts with a specified string...
>>
>> e.g. Suppose I have a CF keyed on full names(first name + last name) of
>> persons...
>> now I want to fetch all those records whose first name is 'John'
>>
>> Right now, I am using OPP and KeyRange in the following way:
>>
>> KeyRange keyRange = new KeyRange();
>> keyRange.setStart_key("John");
>> keyRange.setEnd_key("Joho");
>>
>> but this is sort of hard coding can anyone suggest a better way to
>> achieve this?
>>
>> I would be really grateful... thank you.
>>
>>
>>
>
writing speed test
Hi all, I'm testing writing speed of cassandra with 4 servers. I'm confused by the behavior of cassandra. ---env--- load-data app written in c++, using libcassandra (w/ modified batch insert) 20 writing threads in 2 processes running on 2 servers ---optimization--- 1.turn log level to INFO 2.JVM has 8G heap 3.32 concurrent read & 128 write in storage-conf.xml, other cache enlarged as well. ---result--- 1-monitoring by `date;nodetool -h host ring` I add all load together and measure the writing speed by (load_difference / time_difference), and I get about 15MB/s for the whole cluster. 2-monitoring by `iostat -m 10` I can watch the disk_io from the system level and have about 10MB/s - 65MB/s for a single machine. Very big variance over time. 3-monitoring by `iptraf -g` In this way I watch the communication between servers and get about 10MB/s for a single machine. ---opinion--- So, have you checked the writing speed of cassandra? I feel it's quite slow currently. Could anyone confirm this is the normal writing speed of cassandra, or please provide someway of improving it? -- Shuai Yuan Supertool Corp. ?? 13810436859 yuan-sh...@yuan-shuai.info
Re: nodetool cleanup isn't cleaning up?
ok, let me try and translate your answer ;) Are you saying that the data that was left on the node is non-primary-replicas of rows from the time before the move? So this implies that when a node moves in the ring, it will affect distribution of: - new keys - old keys primary node -- but will not affect distribution of old keys non-primary replicas. If so, still I don't understand something... I would expect even the non-primary replicas of keys to be moved since if they don't, how would they be found? I mean upon reads the serving node should not care about whether the row is new or old, it should have a consistent and global mapping of tokens. So I guess this ruins my theory... What did you mean then? Is this deletions of non-primary replicated data? How does the replication factor affect the load on the moved host then? On Tue, Jun 1, 2010 at 1:19 AM, Jonathan Ellis wrote: > well, there you are then. > > On Mon, May 31, 2010 at 2:34 PM, Ran Tavory wrote: > > yes, replication factor = 2 > > > > On Mon, May 31, 2010 at 10:07 PM, Jonathan Ellis > wrote: > >> > >> you have replication factor > 1 ? > >> > >> On Mon, May 31, 2010 at 7:23 AM, Ran Tavory wrote: > >> > I hope I understand nodetool cleanup correctly - it should clean up > all > >> > data > >> > that does not (currently) belong to this node. If so, I think it might > >> > not > >> > be working correctly. > >> > Look at nodes 192.168.252.124 and 192.168.252.99 below > >> > 192.168.252.99Up 279.35 MB > >> > 3544607988759775661076818827414252202 > >> > |<--| > >> > 192.168.252.124Up 167.23 MB > >> > 56713727820156410577229101238628035242 | ^ > >> > 192.168.252.125Up 82.91 MB > >> > 85070591730234615865843651857942052863 v | > >> > 192.168.254.57Up 366.6 MB > >> > 113427455640312821154458202477256070485| ^ > >> > 192.168.254.58Up 88.44 MB > >> > 141784319550391026443072753096570088106v | > >> > 192.168.254.59Up 88.45 MB > >> > 170141183460469231731687303715884105727|-->| > >> > I wanted 124 to take all the load from 99. So I issued a move command. > >> > $ nodetool -h cass99 -p 9004 move > 56713727820156410577229101238628035243 > >> > > >> > This command tells 99 to take the space b/w > >> > > >> > > (56713727820156410577229101238628035242, > 56713727820156410577229101238628035243] > >> > which is basically just one item in the token space, almost nothing... > I > >> > wanted it to be very slim (just playing around). > >> > So, next I get this: > >> > 192.168.252.124Up 803.33 MB > >> > 56713727820156410577229101238628035242 |<--| > >> > 192.168.252.99Up 352.85 MB > >> > 56713727820156410577229101238628035243 | ^ > >> > 192.168.252.125Up 134.24 MB > >> > 85070591730234615865843651857942052863 v | > >> > 192.168.254.57Up 676.41 MB > >> > 113427455640312821154458202477256070485| ^ > >> > 192.168.254.58Up 99.74 MB > >> > 141784319550391026443072753096570088106v | > >> > 192.168.254.59Up 99.94 MB > >> > 170141183460469231731687303715884105727|-->| > >> > The tokens are correct, but it seems that 99 still has a lot of data. > >> > Why? > >> > OK, that might be b/c it didn't delete its moved data. > >> > So next I issued a nodetool cleanup, which should have taken care of > >> > that. > >> > Only that it didn't, the node 99 still has 352 MB of data. Why? > >> > So, you know what, I waited for 1h. Still no good, data wasn't cleaned > >> > up. > >> > I restarted the server. Still, data wasn't cleaned up... I issued a > >> > cleanup > >> > again... still no good... what's up with this node? > >> > > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Can't get data after building cluster
I don't think it can be the case that "at most data in the token range assigned to that node will be affected" - the new node had no knowledge of any of our data. Any fake "data" that it might have had through some error on my part could not have been within the range of real data. I had 4.25 G of data on the 1st server, and as far as I could tell I couldn't access any of it. On Tue, Jun 1, 2010 at 9:10 AM, Jonathan Ellis wrote: > To elaborate: > > If you manage to screw things up to where it thinks a node has data, > but it does not (adding a node without bootstrap would do this, for > instance, which is probably what you did), at most data in the token > range assigned to that node will be affected. > > On Tue, Jun 1, 2010 at 12:45 AM, David Boxenhorn > wrote: > > You say "no", but that is exactly what I just observed. Can I have some > more > > explanation? > > > > To recap: I added a server to my cluster. It had some junk in the > > system/LocationInfo files from previous, unsuccessful attempts to add the > > server to the cluster. (They were unsuccessful because I hadn't opened > the > > port on that computer.) When I finally succeeded in adding the 2nd > server, > > the 1st server started returning null when I tried to get data using the > > CLI. I stopped the 2nd server, deleted the files in system, restarted, > and > > everything worked. > > > > I'm afraid that this, or some similar scenario will do the same, after I > go > > live. How can I protect myself? > > > > On Mon, May 31, 2010 at 10:10 PM, Jonathan Ellis > wrote: > >> > >> No. > >> > >> On Mon, May 31, 2010 at 10:47 AM, David Boxenhorn > >> wrote: > >> > So this means that I can take my entire cluster off line if I make > a > >> > mistake adding a new server??? Yikes! > >> > > >> > On Mon, May 31, 2010 at 6:41 PM, David Boxenhorn > >> > wrote: > >> >> > >> >> OK. Got it working. > >> >> > >> >> I had some data in the 2nd server from previous failed attempts at > >> >> hooking > >> >> up to the cluster. When I deleted that data and tried again, it said > >> >> "bootstrapping" and my 1st server started working again. > >> >> > >> >> On Mon, May 31, 2010 at 4:50 PM, David Boxenhorn > >> >> wrote: > >> >>> > >> >>> I am trying to get a cluster up and working for the first time. > >> >>> > >> >>> I got one server up and running, with lots of data on it, which I > can > >> >>> see > >> >>> with the CLI. > >> >>> > >> >>> I added my 2nd server, they seem to recognize each other. > >> >>> > >> >>> Now I can't see my data with the CLI. I do a get and it returns > null. > >> >>> The > >> >>> data files seem to be intact. > >> >>> > >> >>> What happened??? How can I fix it? > >> >>> > >> >> > >> > > >> > > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of Riptano, the source for professional Cassandra support > >> http://riptano.com > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Error during startup
Thanks! On Tue, Jun 1, 2010 at 9:09 AM, Jonathan Ellis wrote: > Created https://issues.apache.org/jira/browse/CASSANDRA-1146 > > On Tue, Jun 1, 2010 at 12:46 AM, David Boxenhorn > wrote: > > 0.6.2 > > > > On Mon, May 31, 2010 at 9:50 PM, Jonathan Ellis > wrote: > >> > >> What version of Cassandra was this? > >> > >> On Sun, May 30, 2010 at 8:49 AM, David Boxenhorn > >> wrote: > >> > I deleted the system/LocationInfo files, and now everything works. > >> > > >> > Yay! (...what happened?) > >> > > >> > On Sun, May 30, 2010 at 4:18 PM, David Boxenhorn > >> > wrote: > >> >> > >> >> I'm getting an "Expected both token and generation columns; found > >> >> ColumnFamily" error during startup can anyone tell me what it is? > >> >> Details > >> >> below. > >> >> > >> >> Starting Cassandra Server > >> >> Listening for transport dt_socket at address: > >> >> INFO 16:14:33,459 Auto DiskAccessMode determined to be standard > >> >> INFO 16:14:33,615 Sampling index for > >> >> C:\var\lib\cassandra\data\system\LocationInfo-1-Data.db > >> >> INFO 16:14:33,631 Removing orphan > >> >> C:\var\lib\cassandra\data\Lookin2\Users-tmp-27-Index.db > >> >> INFO 16:14:33,631 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db > >> >> INFO 16:14:33,662 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Users-18-Data.db > >> >> INFO 16:14:33,818 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db > >> >> INFO 16:14:33,850 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db > >> >> INFO 16:14:33,865 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db > >> >> INFO 16:14:33,881 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-580-Data.db > >> >> INFO 16:14:33,896 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-672-Data.db > >> >> INFO 16:14:33,912 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-681-Data.db > >> >> INFO 16:14:33,912 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-691-Data.db > >> >> INFO 16:14:33,928 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-696-Data.db > >> >> INFO 16:14:33,943 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Attractions-17-Data.db > >> >> INFO 16:14:34,006 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-5-Data.db > >> >> INFO 16:14:34,006 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-6-Data.db > >> >> INFO 16:14:34,021 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-29-Data.db > >> >> INFO 16:14:34,350 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-51-Data.db > >> >> INFO 16:14:34,693 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-72-Data.db > >> >> INFO 16:14:35,021 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-77-Data.db > >> >> INFO 16:14:35,225 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-78-Data.db > >> >> INFO 16:14:35,350 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-79-Data.db > >> >> INFO 16:14:35,459 Sampling index for > >> >> > >> >> > C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-80-Data.db > >> >> INFO 16:14:35,459 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-1-Data.db > >> >> INFO 16:14:35,475 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-2-Data.db > >> >> INFO 16:14:35,475 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Content-30-Data.db > >> >> INFO 16:14:35,631 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Content-35-Data.db > >> >> INFO 16:14:35,771 Sampling index for > >> >> C:\var\lib\cassandra\data\Lookin2\Content-40-Data.db > >> >> INFO 16:14:35,959 Compacting > >> >> > >> >> > [org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db')] > >> >> ERROR 16:14:35,975 Exception encountered during startup. > >> >> java.lang.RuntimeException: Expected both token and generation > columns; > >> >> found ColumnFamily(LocationInfo [Generation:false:4...@4,]) > >> >> at > >> >> > org.apache.cassandra.db.SystemTable.initMetadata(SystemTable.java:159) > >> >> at > >> >> > >> >> > org.apache.cassandra.service.StorageService.initServer
