Multiget_slice or composite column keys?

[Charles Blaxland]

Hi All,

New to Cassandra, so apologies if I don't fully grok stuff just yet.

I have data keyed by a key as well as a date. I want to run a query to get
multiple keys across multiple contiguous date ranges simultaneously. I'm
currently storing the date along with the row key like this:

key1|2011-05-15 {  c1 : , c2 :,  c3 : ... }
key1|2011-05-16 {  c1 : , c2 :,  c3 : ... }
key2|2011-05-15 {  c1 : , c2 :,  c3 : ... }
key2|2011-05-16 {  c1 : , c2 :,  c3 : ... }
...

I generate all the key/date combinations that I'm interested in and use
multiget_slice to retrieve them, pulling in all the columns for each key (I
need all the data, but the number of columns is small: less than 100). The
total number of row keys retrieved will only be 100 or so.

Now it strikes me I could also store this using composite columns, like
this:

key1 {  2011-05-15|c1 : , 2011-5-16|c1 : , 2011-05-15|c2 :, 2011-05-16|c2 :
, 2011-05-15|c3 : , 2011-05-16|c3 : , ... }
key2 {  2011-05-15|c1 : , 2011-5-16|c1 : , 2011-05-15|c2 :, 2011-05-16|c2 :
, 2011-05-15|c3 : , 2011-05-16|c3 : , ... }
...

Then use multislice_get again (but with less keys), and use a slice range to
only retrieve the dates I'm interested in.

Another alternative I guess would be to use OPP with the first storage
approach and get_range_slices, but as I understand this would not be great
for performance due to keys being clustered together on a single node?

So my question is, which approach is best? One downside to the latter I
guess is that the number of columns grows without bound (although with 2
billion to play with this isn't gonna be  a problem any time soon). Also
multiget_slice supports only one slice predicate, so I'd guess I'd have to
use multiple queries to get multiple date ranges.

Anyway, any thoughts/tips appreciated.

Thanks,
Charles

Re: Native heap leaks?

[Peter Schuller]

> Ok, so I think I found one major cause contributing to the increasing

Nice job tracking this down! That is useful to know, even outside of
Cassandra use cases. Frankly it's disappointing to learn what nio is
doing.

-- 
/ Peter Schuller

Re: Inconsistent data issues when running nodetool move.

[Peter Schuller]

> That makes sense with my symptoms...  I was doing a CL=ONE for write and a 
> CL=ALL for read, expecting that to be sufficient.
>
> I will try both set to ALL and see if I get better consistency.

Just keep in mind that with ALL queries will fail with just a single
node down. (This is why QUORUM is useful since it allows consistency
semantics similar to ALL but allows you to survive nodes being down,
at the cost of a higher RF (3 at least))

-- 
/ Peter Schuller

Re: Native heap leaks?

[Jonathan Ellis]

Great debugging work!

That workaround sounds like the best alternative to me too.

On Sat, May 14, 2011 at 9:46 PM, Hannes Schmidt  wrote:
> Ok, so I think I found one major cause contributing to the increasing
> resident size of the Cassandra process. Looking at the OpenJDK sources
> was of great help in understanding the problem but my findings also
> apply to the Sun/Oracle JDK because the affected code is shared by
> both.
>
> Each IncomingTcpConnection (ITC) thread handles a socket to another
> node. That socket is a server socket returned from
> ServerSocket.accept() and as such it is implemented on top of an NIO
> socket channel (sun.nio.ch.SocketAdaptor) which in turn makes use of
> direct byte buffers. It obtains these buffers from sun.nio.ch.Util
> which caches the 3 most recently used buffers per thread. If a cached
> buffer isn't large enough for a message, a new one that is will
> replace it. The size of the buffer is determined by the amount of data
> that the application requests to be read. ITC uses the readFully()
> method of DataInputStream (DIS) to read data into a byte array
> allocated to hold the entire message:
>
> int size = socketStream.readInt();
> byte[] buffer = new byte[size];
> socketStream.readFully(buffer);
>
> Whatever the value of 'size' will end up being the size of the direct
> buffer allocated by the socket channel code.
>
> Our application uses range queries whose result sets are around 40
> megabytes in size. If a range isn't hosted on the node the application
> client is connected to, the range result set will be fetched from
> another node. When that other node has prepared the result it will
> send it back (asynchonously, this took me a while to grasp) and it
> will end up in the direct byte buffer that is cached by
> sun.nio.ch.Util for the ITC thread on the original node.
>
> The thing is that the buffer holds the entire message, all 40 megs of
> it. ITC is rather long-lived and so the buffers will simply stick
> around. Our range queries cover the entire ring (we do a lot of "map
> reduce") and so each node ends up with as many 40M buffers as we have
> nodes in the ring, 10 in our case. That's 400M of native heap space
> wasted on each node.
>
> Each ITC thread holds onto the historically largest direct buffer,
> possibly for a long time. This could be alleviated by periodically
> closing the connection and thereby releasing a potentially large
> buffer and replacing it with a new thread that starts with a clean
> slate. If all queries have large result sets, this solution won't
> help. Another alternative is to read the message incrementally rather
> than buffering it in its entirety in a byte array as ITC currently
> does. A third and possibly the simplest solution would be to read the
> messages into the buffer in chunks of say 1M. DIS has offers
> readFully( data, offset, length ) for that. I have tried this solution
> and it fixes this problem for us. I'll open an issue and submit my
> patch. We have observed the issue with 0.6.12 but from looking at ITC
> in trunk it seems to be affected too.
>
> It gets worse though: even after the ITC thread dies, the cached
> buffers stick around as they are being held via SoftReferences. SR's
> are released only as a last resort to prevent an OutOfMemoryException.
> Using SR's for caching direct buffers is silly because direct buffers
> have negligible impact on the Java heap but may have dramatic impact
> on the native heap. I am not the only one who thinks so [1]. In other
> words, sun.nio.ch.Util's buffer caching is severely broken. I tried to
> find a way to explicitly release soft references but haven't found
> anything other than the allocation of an oversized array to force an
> OutOfMemoryException. The only thing we can do is to keep the buffer
> sizes small in order to reduce the impact of the leak. My patch takes
> care of that.
>
> I will post a link to the JIRA issue with the patch shortly.
>
> [1] http://bugs.sun.com/view_bug.do?bug_id=6210541
>
> On Wed, May 4, 2011 at 11:50 AM, Hannes Schmidt  wrote:
>> Hi,
>>
>> We are using Cassandra 0.6.12 in a cluster of 9 nodes. Each node is
>> 64-bit, has 4 cores and 4G of RAM and runs on Ubuntu Lucid with the
>> stock 2.6.32-31-generic kernel. We use the Sun/Oracle JDK.
>>
>> Here's the problem: The Cassandra process starts up with 1.1G resident
>> memory (according to top) but slowly grows to 2.1G at a rate that
>> seems proportional to the write load. No writes, no growth. The node
>> is running other memory-sensitive applications (a second JVM for our
>> in-house webapp and a short-lived C++ program) so we need to ensure
>> that each process stays within certain bounds as far as memory
>> requirements go. The nodes OOM and crash when the Cassandra process is
>> at 2.1G so I can't say if the growth is bounded or not.
>>
>> Looking at the /proc/$pid/smaps for the Cassandra process it seems to
>> me that it is the native heap of the Cassandra JVM that is leaking

Re: Native heap leaks?

[Hannes Schmidt]

As promised: https://issues.apache.org/jira/browse/CASSANDRA-2654

On Sun, May 15, 2011 at 7:09 AM, Jonathan Ellis  wrote:
> Great debugging work!
>
> That workaround sounds like the best alternative to me too.
>
> On Sat, May 14, 2011 at 9:46 PM, Hannes Schmidt  wrote:
>> Ok, so I think I found one major cause contributing to the increasing
>> resident size of the Cassandra process. Looking at the OpenJDK sources
>> was of great help in understanding the problem but my findings also
>> apply to the Sun/Oracle JDK because the affected code is shared by
>> both.
>>
>> Each IncomingTcpConnection (ITC) thread handles a socket to another
>> node. That socket is a server socket returned from
>> ServerSocket.accept() and as such it is implemented on top of an NIO
>> socket channel (sun.nio.ch.SocketAdaptor) which in turn makes use of
>> direct byte buffers. It obtains these buffers from sun.nio.ch.Util
>> which caches the 3 most recently used buffers per thread. If a cached
>> buffer isn't large enough for a message, a new one that is will
>> replace it. The size of the buffer is determined by the amount of data
>> that the application requests to be read. ITC uses the readFully()
>> method of DataInputStream (DIS) to read data into a byte array
>> allocated to hold the entire message:
>>
>> int size = socketStream.readInt();
>> byte[] buffer = new byte[size];
>> socketStream.readFully(buffer);
>>
>> Whatever the value of 'size' will end up being the size of the direct
>> buffer allocated by the socket channel code.
>>
>> Our application uses range queries whose result sets are around 40
>> megabytes in size. If a range isn't hosted on the node the application
>> client is connected to, the range result set will be fetched from
>> another node. When that other node has prepared the result it will
>> send it back (asynchonously, this took me a while to grasp) and it
>> will end up in the direct byte buffer that is cached by
>> sun.nio.ch.Util for the ITC thread on the original node.
>>
>> The thing is that the buffer holds the entire message, all 40 megs of
>> it. ITC is rather long-lived and so the buffers will simply stick
>> around. Our range queries cover the entire ring (we do a lot of "map
>> reduce") and so each node ends up with as many 40M buffers as we have
>> nodes in the ring, 10 in our case. That's 400M of native heap space
>> wasted on each node.
>>
>> Each ITC thread holds onto the historically largest direct buffer,
>> possibly for a long time. This could be alleviated by periodically
>> closing the connection and thereby releasing a potentially large
>> buffer and replacing it with a new thread that starts with a clean
>> slate. If all queries have large result sets, this solution won't
>> help. Another alternative is to read the message incrementally rather
>> than buffering it in its entirety in a byte array as ITC currently
>> does. A third and possibly the simplest solution would be to read the
>> messages into the buffer in chunks of say 1M. DIS has offers
>> readFully( data, offset, length ) for that. I have tried this solution
>> and it fixes this problem for us. I'll open an issue and submit my
>> patch. We have observed the issue with 0.6.12 but from looking at ITC
>> in trunk it seems to be affected too.
>>
>> It gets worse though: even after the ITC thread dies, the cached
>> buffers stick around as they are being held via SoftReferences. SR's
>> are released only as a last resort to prevent an OutOfMemoryException.
>> Using SR's for caching direct buffers is silly because direct buffers
>> have negligible impact on the Java heap but may have dramatic impact
>> on the native heap. I am not the only one who thinks so [1]. In other
>> words, sun.nio.ch.Util's buffer caching is severely broken. I tried to
>> find a way to explicitly release soft references but haven't found
>> anything other than the allocation of an oversized array to force an
>> OutOfMemoryException. The only thing we can do is to keep the buffer
>> sizes small in order to reduce the impact of the leak. My patch takes
>> care of that.
>>
>> I will post a link to the JIRA issue with the patch shortly.
>>
>> [1] http://bugs.sun.com/view_bug.do?bug_id=6210541
>>
>> On Wed, May 4, 2011 at 11:50 AM, Hannes Schmidt  wrote:
>>> Hi,
>>>
>>> We are using Cassandra 0.6.12 in a cluster of 9 nodes. Each node is
>>> 64-bit, has 4 cores and 4G of RAM and runs on Ubuntu Lucid with the
>>> stock 2.6.32-31-generic kernel. We use the Sun/Oracle JDK.
>>>
>>> Here's the problem: The Cassandra process starts up with 1.1G resident
>>> memory (according to top) but slowly grows to 2.1G at a rate that
>>> seems proportional to the write load. No writes, no growth. The node
>>> is running other memory-sensitive applications (a second JVM for our
>>> in-house webapp and a short-lived C++ program) so we need to ensure
>>> that each process stays within certain bounds as far as memory
>>> requirements go. The nodes OOM and crash when the Cassandra

Inconsistent results using secondary indexes between two DC

[Wojciech Pietrzok]

Hello,

I've noticed strange behaviour of Cassandra when using secondary indexes.
There are 2 Data Centers, each with 2 nodes, RF=4, on all nodes
Cassandra 0.7.5 is installed.
When I connect to one of the nodes in DC1 and perform query using
secondary indexes ("get ColumnFamily where column = 'foo'" in
cassandra-cli) I always get correct number of rows returned, no matter
which ConsistencyLevel is set.
When I connect to one of the nodes in DC2 and perform same query using
ConsistencyLevel LOCAL_QUORUM the results are correct. But using
ConsistencyLevel ONE Cassandra doesn't return correct number of rows
(it seems that most of the times there some of the rows are missing).
Tried running nodetool repair, and nodetool scrub but this doesn't seem to help.

What might the cause of such behaviour?

-- 
KosciaK

Re: Import/Export of Schema Migrations

[aaron morton]

Not sure what sort of changes you are making, but this is my approach. 

I've always managed database (my sql, sql server whatever) schema as source 
code (SQL DDL statements, CLI script etc). It makes it a lot easier to cold 
start the system, test changes and see who changed what. 

Once you have your initial schema you can hand roll a CLI script to update  / 
drop existing CF's. For the update column family statement all the attributes 
are delta to the current setting, i.e. you do not need to say comparator is 
ascii again. Other than the indexes, you need to specify all the indexes again 
those not included will be dropped. 

If you want to be able to replay multiple schema changes made during dev 
against other clusters my personal approach would be:

- create a cli script for every change (using update and delete CF), prefixed 
with 000X so you can see the order. 
- manage the scripts in source control
- sanity check to see if they can be collapsed 
- replay the changes in order when applying them to a cluster. 
(you will still need to manually delete data from dropped cf's)
 
changes to conf/cassandra.yaml can be managed using chef 

 
Others will have different ideas

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14 May 2011, at 00:15, David Boxenhorn wrote:

> Actually, I want a way to propagate *any* changes from development to staging 
> to production, but schema changes are the most important.
> 
> Could I use 2221 to propagate schema changes by deleting the schema in the 
> target cluster, doing "show schema" in the source cluster, redirecting to a 
> file, and running the file as a script in the target cluster? 
> 
> Of course, I would have to delete the files of dropped CFs by hand (something 
> I *hate* to do, because I'm afraid of making a mistake), but it would be a 
> big improvement. 
> 
> I am open to any other ideas of how to propagate changes from one cluster to 
> another in an efficient non-error-prone fashion. Our development environment 
> (i.e. development, staging, production) is pretty standard, so I'm sure that 
> I'm not the only one with this problem! 
> 
> 
> On Fri, May 13, 2011 at 12:51 PM, aaron morton  
> wrote:
> What sort of schema changes are you making?  can you manage them as a CLI 
> script under source control ? 
> 
> 
> You may also be interested in  CASSANDRA-2221.
> 
> Cheers
> Aaron
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12 May 2011, at 20:45, David Boxenhorn wrote:
> 
>> My use case is like this: I have a development cluster, a staging cluster 
>> and a production cluster. When I finish a set of migrations (i.e. changes) 
>> on the development cluster, I want to apply them to the staging cluster, and 
>> eventually the production cluster. I don't want to do it by hand, because 
>> it's a painful and error-prone process. What I would like to do is export 
>> the last N migrations from the development cluster as a text file, with 
>> exactly the same format as the original text commands, and import them to 
>> the staging and production clusters.
>> 
>> I think the best place to do this might be the CLI, since you would probably 
>> want to view your migrations before exporting them. Something like this:
>> 
>> show migrations N;Shows the last N migrations.
>> export migrations N ;   Exports the last N migrations to file 
>> fileName.
>> import migrations ; Imports migrations from fileName.
>> 
>> The import process would apply the migrations one at a time giving you 
>> feedback like, "applying migration: update column family...". If a migration 
>> fails, the process should give an appropriate message and stop.
>> 
>> Is anyone else interested in this? I have created a Jira ticket for it here:
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-2636 
>> 
>> 
> 
> 

Re: [howto measure disk usage]

[aaron morton]

Sub columns for a super column do serialise their time stamp, they are just the 
same as regular column. The super column does not have a timestamp of it's own. 
It does have it's own tombstone marker though. 

Super Column does not take a huge amount more disk space, just the name a shot 
int, two ints and a long int.

Some things to consider:

- were their any compacted files on disk ? these are sstables that have one 
zero length file with COMPACTED in the name.  These files will be deleted at 
some point. 
- What did the commit log directory look like ? Flushing should have check 
pointed all the log segments and deleted the log files. 
- I'm assuming this was a single node, if not was the node collecting Hinted 
- Did the standard CF have cache saving enabled ?

Take a poke around the /var/lib/cassandra tree and let us know if you see 
anything interesting. 

Cheers
  
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14 May 2011, at 03:15, Alexis Rodríguez wrote:

> cassandra-people,
> 
> I'm trying to measure disk usage by cassandra after inserting some columns in 
> order to plan disk sizes and configurations for future deploys. 
> 
> My approach is very straightforward:
> 
> clean_data (stop_cassandra && rm -rf 
> /var/lib/cassandra/{dara,commitlog,saved_caches}/*)
> perform_inserts
> measure_disk_usage (nodetool -flush && du -ch /var/lib/cassandra)
> 
> There are two types of inserts:
> In a simple column with key, name and value a random string of size 100
> In a super-column with key, super-column-name, name and value a random string 
> of size 100
> But surprisingly when I'm inserting 100 million columns on a simple column it 
> uses more disk than the same amount in a super-column. How can that be 
> possible?
> 
> For simple column 41984 MB and for super-column 29696, the difference is more 
> than noticeable!
> 
> Somebody told me yesterday that super-columns don't have a per-column 
> timestamp, but... it in my case, even if every data was in the same 
> super-column-key it will not explain the difference!
> 
> 
> ps: sorry, English is not my first language
> 
> 
> 

Re: Converting separate clusters in mutliple dc to one cluster across multiple dcs

[aaron morton]

Rename a cluster 
http://wiki.apache.org/cassandra/FAQ#clustername_mismatch
see also https://issues.apache.org/jira/browse/CASSANDRA-2632

The best approach will depend on 
- how much data you have
- if this is a running system 
- are the schemas identical 
- are the individual clusters using the NetworkTopologyStrategy ? 
- how much excitement you like.  

Initial thoughts, doubt this is correct...

- change the tokens in each dc 
- for a dc that will have the others join it, change 
cassandra-topology.properties to put the new nodes in the new dc
- for a dc joining another, follow the procedure to change the name on each 
node, update the seed list and update the topology to include the dc it is 
joining, remove the schema and migrations SStables, as you do a rolling restart
- update the seed list on the dc the others joined
- confirm there is one schema def
- update the replication factor of the schema to put replicas in each DC  
http://wiki.apache.org/cassandra/Operations#Replication
- nodetool repair to change the RF and then cleanup 

There will be a lot of data moves and I doubt I got it right, is there any 
chance of spinning up a whole new cluster and reloading the data ?

I can give it some more thought later if you need. 

Hope that helps. 

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14 May 2011, at 04:55, Anurag Gujral wrote:

> Hi All,
> I  have 3 separate cassandra clusters running in multiple data 
> centers which I want to convert to  one cassandra cluster across multiple 
> data centers
> Does anyone tried this? If so what are the steps:
> 
> I think I need to do the following:
> a)Change cluster name: What is the procedure to do this?
> b)Change tokens so that each node have a unique token
> c)Change seed nodes so that machines in one data center know about machines 
> in another data center.
> 
> Thanks
> Anurag
> 
> 
> 

Re: assertion error in cassandra when doing nodetool move

[aaron morton]

I've had a bit more of a look and am a bit confused between what I see in the 
stack below and what's in the source. What version are you seeing ?

Thank

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14 May 2011, at 03:23, Anurag Gujral wrote:

> I checked file cassandra-topology.properties everything looks good to me.
> Please Advise on next steps I can do.
> Thanks
> Anurag
> 
> On Fri, May 13, 2011 at 3:52 AM, aaron morton  wrote:
> I think you may have a data centre defined in the strategy_options for the 
> keyspace that does not have any nodes in it. Check the dc names in the create 
> keyspace statement match the names in your cassandra-topology.propeties file 
> (assuming you are using the PropertyFileSnitch), or that you have nodes 
> assigned to all dc's.
> 
> Hope that helps.
> 
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 13 May 2011, at 13:45, Anurag Gujral wrote:
> 
> > Hi All,
> > I run following command on one of my nodes to  move the token 
> > from 0 to 2.
> > /usr/cassandra/cassandra/bin/nodetool -h 10.170.195.204 -p 8080 move 2. I 
> > dont understand why is this happening?
> >
> > I am getting the following assertion error:
> > Exception in thread "main" java.lang.AssertionError
> > at 
> > org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:389)
> > at 
> > org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:414)
> > at 
> > org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:94)
> > at 
> > org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
> > at 
> > org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
> > at 
> > org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595)
> > at 
> > org.apache.cassandra.service.StorageService.move(StorageService.java:1733)
> > at 
> > org.apache.cassandra.service.StorageService.move(StorageService.java:1708)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at 
> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> > at 
> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> > at 
> > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> > at 
> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> > at 
> > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> > at 
> > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> > at 
> > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> > at 
> > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> > at 
> > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> > at 
> > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:616)
> > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> > at sun.rmi.transport.Transport$1.run(Transport.java:177)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> > at 
> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> > at 
> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> > at 
> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> > at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:636)
> >
> > Thanks
> > Anurag
> >
> 
> 

Re: problems using property snitch file

[aaron morton]

Whats does your cassandra-topology.properties file look like ?

Cheers

-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 14 May 2011, at 11:48, Anurag Gujral wrote:

> Hi All,
>   I have built a cassandra cluster which is using property snitch 
> file to specify my network topology to cassandra.The keyspace I have use 
> network topology strategy.
> 
> When I tried to add new node to this cluster,I get the following error:
> RROR [main] 2011-05-13 23:45:44,152 AbstractCassandraDaemon.java (line 333) 
> Exception encountered during startup.
> java.lang.NullPointerException
> at 
> org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:96)
> at 
> org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:87)
> at 
> org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
> at 
> org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:230)
> at 
> org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:195)
> at 
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
> at 
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
> at 
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
> at 
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
> at 
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
> at 
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
> at 
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
> 
> 
> Please Advise.
> 
> 
> Thanks
> Anurag
> 

Re: JMX access sample by jython (Was: How to invoke getNaturalEndpoints with jconsole?)

[aaron morton]

Nice,

Thanks
-
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 15 May 2011, at 01:12, Maki Watanabe wrote:

> Just FYI for beginners like me: I've also write it with jython.
> Getting attributes are more easier than invoke Operations.  I feel
> jython will be a good option to create custom monitoring/management
> tools.
> 

Re: assertion error in cassandra when doing nodetool move

[Anurag Gujral]

I am  using version 0.7.3
Thanks
Anurag

On Fri, May 13, 2011 at 11:15 AM, Anurag Gujral wrote:

> I am using network Topology strategy for one of my keyspaces.
> Thanks
> Anurag
>
>
> On Thu, May 12, 2011 at 6:45 PM, Anurag Gujral wrote:
>
>> Hi All,
>> I run following command on one of my nodes to  move the token
>> from 0 to 2.
>> /usr/cassandra/cassandra/bin/nodetool -h 10.170.195.204 -p 8080 move 2. I
>> dont understand why is this happening?
>>
>> I am getting the following assertion error:
>> Exception in thread "main" java.lang.AssertionError
>> at
>> org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:389)
>> at
>> org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:414)
>> at
>> org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:94)
>> at
>> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929)
>> at
>> org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895)
>> at
>> org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595)
>> at
>> org.apache.cassandra.service.StorageService.move(StorageService.java:1733)
>> at
>> org.apache.cassandra.service.StorageService.move(StorageService.java:1708)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
>> at
>> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
>> at
>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
>> at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
>> at
>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
>> at sun.rmi.transport.Transport$1.run(Transport.java:177)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
>> at
>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
>> at
>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> Thanks
>> Anurag
>>
>>
>

Re: problems using property snitch file

[Anurag Gujral]

IPAddress1=DC1:RAC1
IPAddress1=DC2:RAC1
IPAddress1=DC2:RAC1
IPAddress1=DC2:RAC1

Thanks
Anurag

On Sun, May 15, 2011 at 4:52 PM, aaron morton wrote:

> Whats does your cassandra-topology.properties file look like ?
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 14 May 2011, at 11:48, Anurag Gujral wrote:
>
> > Hi All,
> >   I have built a cassandra cluster which is using property snitch
> file to specify my network topology to cassandra.The keyspace I have use
> network topology strategy.
> >
> > When I tried to add new node to this cluster,I get the following error:
> > RROR [main] 2011-05-13 23:45:44,152 AbstractCassandraDaemon.java (line
> 333) Exception encountered during startup.
> > java.lang.NullPointerException
> > at
> org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:96)
> > at
> org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:87)
> > at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
> > at
> org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:230)
> > at
> org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:195)
> > at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
> > at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
> > at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
> > at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
> > at
> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
> > at
> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
> > at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
> >
> >
> > Please Advise.
> >
> >
> > Thanks
> > Anurag
> >
>
>

Re: Converting separate clusters in mutliple dc to one cluster across multiple dcs

[Anurag Gujral]

Hi Aaron,
  Thanks for your mail. I am leaning towards making a new
cluster since I am using cassandra version
0.7.3 and the fix you have put in is in cassandra 0.7.5, and I am less
adventurous primarily because it is live system and service disruption is
not what is desirable.

I have two more questions for you:

a)Is there a way to specify snitch in cassandra per CF rather than one
snitch for all column family.

b)I have a weird issue with 0.7.3 when using property file snitch and a new
node is added to an existing running
cassandra cluster(which uses property file snitch) I have to add new node's
IP to the cassandra-topology.properties of all the existing  cassandra nodes
otherwise the new node gives error
ERROR [main] 2011-05-13 23:41:30,821 AbstractCassandraDaemon.java (line 333)
Exception encountered during startup.
java.lang.NullPointerException
at
org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:96)
at
org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:87)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:230)
at
org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:195)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
java.lang.NullPointerException
at
org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:96)
at
org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:87)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getAddressRanges(AbstractReplicationStrategy.java:196)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getPendingAddressRanges(AbstractReplicationStrategy.java:230)
at
org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:195)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:403)
at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194)
at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:316)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)

Thanks for all your help.

Anurag


On Sun, May 15, 2011 at 3:56 PM, aaron morton wrote:

> Rename a cluster
> http://wiki.apache.org/cassandra/FAQ#clustername_mismatch
> see also https://issues.apache.org/jira/browse/CASSANDRA-2632
>
> The best approach will depend on
> - how much data you have
> - if this is a running system
> - are the schemas identical
> - are the individual clusters using the NetworkTopologyStrategy ?
> - how much excitement you like.
>
> Initial
> thoughts, doubt this is correct...
>
> - change the tokens in each dc
> - for a dc that will have the others join it, change
> cassandra-topology.properties to put the new nodes in the new dc
> - for a dc joining another, follow the procedure to change the name on each
> node, update the seed list and update the topology to include the dc it is
> joining, remove the schema and migrations SStables, as you do a rolling
> restart
> - update the seed list on the dc the others joined
> - confirm there is one schema def
> - update the replication factor of the schema to put replicas in each DC
> http://wiki.apache.org/cassandra/Operations#Replication
> - nodetool repair to change the RF and then cleanup
>
> There will be a lot of data moves and I doubt I got it right, is there any
> chance of spinning up a whole new cluster and reloading the data ?
>
> I can give it some more thought later if you need.
>
> Hope that helps.
>
> -
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 14 May 2011, at 04:55, Anurag Gujral wrote:
>
> Hi All,
> I  have 3 separate cassandra clusters running in multiple data
> centers which I want