Digg's data model
I am a newbie to bigtable like model and have a question as follows. Take Digg as an example, I want to find a list users who dug a URL and also want to find a list of URLs a user dug. How should the data model look like for the queries to be efficient? If I use the username and the URL for two rows, when a user digs a URL, I will have to update two rows so I need a transaction to keep data consistent. Any thoughts? Thanks, Gary
LazyBoy question
I am trying out the lazyboy library to access cassandra, I was able to get the data in and out using Record save/load functions. Is there a way to get a slice, or all the records under a CF so I can iterate? It is probably a naive question, as I am just getting into this field Thanks, Gary
Upgrade and Consistency Question
I had a single-node instance (0.5) that has some data (1 SCF has 1 key and about 500k columns). I upgraded the node to 0.6 release yesterday and added another 0.6 node to try out clustering. The two nodes seem to work fine (discovered each other), but reading the data on either node only returns a tiny fraction (using pycassa). I had set the replication factor to be 2. I shut down the two nodes and reverted the first node back to 0.5, and can read the data out fine. Then I tried to read data from that node with 0.6, and it is also fine, so it is not pycassa's problem. I wonder what is the right way to do clustering in my case and how long "eventual" consistency will take. I want to use one node to collect data and the other node for analysis. Thanks for your time.
NGCC 2017 Announcment
We are pleased to announce the 2017 Next Generation Cassandra Conference (NGCC) that will be held on Tuesday September 26th at the Geekdom Event Center in San Antonio, Texas. This one day conference is targed toward Apache Cassandra developers and large-scale cluster operators, but is opened to anyone interested in the Cassandra community. Presentation topics will cover the current state of project development as well as future development. The organizers have alotted a number of opened slots for presentations. So, if you would like to participate by giving a presentation, please complete this questionnaire [1]. Presentations will be selected by a committee formed from the Apache Cassandra PMC. For more information, or to register, please go to our Eventbrite page[2], where you can find a schedule of events as well as accomodation information. If you have any questions, please reach out to myself or Eric Evans ( john.eric.ev...@gmail.com). A special thanks to Instaclustr, who are sponsoring this event. Thanks, Gary. Please Note: Apache Cassandra is a trademark of the Apache Software Organization, which has neither endorsed or organized this event. https://docs.google.com/forms/d/e/1FAIpQLSf2spZG_5gf68Jf4m70SoZGNj2meEpUGUY9Ac08lGstE3N7VQ/viewform?usp=sf_link https://www.eventbrite.com/e/next-generation-cassandra-conference-ngcc-tickets-36160855091
Scala driver
Hi Could you recommend a Scala driver and share your experiences of using it. Im thinking if i use java driver in Scala directly. Thanks
Java sample code for non-blocking async query
Hello I'm looking for non-blocking async query sample code. The one I found in the following link is async query but blocking. Could anyone share such code? http://www.datastax.com/documentation/developer/java-driver/1.0/java-driver/asynchronous_t.html Thanks Gary
Re: Scala driver
Thanks Jan. I decided to use Java driver directly. It's not hard to use. On Sun, Aug 31, 2014 at 1:08 AM, Jan Algermissen wrote: > Hi Gary, > > On 31 Aug 2014, at 07:19, Gary Zhao wrote: > > Hi > > Could you recommend a Scala driver and share your experiences of using it. > Im thinking if i use java driver in Scala directly > > > I am using Martin’s approach without any problems: > > https://github.com/magro/play2-scala-cassandra-sample > > The actual mapping from Java to Scala futures for the async case is in > > > https://github.com/magro/play2-scala-cassandra-sample/blob/master/app/models/Utils.scala > > HTH, > > Jan > > > > Thanks > > >
Dependency Hell: STORM 0.9.2 and Cassandra 2.0
Hello Anyone encountered the following issue and any workaround? Our Storm topology was written in Clojure. Our team is upgrading one of our storm topology from using cassandra 1.2 to cassandra 2.0, and we have found one problem that is difficult to tackle. Cassandra 2.0Java driver requires google guava 1.6. Unfortuanately, storm 0.9.2 provides a lower version. Because of that, a topology will not be able to contact Cassandra databases. Thanks Gary
Cassandra at Apache: Big Data, Sept 28-30 in Budapest
Hi everybody! Apache is splitting up ApacheCon into two conferences. One will focus on Big Data technologies, the other will focus on everything else. Apache: Big Data will be Sept 28-30 in Budapest, Hungary. The timing is a bit unfortunate, as it is the week following the Cassandra Summit in the USA. If you are in Europe and find yourself unable to attend the Cassandra Summit but would like to have a similar experience, I encourage you to respond to the call for presentations to Apache: Big Data. It would be great to have enough presentations for a Cassandra track. See the CFP at http://events.linuxfoundation.org/events/apache-big-data-europe/program/cfp You have until 10 July to put your proposals together. You will be notified by 24 July whether or not your proposal is accepted. Kind Regards, Gary Dusbabek
java.lang.OutOfMemoryError: Java heap space
Hello I have a three nodes cluster. I noticed one node was always down. Restarting Cassandra fixes it but it will go down again after a couple of days. I'm pretty new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as below. INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java (line 141) Stop listening to thrift clients ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:4,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:3,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) Exception in thread Thread[GossipStage:2,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) Exception in thread Thread[Thread-222,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.OutOfMemoryError: Java heap space ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) Exception in thread Thread[GossipTasks:1,5,main] java.lang.IllegalThreadStateException at java.lang.Thread.start(Thread.java:704) at org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 181) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 1251) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 MessagingService.java (line 667) Waiting for messaging service to quiesce Thanks Gary
Re: java.lang.OutOfMemoryError: Java heap space
BTW, the CPU usage on this node is pretty high, but data size is pretty small. PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao wrote: > Hello > > I have a three nodes cluster. I noticed one node was always down. > Restarting Cassandra fixes it but it will go down again after a couple of > days. I'm pretty new to Cassandra so I'm wondering how I should > troubleshoot it. Logs is as below. > > INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 > ThriftServer.java (line 141) Stop listening to thrift clients > ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line > 196) Exception in thread Thread[GossipStage:4,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java > (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line > 196) Exception in thread Thread[GossipStage:3,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line > 196) Exception in thread Thread[GossipStage:2,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) > Exception in thread Thread[Thread-222,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line > 196) Exception in thread Thread[GossipTasks:1,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line > 196) Exception in thread Thread[GossipTasks:1,5,main] > java.lang.IllegalThreadStateException > at java.lang.Thread.start(Thread.java:704) > at > org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) > at > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java > (line 181) Stop listening for CQL clients > INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java > (line 1251) Announcing shutdown > INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 > MessagingService.java (line 667) Waiting for messaging service to quiesce > > Thanks > Gary >
Re: java.lang.OutOfMemoryError: Java heap space
Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue or we should use physical machines? On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav wrote: > Are the virtual machines? The last time I had this issues was because of > VMWare "ballooning". > If not, what versions of Cassandra and Java are you using? > > > On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao wrote: > >> BTW, the CPU usage on this node is pretty high, but data size is pretty >> small. >> >> PID USERNAME THR PRI NICE SIZE RES SHR STATE TIMECPU COMMAND >> 28674 cassandr 89 250 9451M 8970M 525M sleep 32.1H 329% java >> >> UN 8.92 GB256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 >> UN 5.32 GB256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 >> DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 >> >> >> On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao wrote: >> >>> Hello >>> >>> I have a three nodes cluster. I noticed one node was always down. >>> Restarting Cassandra fixes it but it will go down again after a couple of >>> days. I'm pretty new to Cassandra so I'm wondering how I should >>> troubleshoot it. Logs is as below. >>> >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 >>> ThriftServer.java (line 141) Stop listening to thrift clients >>> ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:4,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 >>> CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/ >>> 10.20.132.44,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:3,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:2,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line >>> 196) Exception in thread Thread[Thread-222,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipTasks:1,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipTasks:1,5,main] >>> java.lang.IllegalThreadStateException >>> at java.lang.Thread.start(Thread.java:704) >>> at >>> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) >>> at >>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java >>> (line 181) Stop listening for CQL clients >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java >>> (line 1251) Announcing shutdown >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 >>> MessagingService.java (line 667) Waiting for messaging service to quiesce >>> >>> Thanks >>> Gary >>> >> >> >
How to monitor progress of nodetool repair?
Running v0.8.2, I can't see how to monitor the status/progress of a nodetool repair. Any advice? The nodetool repair command from the command line is not returning, so I assume it's still running. But there's little CPU or disk activity. Using jconsole to look at the AntiEntropyStage attributes, the CompletedTasks has a non-zero value, but all the other attributes like ActiveCount, PendingTasks, CurrentlyBlockedTasks are at 0. nodetool tpstats reports 0 for active, pending, blocked for all attributes nodetool netstats has a few pending tasks occasionally, and the completed tasks are going up. system.log at INFO level has the initial "Sending AEService tree" message, but nothing else. -gary
cassandra on laptop
I'm running an underpowered laptop (ubuntu) for development work. Installing Cassandra was easy, and getting the twissandra example app up and working was also easy. Here's the problem: after about a day of letting it run (with no load generated to webapp or db), my laptop now becomes unresponsive. If I'm patient, I can shutdown the cassandra service and return things to normal. In each of these cases, the cassandra process is eating up almost all memory, and everything goes to swap. I can't develop against Cassandra in this environment. I know it isn't set up by default to work efficiently on a meager laptop, but are there some common setting somewhere that I can just tweak to make life not be so miserable? I just want to play with it and try it out for this project I'm working on, but that's impractical with default settings. I'm going to have to flee to mongodb or something not as good... I'm also a little nervous about this running on a server now -- I've read enough to understand that by default it's set up to eat lots of memory, and I'm fine with that... but it just lends itself to all the java bigotry that some of us accumulate over the years. Anyway, if someone can give me a pointer on how to set up to run on a laptop in a development setting, big thanks. Thanks! Gary
Why SSTable is sorted by tokens instead of row keys?
I want to save time series event logs into Cassandra, and I need to load them by key range (row key is time-based). But we can't use RandomPartitioner in this way, while OrderPreservingPartitioner leads to hot spot problem. So I wonder why Cassandra save SSTable by sorted row tokens instead of keys: if rows in SSTable are sorted by keys, it should be quite easy to return rows by key range -- token should be used to determine which node contains the data. For key range requests, Cassandra could ask every node for that range of rows, merge them and return to the caller. -- regards, Gary Shi
Re: Why SSTable is sorted by tokens instead of row keys?
Thank you. I agree that request "lots of" machines process a single query could be slow, if there are hundreds of them instead of dozens. Will a cluster of e.g. 4-20 nodes behave well if we spread the query to all nodes? Many articles suggest model TimeUUID in columns instead of rows, but since only one node can serve a single row, won't this lead to hot spot problems? 在 2011-11-4 晚上10:28，"Sylvain Lebresne" 写道： > On Fri, Nov 4, 2011 at 1:49 PM, Gary Shi wrote: > > I want to save time series event logs into Cassandra, and I need to load > > them by key range (row key is time-based). But we can't use > > RandomPartitioner in this way, while OrderPreservingPartitioner leads to > hot > > spot problem. > > > > So I wonder why Cassandra save SSTable by sorted row tokens instead of > keys: > > if rows in SSTable are sorted by keys, it should be quite easy to return > > rows by key range -- token should be used to determine which node > contains > > the data. For key range requests, Cassandra could ask every node for that > > range of rows, merge them and return to the caller. > > Without going for exhaustiveness: > - Requesting every node is not too scalable. Cassandra is built to target > the > 'lots of cheap machines' kind of cluster, so that kind of operation is > going the > exact opposite way. In other words, that would be slow enough that you're > better off modeling this using columns for time series. > - That would make topology operations (bootstrap, move, decommission) > much more costly, because we wouldn't be able to tell which keys to move > unless we iterate over all the data each time. > > -- > Sylvain > > > > > -- > > regards, > > Gary Shi > > >
CounterColumn java.lang.AssertionError: Wrong class type.
Using v1.0.7, we see many of the following errors. Any thoughts on why this is occurring? Thanks in advance. -gary ERROR [ReadRepairStage:9] 2012-02-24 18:31:28,623 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadRepairStage:9,5,main] java.lang.AssertionError: Wrong class type. at org.apache.cassandra.db.CounterColumn.diff(CounterColumn.java:112) at org.apache.cassandra.db.ColumnFamily.diff(ColumnFamily.java:230) at org.apache.cassandra.db.ColumnFamily.diff(ColumnFamily.java:309) at org.apache.cassandra.service.RowRepairResolver.scheduleRepairs(RowRepairReso lver.java:117) at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.jav a:94) at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCa llback.java:54) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 10) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 03) at java.lang.Thread.run(Thread.java:722) ERROR [ReadRepairStage:9] 2012-02-24 18:31:28,625 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ReadRepairStage:9,5,main] -- >From cassandra-cli "show schema", I think the relevant CF is: create column family QOSCounters with column_type = 'Standard' and comparator = 'UTF8Type' and default_validation_class = 'CounterColumnType' and key_validation_class = 'UTF8Type' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 20.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 604800 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and row_cache_provider = 'SerializingCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';
Unsupported major.minor version 51.0
Hello I just saw this error. Anyone knows how to fix it? [root@gary-vm1 apache-cassandra-2.0.0]# bin/cassandra -f xss = -ea -javaagent:bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms4014M -Xmx4014M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError -Xss180k Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/cassandra/service/CassandraDaemon : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.cassandra.service.CassandraDaemon. Program will exit. [root@gary-vm1 apache-cassandra-2.0.0]# java -version java version "1.6.0_24" Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) Thanks Gary
Re: Unsupported major.minor version 51.0
Thanks Jason. Does Node.js work with 2.0? I'm wondering which version should I run. Thanks. On Tue, Sep 17, 2013 at 8:24 PM, Jason Wee wrote: > cassandra 2.0, then use oracle or open jdk version 7. > > Jason > > > On Wed, Sep 18, 2013 at 11:21 AM, Gary Zhao wrote: > >> Hello >> >> I just saw this error. Anyone knows how to fix it? >> >> [root@gary-vm1 apache-cassandra-2.0.0]# bin/cassandra -f >> xss = -ea -javaagent:bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities >> -XX:ThreadPriorityPolicy=42 -Xms4014M -Xmx4014M -Xmn400M >> -XX:+HeapDumpOnOutOfMemoryError -Xss180k >> Exception in thread "main" java.lang.UnsupportedClassVersionError: >> org/apache/cassandra/service/CassandraDaemon : Unsupported major.minor >> version 51.0 >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:616) >> at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:58) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:197) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248) >> Could not find the main class: >> org.apache.cassandra.service.CassandraDaemon. Program will exit. >> [root@gary-vm1 apache-cassandra-2.0.0]# java -version >> java version "1.6.0_24" >> Java(TM) SE Runtime Environment (build 1.6.0_24-b07) >> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) >> >> Thanks >> Gary >> > >
Re: cassandra version update and my cluster
0.6->0.7 will require a full cluster restart. 0.7->0.8 will require a rolling restart, so long as CASSANDRA-1015 is completed. keyspace/cf definitions will be readable across major versions, or there will be a way provided to migrate them if that is not possible. Gary. On Wed, Dec 1, 2010 at 19:50, Nick Santini wrote: > Hi, > this is an hypothetical situation highly likely to happen: > I have a cassandra 0.7 cluster, filled with production data, and I want to > upgrade cassandra to the 0.8 version (and so on as new versions get > released) > what happen with my data and keyspace / column families definitions? whats > the process to upgrade cassandra in a production cluster? > > Thahks > > > Nicolas Santini
Re: Running multiple instances on a single server --micrandra ??
On Tue, Dec 7, 2010 at 20:25, Edward Capriolo wrote: > I am quite ready to be stoned for this thread but I have been thinking > about this for a while and I just wanted to bounce these ideas of some > guru's. > > ... > > The upsides ? > 1) Since disk/instance failure only degrades the overall performance > 1/6th (RAID0 you lost the entire node) (RAID5 still takes a hit when > down a disk) > 2) Moves and joins have less work to do > 3) Can scale up a single node by adding a single disk to an existing > system (assuming the ram and cpu is light) > 4) OPP would be "easier" to balance out hot spots (maybe not on this > one in not an OPP) > Sorry for chiming in so late, but another benefit is that it amortizes stop-the-world garbage collection across 6 jvms. > What does everyone thing? Does it ever make sense to run this way? > I think it would be a great way of utilizing CPU and memory, assuming you can come up with the IO bandwidth. Gary.
Re: Having trouble getting cassandra to stay up
You might want to try starting over. Configure your initial keyspaces
in conf/cassandra.yaml and load them into your cluster with
bin/schematool.
That nasty stack trace indicates the server is getting data that is
not formatted the way it expects. Please verify that your cassandra
servers are both running the same version.
Your earlier error when adding a keyspace through pycassa was
confusing. You stated that you tried to create a keyspace, but the
error you pasted appeared to error in a drop_keyspace call. Something
doesn't add up.
Gary.
On Fri, Dec 24, 2010 at 11:48, Alex Quan wrote:
> Sorry but I am not sure how to answer all the question that you have posed
> since a lot of the stuff I am working with is quite new to me and I haven't
> use many of the tools that are talked about but I will try my best to answer
> the question to the best of my knowledge. I am trying to get the cassandra
> to run between 2 nodes that are both Amazon's ec2 micro instances, I believe
> they are using a 64 bit linux ubuntu 10.01 using java version 1.6.0_23. When
> I said killed it was what was outputted into the console when the process
> died so I am not sure what that exactly means. Here is some of the info
> before cassandra went down:
>
> ring:
>
> Address Status State Load Owns
> Token
>
> 111232248257764777335763873822010980488
> 10.127.155.205 Up Normal 85.17 KB 59.06%
> 41570168072350555868554892080805525145
> 10.122.123.210 Up Normal 91.1 KB 40.94%
> 111232248257764777335763873822010980488
>
> vmstat before cassandra is up:
>
> procs ---memory-- ---swap-- -io -system--
> cpu
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 0 0 328196 632 13936 0 0 12 4 13 1 0 0 99
> 0
>
> vmstat after cassandra is up:
>
> procs ---memory-- ---swap-- -io -system--
> cpu
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 0 2 0 5660 116 10312 0 0 12 4 13 1 0 0 99
> 0
>
> Then after I run a line like sys.create_keyspace('testing', 1) in pycassa
> with the connections setup to point to my machine I get the following error:
>
>
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/usr/local/lib/python2.6/dist-packages/pycassa-1.0.2-py2.6.egg/pycassa/system_manager.py",
> line 365, in drop_keyspace
> schema_version = self._conn.system_drop_keyspace(keyspace)
> File
> "/usr/local/lib/python2.6/dist-packages/pycassa-1.0.2-py2.6.egg/pycassa/cassandra/Cassandra.py",
> line 1255, in system_drop_keyspace
> return self.recv_system_drop_keyspace()
> File
> "/usr/local/lib/python2.6/dist-packages/pycassa-1.0.2-py2.6.egg/pycassa/cassandra/Cassandra.py",
> line 1266, in recv_system_drop_keyspace
> (fname, mtype, rseqid) = self._iprot.readMessageBegin()
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py",
> line 126, in readMessageBegin
> sz = self.readI32()
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/protocol/TBinaryProtocol.py",
> line 203, in readI32
> buff = self.trans.readAll(4)
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py",
> line 58, in readAll
> chunk = self.read(sz-have)
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py",
> line 272, in read
> self.readFrame()
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py",
> line 276, in readFrame
> buff = self.__trans.readAll(4)
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/transport/TTransport.py",
> line 58, in readAll
> chunk = self.read(sz-have)
> File
> "/usr/local/lib/python2.6/dist-packages/thrift05-0.5.0-py2.6-linux-x86_64.egg/thrift/transport/TSocket.py",
> line 108, in read
> raise TTransportException(type=TTransportException.END_OF_FILE,
> message='TSocket read 0 bytes')
> thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
>
> and then cassandra on the machine dies, here is the log some of the log of
> the machine that died:
>
> INFO [FlushWriter:1] 2010-12-24 03:24:01,999 Memtable.java (line 162)
> Completed flushing /var/lib/cassandra/data/system/L
Re: cassandra.yaml customization per node
Funny you should mention that. I just put my half-baked solution to this problem on github today. https://github.com/gdusbabek/cassandra-maker. It generates cassandra.yaml files (and entire config directories too) that are based on the svn defaults. It should be easy to customize it to fit your needs. As for the notion of yaml includes, a quick google made me think it's not possible. Gary. On Thu, Dec 30, 2010 at 14:34, B. Todd Burruss wrote: > how are folks customizing the cassandra.yaml for each node in the cluster. > specifically the token and IP address. > > with XML i used entities, but i'm not familiar with YAML. does yaml support > the same concept? or any sort of textual substitution? > > thx >
Re: Newbie Replication/Cluster Question
It is impossible to properly bootstrap a new node into a system where there are not enough nodes to satisfy the replication factor. The cluster as it stands doesn't contain all the data you are asking it to replicate on the new node. Gary. On Thu, Jan 13, 2011 at 13:13, Mark Moseley wrote: > I'm just starting to play with Cassandra, so this is almost certainly > a conceptual problem on my part, so apologies in advance. I was > testing out how I'd do things like bring up new nodes. I've got a > simple 2-node cluster with my only keyspace having > replication_factor=2. This is on 32-bit Debian Squeeze. Java==Java(TM) > SE Runtime Environment (build 1.6.0_22-b04). This is using the > just-released 0.7.0 binaries. Configuration is pretty minimal besides > using SimpleAuthentication module. > > The issue is that whenever I kill a node in the cluster and wipe its > datadir (i.e. rm -rf /var/lib/cassandra/*) and try to bootstrap it > back into the cluster (and this occurs in both the scenario of both > nodes being present during the writing of data as well as only a > single node being up during writing of data), it seems to join the > cluster and chug along till it keels over and dies with this: > > INFO [main] 2011-01-13 13:56:23,385 StorageService.java (line 399) > Bootstrapping > ERROR [main] 2011-01-13 13:56:23,402 AbstractCassandraDaemon.java > (line 234) Exception encountered during startup. > java.lang.IllegalStateException: replication factor (2) exceeds number > of endpoints (1) > at > org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:204) > at > org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:198) > at > org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:417) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:361) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:161) > at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) > Exception encountered during startup. > java.lang.IllegalStateException: replication factor (2) exceeds number > of endpoints (1) > at > org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:204) > at > org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:198) > at > org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83) > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:417) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:361) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:161) > at > org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55) > at > org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:217) > at > org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134) > > > Seems like something of a chicken-or-the-egg problem of it not liking > there only being 1 node but not letting node 2 join. Being that I've > been messing with Cassandra for only a couple of days, I'm assuming > I'm doing something wrong, but the only google'ing I can find for the > above error is just a couple of 4+ month-old tickets that all sound > resolved. It's probably worth mentioning that if both nodes are > started when I create the keyspace, the cluster appears to work just > fine and I can start/stop either node and get at any piece of data. > > The nodetool ring output looks like this: > >> Prior to starting 10.1.58.4 and then for a while after startup > Address Status State Load Owns Token > 10.1.58.3 Up Normal 524.99 KB 100.00% > 74198390702807803312208811144092384306 > >> 10.1.58.4 seems to be joining > Address Status State Load Owns Token > > 74198390702807803312208811144092384306 > 10.1.58.4 Up Joining 72.06
Moving data
I have several large SQL Server 2005 tables. I need to load the data in these tables into Cassandra. FYI, the Cassandra installation is on a linux server running CentOS. Can anyone suggest the best way to accomplish this? I am a newbie to Cassandra, so any advice would be greatly appreciated. Best, Gary
Re: ApplicationState Schema has drifted from DatabaseDescriptor
Aaron,
It looks like you're experiencing a side-effect of CASSANDRA-2083.
There was at least one place (when node B received updated schema from
node A) where gossip was not being updated with the correct schema
even though DatabaseDescriptor had the right version. I'm pretty sure
this is what you're seeing.
Gary.
On Wed, Feb 9, 2011 at 00:08, Aaron Morton wrote:
> I noticed this after I upgraded one node in a 0.7 cluster of 5 to the latest
> stable 0.7 build "2011-02-08_20-41-25" (upgraded node was jb-cass1 below).
> This is a long email, you can jump to the end and help me out by checking
> something on your 07 cluster.
> This is the value from o.a.c.gms.FailureDetector.AllEndpointStates on
> jb-cass05 9114.67)
> /192.168.114.63 X3:2011-02-08_20-41-25
> SCHEMA:2f555eb0-3332-11e0-9e8d-c4f8bbf76455 LOAD:2.84182972E8
> STATUS:NORMAL,0
> /192.168.114.64 SCHEMA:2f555eb0-3332-11e0-9e8d-c4f8bbf76455
> LOAD:2.84354156E8 STATUS:NORMAL,34028236692093846346337460743176821145
> /192.168.114.66 SCHEMA:075cbd1f-3316-11e0-9e8d-c4f8bbf76455
> LOAD:2.59171601E8 STATUS:NORMAL,102084710076281539039012382229530463435
> /192.168.114.65 SCHEMA:075cbd1f-3316-11e0-9e8d-c4f8bbf76455
> LOAD:2.70907168E8 STATUS:NORMAL,68056473384187692692674921486353642290
> jb08.wetafx.co.nz/192.168.114.67
> SCHEMA:075cbd1f-3316-11e0-9e8d-c4f8bbf76455 LOAD:1.155260665E9
> STATUS:NORMAL,136112946768375385385349842972707284580
> Notice the schema for nodes 63 and 64 starts with 2f55 and for 65, 66 and 67
> it starts with 075.
> This is the output from pycassa calling describe_versions when connected to
> both the 63 (jb-cass1) and 67 (jb-cass5) nodes
> In [34]: sys.describe_schema_versions()
> Out[34]:
> {'2f555eb0-3332-11e0-9e8d-c4f8bbf76455': ['192.168.114.63',
> '192.168.114.64',
> '192.168.114.65',
> '192.168.114.66',
> '192.168.114.67']}
> It's reporting all nodes on the 2f55 schema. The SchemaCheckVerbHandler is
> getting the value from DatabaseDescriptor. FailureDetector MBean is getting
> them from Gossiper.endpointStateMap . Requests are working though, so the
> CFid's must be matching up.
> Commit https://github.com/apache/cassandra/commit/ecbd71f6b4bb004d26e585ca8a7e642436a5c1a4 added
> code to the 0.7 branch in the HintedHandOffManager to check the schema
> versions of nodes it has hints for. This is now failing on the new node as
> follows...
> ERROR [HintedHandoff:1] 2011-02-09 16:11:23,559 AbstractCassandraDaemon.java
> (line
> org.apache.cassandra.service.AbstractCassandraDaemon$1.uncaughtException(AbstractCassandraDaemon.java:114))
> Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach
> schema agreement with /192.168.114.64 in 6ms
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement with
> /192.168114.64 in 6ms
> at
> org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:256)
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:267)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88)
> at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:391)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 3 more
> (the nodes can all see each other, checked with notetool during the 60
> seconds)
> If I restart one of the nodes with the 075 schema (without upgrading it) it
> reads the schema from the system tables and goes back to the 2f55 schema.
> e.g. the 64 node was also on the 075 schema, I restarted and it moved to the
> 2f55 and logged appropriately. While writing this email I checked again with
> the 65 node, and the schema if was reporting to other nodes changed after a
> restart from 075 to 2f55
> INFO [main] 2011-02-09 17:17:11,457 DatabaseDescriptor.java (line
> org.apache.cassandra.config.DatabaseDescriptor) Loading schema version
> 2f555eb0-3332-11e0-9e8d-c4f8bbf76455
> I've been reading the code for migrations and gossip don't have a theo
Re: Keyspace additions are not replicated to one node in the cluster
The first thing you should try is to restart the node that is not
syncing. That will reset the version it gossips, which was not always
getting updated (fixed in CASSANDRA-2083).
Gary.
On Tue, Feb 15, 2011 at 18:25, Shu Zhang wrote:
> Hi, a node in my cassandra cluster will not accept keyspace additions applied
> to other nodes. In its logs, it says:
>
> DEBUG [MigrationStage:1] 2011-02-15 15:39:57,995
> DefinitionsUpdateResponseVerbHandler.java (line 71) Applying AddKeyspace from
> {X}
> DEBUG [MigrationStage:1] 2011-02-15 15:39:57,995
> DefinitionsUpdateResponseVerbHandler.java (line 79) Migration not applied
> Previous version mismatch. cannot apply.
>
> My cassandra nodes' version is 0.7-rc4.
>
> Does anyone know how I can recover from this problem? I'm fine with this node
> being synced to whatever data definition is defined on the rest of the
> cluster.
>
> Thanks,
> Shu
Re: rename index
I haven't tried this, but it should work in theory... Keep the wrongly named index for now. Create the rightly named index, then bring down the node and swap file names for the indexes. After that, you can safely drop the wrongly named index. Unfortunately, you need to do this for each node. Gary. On Wed, Feb 16, 2011 at 11:15, Roland Gude wrote: > Hi, > > unfortiunately i made a copy paste error and created two indexes called > “myindex” on different columnfamilies. > > What can I do to fix this? > > > > Below the output from describe keyspace > > > > ColumnFamily: A > > Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType > > Row cache size / save period: 0.0/0 > > Key cache size / save period: 20.0/14400 > > Memtable thresholds: 1.1203125/239/60 > > GC grace seconds: 864000 > > Compaction min/max thresholds: 4/32 > > Read repair chance: 1.0 > > Column Metadata: > > Column Name: --1000-- > > Validation Class: org.apache.cassandra.db.marshal.UTF8Type > > Index Name: MyIndex > > Index Type: KEYS > > ColumnFamily: B > > Columns sorted by: org.apache.cassandra.db.marshal.TimeUUIDType > > Row cache size / save period: 0.0/0 > > Key cache size / save period: 20.0/14400 > > Memtable thresholds: 1.1203125/239/60 > > GC grace seconds: 864000 > > Compaction min/max thresholds: 4/32 > > Read repair chance: 1.0 > > Column Metadata: > > Column Name: --1000-- > > Validation Class: org.apache.cassandra.db.marshal.UTF8Type > > Index Name: MyIndex > > Index Type: KEYS > > > > -- > > YOOCHOOSE GmbH > > > > Roland Gude > > Software Engineer > > > > Im Mediapark 8, 50670 Köln > > > > +49 221 4544151 (Tel) > > +49 221 4544159 (Fax) > > +49 171 7894057 (Mobil) > > > > > > Email: roland.g...@yoochoose.com > > WWW: www.yoochoose.com > > > > YOOCHOOSE GmbH > > Geschäftsführer: Dr. Uwe Alkemper, Michael Friedmann > > Handelsregister: Amtsgericht Köln HRB 65275 > > Ust-Ident-Nr: DE 264 773 520 > > Sitz der Gesellschaft: Köln > >
Re: Running multiple compactions concurrently
If your cluster has the overall IO capacity to perform a simultaneous compaction on every node and still adequately service reads and writes, then yes. If you're concerned about availability, your best bet will be to stagger the compactions. Gary. On Fri, Feb 25, 2011 at 04:24, Daniel Josefsson wrote: > We experienced the java.lang.NegativeArraySizeException when upgrading to > 0.7.2 in staging. The proposed solution (running compaction) seems to have > solved this. However it took a lot of time to run. > > Is it safe to invoke a major compaction on all of the machines concurrently? > > I can't see a reason why it wouldn't, but I want to be sure :) > > Thanks, > Daniel > >
Re: Strange streaming behaviour
Martin, Please create a ticket and include the relevant parts of your storage-conf. To summarize, the output gives you the impression that bootstrap has completed normally, but when you check, it appears to be hung on the receiving nodes? Do you mind turning debug on and and seeing if you can reproduce? The strange part is that the source node doesn't think it's streaming at all. Gary. On Mon, May 3, 2010 at 02:30, Dr. Martin Grabmüller wrote: > Hello list, > > I encountered a problem with streaming in my test cluster but > found nothing like this in JIRA or on the list. > > I'm running a test cluster of three nodes, RF=3, Cassandra 0.6.1. > I started the first node and inserted some data, then bootstrapped > the other machines one after the other. No problems here. > > Then, I inserted data over the weekend and checked back this morning, > finding a lot of -tmp-files in the data directory of the two bootstrapepd > machines which seem to be leftovers from streaming. > > Interesting thing is, they are mostly all about the same size on each node. > Additionally, > this size is the same as the size of some file to be streamed (see the output > of > nodetool streams at the end of this post): > > [Node 2:] > > -rw-r--r-- 1 cassandra cassandra 2360 2010-05-03 07:39 > AccountList-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 2360 2010-05-03 00:39 > AddressList-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 2360 2010-05-03 07:53 > IndexQueue-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 2360 2010-05-03 05:19 > MailMetadata-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 2360 2010-05-03 00:53 > Statistics-tmp-7-Data.db > > [Node 3:] > > -rw-r--r-- 1 cassandra cassandra 10994 2010-05-03 07:39 > AccountList-tmp-6-Data.db > -rw-r--r-- 1 cassandra cassandra 325 2010-05-03 00:39 > AddressList-tmp-6-Data.db > -rw-r--r-- 1 cassandra cassandra 10994 2010-05-03 08:02 > CustomerList-tmp-6-Data.db > -rw-r--r-- 1 cassandra cassandra 10994 2010-05-03 07:53 > IndexQueue-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 10994 2010-05-03 01:02 > MailMetadata-tmp-7-Data.db > -rw-r--r-- 1 cassandra cassandra 10994 2010-05-03 00:53 > Statistics-tmp-7-Data.db > > Checking the logs, I find a few streaming-related error messages: > > [Node 1 (yes, the node without leftover files):] > > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 > DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor > java.lang.RuntimeException: java.io.FileNotFoundException: > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No > such file or directory) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No > such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > at > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > ERROR [MESSAGE-STREAMING-POOL:1] 2010-05-02 14:39:10,675 CassandraDaemon.java > (line 78) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:1,5,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No > such file or directory) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.io.FileNotFoundException: > /mnt/data000/cassandra/data/Archive/stream/LocalpartMapping-2-Data.db (No > such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:233) > at > org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:84) > at > org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:63) > at > org.apache.cassandra
Re: Add a new keyspace in cassandra cluster
Yes. You need to edit the configuration of every machine in the cluster. Gary. On Wed, May 5, 2010 at 06:27, Yu-Chun Chang wrote: > Dear all, > I'm a starter for playing Cassandra. I've set up three machines (called > PC_A, PC_B, PC_C) and run them as a Cassandra cluster. > I've inserted a record in the running cluster and saw it did work through > three machines. > Then I shutdowned Cassandra of PC_A, and modified storage-conf.xml to add > new keyspace (called new1). The new record and new > keyspace (new1) were displayed in PC_A after restarting Cassandra. But I > cannot see the new keyspace (new1) added in PC_B and > PC_C. Do I miss the procedure of running the cluster or have to modify the > same config in both PC_B and PC_C ? > Many thanks. > Best regards, > Yu-Chun > >
Re: why is streaming done in 32 MB chunks ?
64 MB gave Windows heartburn. We never considered 16 MB chunks. Gary. On Thu, May 6, 2010 at 02:30, vineet daniel wrote: > Hi > > Just out of curiosity want to know why streaming is done with 32MB chunks > and not with 16 or 64 MB chunks. Any specific reasons behind 32 MB or its > just like that ? > > > ___ > Vineet Daniel > ___ > > Let your email find you >
Re: Rolling upgrade
On Mon, May 10, 2010 at 17:01, Tatsuya Kawano wrote: > Hi, > > Does Cassandra support rolling restart recipe between minor version > upgrade? I mean rolling restart is a way to upgrade Cassandra version > or change configuration **without** bringing down the whole cluster. > The recipe will be something like killing a couple of nodes at a time > and starting them again with newer version (or newer configuration.) We aim for this, but fell short on the 0.6 -> 0.6.1 upgrade. A change made it in that broker inter-node communication across versions. Sorry. > > To make this possible, I think at least the RPC and SSTable format > should be compatible between older and newer versions. Do you have any > plan to change them in the future? Also, there might be some > exceptions and certain combinations of versions can't coexist in a > cluster (e.g. RPC or SSTable bug in the older version.) It will be > great if such incompatibilities are documented in the release note. > RPC has versioning built in, so clients should be able to communicate with servers that share the same major version. Note that this is not the same as the inter-node message format that nodes use to communicate with each other. The SSTable format has been stable since 0.4 (compatible with 0.5 and 0.6). There are some breaking changes scheduled for 0.7, but none of them have made it into trunk yet. The commit log disk format has changed and will continue to change from version to version. You should flush it prior to upgrading a node. 0.6.1 added 'drain' to the nodetool command to make this process easier. Gary. > Thanks, > Tatsuya >
Re: updating column names
On Tue, May 11, 2010 at 04:45, vd wrote: > Hi > > I have a column named colom. Can we update column name "colom" to > "column" during runtime or via API ? > This will require two operations: remove and insert. Gary.
Re: What is the optimal size of batch mutate batches?
On Tue, May 11, 2010 at 06:54, David Boxenhorn wrote: > My problem is that my rows are of very varying degrees of bushiness (i.e. > number of supercolums and columns per row). I inserted 592,500 rows > successfully, in a few minutes, and then I hit a batch of exceptionally > bushy rows and ran out of memory. > > Does anyone have any suggestions about how to deal with this problem? I think this is a matter of the size of data and not the "bushiness" rows (fabulous term, btw). For example, you could probably get away with bushy rows to a certain degree if the columns themselves were very small. So you should probably approach this problem from a "how many MB do I want to send in a single message" approach instead. When the rows get bushy, you should send less of them. Gary.
Re: How to setup multinode cluster over different servers through firewall
You didn't give a lot of details about the remoteness of the remote server. Remote hosts not be able to contact any host on the 192.168.*.* over the internet without routing support. If the remote host is on the same network as the 192.168.*.* host, it should work unless one of those hosts is running a local firewall. Gary. On Tue, May 11, 2010 at 02:54, Xu Hui Hui wrote: > He there! > > 1. I'm new to Cassandra. > 2. I created multinode cluster which worked well between two computers > in LAN (192.168.1.31, 192.168.1.33) with reference to the official > wiki. > 3. When creating multinode cluster between my computer (192.168.1.31) > and a remote server ( y.y.y.y ) it didn't work. > I referred to > http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx > to let JMX use a fixed port rather than randomized ones, but it > didn't help. > I also passed "-Djava.rmi.server.hostname=x.x.x.x" in my local > computer (where x.x.x.x my public IP, and on the router I forwarded > all necessary ports to 192.168.1.31), still not work. It seems random > ports are still used which are blocked my the firewall/router. > > Please point out what I have mistaken. Thanks a lot! > > -- > Xu Hui Hui >
Re: list of columns
We have get_count at the thrift level. You supply a predicate and it returns the number of columns that match. There is also multi_get_count, which is the same operation against multiple keys. Gary. On Thu, May 13, 2010 at 04:18, Bill de hOra wrote: > Admin question - is there a way to list the columns for a particular key? > > Bill >
Re: how does cassandra compare with mongodb?
Cassandra has always enforced the tiniest bit of schema. You basically define how you want your columns and subcolumns to be sorted within column families. You also name the column families and keyspaces. That's all though. The part that is changing is that the keyspaces and column families will no longer be defined statically at runtime. [column families | keyspaces] may be [added | dropped] on a live cluster. Think of it as [CREATE|DROP|ALTER] TABLE for Cassandra. Gary. On Thu, May 13, 2010 at 14:48, Steve Lihn wrote: > What is changing? A more flexible schema or no need to restart (some kind of > hot-reboot)? > > Mongo guys claims that Mongo's advantage is a schema-less design. Basically > you can have any data structure you want and you can change them anyway you > want. This is done in the name of "flexibility", but I am not sure this is a > good practice. People argued for years that perl is bad because it is > typeless and java is strong typed and is better. Now the java community is > developing a database like Mongo that is schema-less. How does this > complements the strong-type argument? > > The less requirement is put on database schema design, the more burden is > put on the application to maintain data integrity. Why is this a good trend? > Can someone kindly explain? > > Steve > > > > On Thu, May 13, 2010 at 1:22 PM, Vijay wrote: >> >> "Cassandra requires the schema to be defined before the database starts, >> MongoDB can have any schema at run-time just like a normal database." >> This is changing in 0.7 >> Regards, >> >> >
Re: batch mutate + deletion + slice range predicate unsupported
Yes--0.7. I aim to make it part of https://issues.apache.org/jira/browse/CASSANDRA-494 (remove_slice). Gary. On Thu, May 13, 2010 at 16:08, B. Todd Burruss wrote: > i just figured out that can't do a batch mutate + deletion that uses a slice > range predicate. is adding this functionality targeted for a particular > release? what i am trying to do is delete the first X columns in a row. i > can get around it by requesting all the columns in question and then > removing, but not as optimal. > > InvalidRequestException(why:Deletion does not yet support SliceRange > predicates.) > at > org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:15345) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:796) > at > org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:772) > >
Re: Avro Example Code
Be warned: avro support is experimental and incomplete. The avro system tests are probably the best place to start looking. http://svn.apache.org/repos/asf/cassandra/trunk/test/system Gary. On Fri, May 14, 2010 at 15:52, David Wellman wrote: > Does anyone have a good link or example code that we can use to spike on Avro > with Cassandra? > > Cheers.
Re: decommission and org.apache.thrift.TApplicationException: get_slice failed: unknown result
If I had to guess, I'd say that something at the transport layer had trouble. Possibly some kind of thrift hiccup that we haven't seen before. Your description makes it sound as if the decommission is proceeding normally though. Gary. On Tue, May 18, 2010 at 04:42, Ran Tavory wrote: > What's the correct way to remove a node from a cluster? > According to this page http://wiki.apache.org/cassandra/Operations a > decommission call should be enough. > When decommissioning one of the nodes from my cluster I see an error in the > client: > org.apache.thrift.TApplicationException: get_slice failed: unknown result > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:407) > at > org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:367) > > The client isn't talking to the decommissioned node, it's connected to > another node, so I'd expect all operations to continue as normal (although > slower), right? > I simply called "nodetool -h ... decommission" on the host and waited. After > a while, while the node is still decommissioning I saw the error at the > client. > The current state of the node is Decommissioned and it's not in the ring > now. It is still moving streams to other hosts, though. I can't be sure, > though whether the error happened during the time it was Leaving the ring or > was it already Decommissioned. > The server logs don't show something of note (no errors or warnings). > What do you think?
Re: New Changes in Cass 0.7 Thrift API Interface
On Thu, May 20, 2010 at 22:16, Arya Goudarzi wrote: > > P.S. By the way, if someone grants me access, I'd like to contribute to the > documentaions on Apache Cassandra. > I believe anybody can create a wiki account and make changes. Have at it! Gary.
Re: Questions regarding batch mutates and transactions
On Wed, May 26, 2010 at 04:45, Todd Nine wrote: > > Now, here is where I can't find what I need in the doc. In case 1, if my > mutation from biz op 2 were to fail during a batch mutate operation > encapsulating all mutations, does the batch mutation as a whole not get > executed, or would I still have the mutation from op 1 written to cassandra > while the op 2 write fails? > Thanks, batch_mutate makes no atomicity guarantees. It s intended to help avoiding many round-trips. It can fail half-way through leaving you with a partially completed batch. Gary.
Re: Error during startup
I was able to reproduce the error by staring up a node using RandomPartioner, kill it, switch to OrderPreservingPartitioner, restart, kill, switch back to RandomPartitioner, BANG! So it looks like you tinkered with the partitioner at some point. This has the unfortunate effect of corrupting your system table. I'm trying to figure out a way to detect this and abort before data is overwritten. Gary. On Sun, May 30, 2010 at 06:49, David Boxenhorn wrote: > I deleted the system/LocationInfo files, and now everything works. > > Yay! (...what happened?) > > On Sun, May 30, 2010 at 4:18 PM, David Boxenhorn wrote: >> >> I'm getting an "Expected both token and generation columns; found >> ColumnFamily" error during startup can anyone tell me what it is? Details >> below. >> >> Starting Cassandra Server >> Listening for transport dt_socket at address: >> INFO 16:14:33,459 Auto DiskAccessMode determined to be standard >> INFO 16:14:33,615 Sampling index for >> C:\var\lib\cassandra\data\system\LocationInfo-1-Data.db >> INFO 16:14:33,631 Removing orphan >> C:\var\lib\cassandra\data\Lookin2\Users-tmp-27-Index.db >> INFO 16:14:33,631 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db >> INFO 16:14:33,662 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-18-Data.db >> INFO 16:14:33,818 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db >> INFO 16:14:33,850 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db >> INFO 16:14:33,865 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db >> INFO 16:14:33,881 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-580-Data.db >> INFO 16:14:33,896 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-672-Data.db >> INFO 16:14:33,912 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-681-Data.db >> INFO 16:14:33,912 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-691-Data.db >> INFO 16:14:33,928 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestIdx-696-Data.db >> INFO 16:14:33,943 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Attractions-17-Data.db >> INFO 16:14:34,006 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-5-Data.db >> INFO 16:14:34,006 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestTrendsetterIdx-6-Data.db >> INFO 16:14:34,021 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-29-Data.db >> INFO 16:14:34,350 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-51-Data.db >> INFO 16:14:34,693 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-72-Data.db >> INFO 16:14:35,021 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-77-Data.db >> INFO 16:14:35,225 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-78-Data.db >> INFO 16:14:35,350 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-79-Data.db >> INFO 16:14:35,459 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\GeoSiteInterestPeerGroupIdx-80-Data.db >> INFO 16:14:35,459 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-1-Data.db >> INFO 16:14:35,475 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Taxonomy-2-Data.db >> INFO 16:14:35,475 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-30-Data.db >> INFO 16:14:35,631 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-35-Data.db >> INFO 16:14:35,771 Sampling index for >> C:\var\lib\cassandra\data\Lookin2\Content-40-Data.db >> INFO 16:14:35,959 Compacting >> [org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-19-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-20-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-21-Data.db'),org.apache.cassandra.io.SSTableReader(path='C:\var\lib\cassandra\data\Lookin2\Users-22-Data.db')] >> ERROR 16:14:35,975 Exception encountered during startup. >> java.lang.RuntimeException: Expected both token and generation columns; >> found ColumnFamily(LocationInfo [Generation:false:4...@4,]) >> at >> org.apache.cassandra.db.SystemTable.initMetadata(SystemTable.java:159) >> at >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:305) >> at >> org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:99) >> at >> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177) >> Exception encountered during startup. >> > >
Re: Read operation with CL.ALL, not yet supported?
Yuki, Can you file a jira ticket for this (https://issues.apache.org/jira/browse/CASSANDRA)? The wiki indicates that this should be allowed: http://wiki.apache.org/cassandra/API Regards, Gary. On Tue, Jun 1, 2010 at 21:50, Yuki Morishita wrote: > Hi, > > I'm testing several read operations(get, get_slice, get_count, etc.) with > various ConsistencyLevel and noticed that ConsistencyLevel.ALL is > "not yet supported" in most of read ops (other than get_range_slice). > > I've looked up code in StorageProxy#readProtocol and it seems > to be able to handle CL.ALL, but in thrift.CassandraServer#readColumnFamily, > there is code that just throws exception when consistency_level == ALL. > Is there any reason that CL.ALL is "not yet supported"? > > > Yuki Morishita > t:yukim (http://twitter.com/yukim) >
Re: OutOfMemoryError
Are you running "ant test"? It defaults to setting memory to 1G. If you're running them outside of ant, you'll need to set max memory manually. Gary. On Thu, Jun 3, 2010 at 10:35, Lev Stesin wrote: > Hi, > > I am getting OOM during load tests: > > java.lang.OutOfMemoryError: Java heap space > at java.util.HashSet.(HashSet.java:125) > at > com.google.common.collect.Sets.newHashSetWithExpectedSize(Sets.java:181) > at > com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:112) > at > com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:47) > at > com.google.common.collect.AbstractMultimap.createCollection(AbstractMultimap.java:155) > at > com.google.common.collect.AbstractMultimap.getOrCreateCollection(AbstractMultimap.java:207) > at > com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:194) > at > com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:80) > at com.google.common.collect.HashMultimap.put(HashMultimap.java:47) > at > org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:87) > at > org.apache.cassandra.service.StorageProxy.mutateBlocking(StorageProxy.java:204) > at > org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:451) > at > org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:361) > at > org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:1484) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1125) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > Any suggestions as to how to fix/ease the problem? Thanks. > > -- > Lev >
Re: OutOfMemoryError
It's set in the build file: But I'm not sure if you're using the build file or not. It kind of sounds like you are not. Gary. On Thu, Jun 3, 2010 at 11:24, Lev Stesin wrote: > Gary, > > Is there a directive to set it? Or should I modify the cassandra > script itself? Thanks. > > Lev. > > On Thu, Jun 3, 2010 at 10:48 AM, Gary Dusbabek wrote: >> Are you running "ant test"? It defaults to setting memory to 1G. If >> you're running them outside of ant, you'll need to set max memory >> manually. >> >> Gary. >> >> On Thu, Jun 3, 2010 at 10:35, Lev Stesin wrote: >>> Hi, >>> >>> I am getting OOM during load tests: >>> >>> java.lang.OutOfMemoryError: Java heap space >>> at java.util.HashSet.(HashSet.java:125) >>> at >>> com.google.common.collect.Sets.newHashSetWithExpectedSize(Sets.java:181) >>> at >>> com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:112) >>> at >>> com.google.common.collect.HashMultimap.createCollection(HashMultimap.java:47) >>> at >>> com.google.common.collect.AbstractMultimap.createCollection(AbstractMultimap.java:155) >>> at >>> com.google.common.collect.AbstractMultimap.getOrCreateCollection(AbstractMultimap.java:207) >>> at >>> com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:194) >>> at >>> com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:80) >>> at com.google.common.collect.HashMultimap.put(HashMultimap.java:47) >>> at >>> org.apache.cassandra.locator.AbstractReplicationStrategy.getHintedEndpoints(AbstractReplicationStrategy.java:87) >>> at >>> org.apache.cassandra.service.StorageProxy.mutateBlocking(StorageProxy.java:204) >>> at >>> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:451) >>> at >>> org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:361) >>> at >>> org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:1484) >>> at >>> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1125) >>> at >>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> >>> Any suggestions as to how to fix/ease the problem? Thanks. >>> >>> -- >>> Lev >>> >> > > > > -- > Lev >
Re: Cassandra Cluster Setup
Your replication factor is only set to 1, which means that each key will only live on a single node. If you do wait for bootstrapping to commence (takes 90s in trunk, I don't recall in 0.6), you should see some keys moving unless your inserts were all into a small range. Perhaps your being impatient. If not, I recommend that you start over, set the replication factor to 3, wait a good while for all the nodes to fully join the cluster, and then make sure the keys you write to are random. Gary. On Thu, Jun 3, 2010 at 13:07, Stephan Pfammatter wrote: > I’m having difficulties setting up a 3 way cassandra cluster. Any > comments/help would be appreciated. > > > > My goal is that all data should be fully replicated amongst the 3 nodes. I > want to simulate the failure of one node and proof that the test column > family still can be accessed. > > In a nutshell I did the following 3 steps. The excerpt below shows the > changed storage-con.xml: > > (1) I started Cassandra-ca and worked fine. I added 3 rows to a cf > cf_test. > > (2) Then I started Cassandra-or. It comes up ok but goes into “sleeping > 3”…..mode > > (3) Nodetool streams doesn’t reveal anything. > > > > > > > > Node: Cassandra-ca > > false > > > > > > > org.apache.cassandra.locator.RackUnawareStrategy > > 1 > > > org.apache.cassandra.locator.EndPointSnitch > > > > cassandra-ca > > cassandra-us > > 9160 > > cassandra-us > > > > Cassandra-or > > true > > > > > > > org.apache.cassandra.locator.RackUnawareStrategy > > 1 > > > org.apache.cassandra.locator.EndPointSnitch > > > > cassandra-or > > cassandra-or > > 9160 > > cassandra-or > > > > Cassandra-az > > > > Cassandra-or > > true > > > > > > > org.apache.cassandra.locator.RackUnawareStrategy > > 1 > > > org.apache.cassandra.locator.EndPointSnitch > > > > cassandra-az > > cassandra-az > > 9160 > > cassandra-az > > > > > >
Re: High CPU Usage since 0.6.2
Chris, Can you get me a stack dump of one of the busy nodes (kill -3)? Gary On Thu, Jun 3, 2010 at 22:50, Chris Goffinet wrote: > We're seeing this as well. We were testing with a 40+ node cluster on the > latest 0.6 branch from few days ago. > > -Chris > > On Jun 3, 2010, at 9:55 PM, Lu Ming wrote: > >> >> I have ten 0.5.1 Cassandra nodes in my cluster, and I update them to >> cassandra to 0.6.2 yesterday. >> But today I find six cassandra nodes have high CPU usage more than 400% in >> my 8-core CPU sever. >> The worst one is more than 760%. It is very serious. >> >> I use jvisualvm to watch the worst node, and I found that there are many >> running threads named "thread-xxx" >> the status of other threads is waiting and sleeping. >> >> "Thread-130" - Thread t...@240 >> java.lang.Thread.State: RUNNABLE >> at sun.misc.Unsafe.setMemory(Native Method) >> at sun.nio.ch.Util.erase(Util.java:202) >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> Locked ownable synchronizers: >> - None >> >> "Thread-126" - Thread t...@236 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) >> at sun.nio.ch.IOUtil.read(IOUtil.java:200) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> - locked java.lang.obj...@10808561 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> "Thread-119" - Thread t...@229 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.NativeThread.current(Native Method) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:182) >> - locked java.lang.obj...@65b4abbd >> - locked java.lang.obj...@38773975 >> at >> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:565) >> at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603) >> at >> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62) >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66) >> >> Locked ownable synchronizers: >> - None >> >> > >
Re: Cassandra Cluster Setup
Great. It looks as if the replication factor is still 1, not 3. This means that each key lives only on one node. By increasing it to 3, data will be replicated across all 3 nodes. Gary. On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter wrote: > Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca > and changed the replication factor. > Now I can see the sample keys being spread across the 3 nodes: > Cassandra-ca holds rows (keys: 1,5,8) > Cassandra-az holds rows (keys: 11,55) > Cassandra-or hodls rows (keys: 88) > > I don't see the data being duplicated yet. Am I missing a storage-conf > setting? I need to proof that if Cassandra-az fails that I can still provide > key 11,55. > > -Original Message- > From: Nahor [mailto:nahor.j+gm...@gmail.com] > Sent: Thursday, June 03, 2010 8:25 PM > To: user@cassandra.apache.org > Subject: Re: Cassandra Cluster Setup > > On 2010-06-03 13:07, Stephan Pfammatter wrote: >> Cassandra-or >> [...] >> cassandra-or > > > Aside from the replication factor noted by Gary, this should point to > your existing node (cassandra-ca) otherwise, how will this node know > where existing node is and where to get the data from? > > >> Cassandra-az >> [...] >> cassandra-az > > Same here > > >
Re: Expected wait while bootstrapping?
Most of the streaming messages are DEBUG, so you'll have to amp up logging. Gary. On Fri, Jun 4, 2010 at 12:26, Aaron Lav wrote: > I'm trying to bring up a new node. The log on the new node says: > > INFO 19:00:47,794 Joining: getting bootstrap token > INFO 19:00:49,003 New token will be 98546264352590127933376074587615077134 to > assume load from /xx.xxx.xx.xxx > > I had expected to see messages about the beginning of anticompaction > on the host load is being assumed from, but I don't see them in that > host's log. > > I've tried restarting the node with the load and then the new node, > but still nothing happens. > > I've been waiting for ~15 minutes: how long should I expect to wait, and > is there anything further I can look at? > > Thanks, > Aaron Lav (a...@pobox.com) >
Re: Cassandra Cluster Setup
It should take roughly 30 seconds (please clarify--not minutes) to join the cluster. Gary. On Mon, Jun 7, 2010 at 08:23, Stephan Pfammatter wrote: > I still have one open issue: My cluster has only 3 local nodes with 1 CF (5 > rows). > The initial start works fine. I can see the replicated data and can do all > admin tasks I want, successfully. > > Now I'm simulating a node failure by stopping Cassandra. I remove > (a) commit log > (b) content in system keyspace > (c) application cf > > When I bring the node back up I can see the new token assignment, streaming > notification (we are talking about 5 rows in 1 cf). Thrift port is not > available for ~30 min. Why is that? I would expect it to be back in a matter > of maybe <1' (this should be data quantity related). > > Thanks, Stephan > > > -Original Message- > From: Gary Dusbabek [mailto:gdusba...@gmail.com] > Sent: Friday, June 04, 2010 11:49 AM > To: user@cassandra.apache.org > Subject: Re: Cassandra Cluster Setup > > Great. It looks as if the replication factor is still 1, not 3. This > means that each key lives only on one node. By increasing it to 3, > data will be replicated across all 3 nodes. > > Gary. > > > On Fri, Jun 4, 2010 at 06:51, Stephan Pfammatter > wrote: >> Tx Nahor/Gary/Ben. I blew everything away, adjusted the seed to Cassandra-ca >> and changed the replication factor. >> Now I can see the sample keys being spread across the 3 nodes: >> Cassandra-ca holds rows (keys: 1,5,8) >> Cassandra-az holds rows (keys: 11,55) >> Cassandra-or hodls rows (keys: 88) >> >> I don't see the data being duplicated yet. Am I missing a storage-conf >> setting? I need to proof that if Cassandra-az fails that I can still provide >> key 11,55. >> >> -Original Message- >> From: Nahor [mailto:nahor.j+gm...@gmail.com] >> Sent: Thursday, June 03, 2010 8:25 PM >> To: user@cassandra.apache.org >> Subject: Re: Cassandra Cluster Setup >> >> On 2010-06-03 13:07, Stephan Pfammatter wrote: >>> Cassandra-or >>> [...] >>> cassandra-or >> >> >> Aside from the replication factor noted by Gary, this should point to >> your existing node (cassandra-ca) otherwise, how will this node know >> where existing node is and where to get the data from? >> >> >>> Cassandra-az >>> [...] >>> cassandra-az >> >> Same here >> >> >> >
Re: Running Cassandra as a Windows Service
IMO this is one of those things that would bitrot fairly quickly if it were not maintained. It may be useful in contrib, where curious parties could pick it up, get it back in shape, and send in the changes to be committed. Judging by the sparse interest so far, this probably wouldn't be a good fit in core since there don't seem to be many (any?) cassandra developers who run windows. Gary. On Thu, Jun 10, 2010 at 12:34, Kochheiser,Todd W - TO-DITT1 wrote: > For various reasons I am required to deploy systems on Windows. As such, I > went looking for information on running Cassandra as a Windows service. > I’ve read some of the user threads regarding running Cassandra as a Windows > service, such as this one: > > http://www.mail-archive.com/user@cassandra.apache.org/msg01656.html > > I also found the following JIRA issue: > > https://issues.apache.org/jira/browse/CASSANDRA-292 > > As it didn’t look like anyone has contributed a formal solution and having > some experience using Apache’s Procrun > (http://commons.apache.org/daemon/procrun.html), I decided to go ahead and > write a batch script and a simple “WindowsService” class to accomplish the > task. The WindowsService class only makes calls to public methods in > CassandraDeamon and is fairly simple. In combination with the batch script, > it is very easy to install and remove the service. At this point, I’ve > installed Cassandra as a Windows service on XP (32 bit), Windows 7 (64 bit) > and Windows Server 2008 R1/R2 (64 bit). It should work fine on other > version of Windows (2K, 2K3). > > Questions: > > > Has anyone else already done this work? > If not, I wouldn’t mind sharing the code/script or contributing it back to > the project. Is there any interest in this from the Cassandra dev team or > the user community? > > > Ideally the WindowsService could be included in the distributed > source/binary distributions (perhaps in a contrib area) as well as the batch > script and associated procrun executables. Or, perhaps it could be posted > to a Cassandra community site (is there one?). > > Todd > > > > >
Re: Running Cassandra as a Windows Service
Sure. Please create a jira ticket (https://issues.apache.org/jira/browse/CASSANDRA) and attach the files you wish to contribute. One of the committers (probably myself) will review them and decide how to integrate them into the project. If it's not too much trouble, an ant build script would be excellent and would help to automate the process of generating testable builds. Gary. On Thu, Jun 10, 2010 at 17:31, Kochheiser,Todd W - TO-DITT1 wrote: > I agree that bitrot might be happen if all of the core Cassandra developers > are using Linux. Your suggestion of putting things in a contrib area where > curious (or desperate) parties suffering on the Windows platform could pick > it up seems like a reasonable place to start. It might also be an > opportunity to increase the number of "application" developers using > Cassandra if Cassandra was slightly more approachable on the Windows platform. > > Any suggestions on next steps? > > Todd. > > -Original Message- > From: Gary Dusbabek [mailto:gdusba...@gmail.com] > Sent: Thursday, June 10, 2010 10:59 AM > To: user@cassandra.apache.org > Subject: Re: Running Cassandra as a Windows Service > > IMO this is one of those things that would bitrot fairly quickly if it > were not maintained. It may be useful in contrib, where curious > parties could pick it up, get it back in shape, and send in the > changes to be committed. > > Judging by the sparse interest so far, this probably wouldn't be a > good fit in core since there don't seem to be many (any?) cassandra > developers who run windows. > > Gary. > > > On Thu, Jun 10, 2010 at 12:34, Kochheiser,Todd W - TO-DITT1 > wrote: >> For various reasons I am required to deploy systems on Windows. As such, I >> went looking for information on running Cassandra as a Windows service. >> I've read some of the user threads regarding running Cassandra as a Windows >> service, such as this one: >> >> http://www.mail-archive.com/user@cassandra.apache.org/msg01656.html >> >> I also found the following JIRA issue: >> >> https://issues.apache.org/jira/browse/CASSANDRA-292 >> >> As it didn't look like anyone has contributed a formal solution and having >> some experience using Apache's Procrun >> (http://commons.apache.org/daemon/procrun.html), I decided to go ahead and >> write a batch script and a simple "WindowsService" class to accomplish the >> task. The WindowsService class only makes calls to public methods in >> CassandraDeamon and is fairly simple. In combination with the batch script, >> it is very easy to install and remove the service. At this point, I've >> installed Cassandra as a Windows service on XP (32 bit), Windows 7 (64 bit) >> and Windows Server 2008 R1/R2 (64 bit). It should work fine on other >> version of Windows (2K, 2K3). >> >> Questions: >> >> >> Has anyone else already done this work? >> If not, I wouldn't mind sharing the code/script or contributing it back to >> the project. Is there any interest in this from the Cassandra dev team or >> the user community? >> >> >> Ideally the WindowsService could be included in the distributed >> source/binary distributions (perhaps in a contrib area) as well as the batch >> script and associated procrun executables. Or, perhaps it could be posted >> to a Cassandra community site (is there one?). >> >> Todd >> >> >> >> >> >
Re: cassandra out of heap space crash
On Fri, Jun 11, 2010 at 10:14, Julie wrote: > Ran Tavory gmail.com> writes: > >> >> I can't say exactly how much memory is the correct amount, but surely 1G is > very little. By replicating 3 times your cluster now makes 3 times more work > than it used to do, both on reads and on writes while the readers/writers > continue hammering it the same pace. >> >> >> So once you've upped your memory (try 4g, if not enough 8g etc) if this still > doesn't help, you want to look at either adding capacity or slowing down your > writes. >> Which consistency level are you writing with? You can try ALL, this will slow > down your writes just as much needed by the cluster to catch its breath (or > so I > hope, I never actually tried that...) > > > > Thank you so much, Ran! I am running on ec2-smalls and they have only 1.7GB > of memory in total so 1GB of heap space is probably the highest I can go until > we move to larger VMs. But you bring up an EXCELLENT point -- changing the > consistency level. Because we are trying to populate the database as quickly > as possible for our prototyping work, we use consistency of ZERO(!) I am going > to try changing it to ALL as you suggest and also possibly throttle the > writes. Writing at CL.ZERO is an excellent way to run out of memory since the operations have to be buffered until there are cycles to execute them. ONE would give you decent throughput without so much risk of OOM. > > Does this seem like a Cassandra bug or is it well known that Cassandra always > needs more than 1GB of heap space? Not a bug at all. You were putting a lot of memory pressure on Cassandra by writing at CL.ZERO. Gary.
Re: batch_mutate atomic?
This question has been coming up quite regularly now. I've added an entry to the FAQ. Please feel free to expand an clarify. http://wiki.apache.org/cassandra/FAQ#batch_mutate_atomic Gary. On Mon, Jun 14, 2010 at 06:43, Ran Tavory wrote: > no, it's not atomic. it just shortens the roundtrip of many update requests. > Some may fail and some may succeed > > On Mon, Jun 14, 2010 at 2:40 PM, Per Olesen wrote: >> >> Can I expect batch_mutate to work in what I would think of as an atomic >> operation? >> >> That either all the mutations in the batch_mutate call are executed or >> none of them are? Or can some of them fail while some of them succeeds? >> > >
Re: CF that is like a non-clustered index, are key lookups that fast?
On Tue, Jun 15, 2010 at 04:29, S Ahmed wrote:
> If you store only the key mappings in a column family, for custom ordering
> of rows etc. for things like:
> friends = {
>
> user_id : { friendid1, friendid2, }
> }
> or
> topForumPosts = {
>
> forum_id1 : { post2343, post32343, post32223, ...}
> }
>
> Now on friends page or on the top_forum_posts page you will get back a list
> of post_ids, you will then have to perform lookups on the main 'posts' CF to
> get the actual data. So if a page is displaying 10, 25, or 50 posts you
> will have 10, 25 or 50 key based lookups for each page view.
> Is this the suggested way? i.e. a look based on a slice to get a list of
> post_id's, then a seperate call to actually fetch the data for the given
> entity.
> Or is cassandra so fast that 50 key based calls is no reason to worry?
You should look at using either multi_get_slice or get_range_slices.
You'll save on network trips and the amount of work required of the
cluster.
Gary.
Re: Some questions about using Cassandra
On Tue, Jun 15, 2010 at 19:49, Anthony Ikeda wrote: > Is there any concept of Listeners such that when data is added to Cassandra > we can fire off another process to do something with that data? E.g. create > a copy in a secondary database for Business Intelligence reports? Send the > data to an LDAP server? > Typically, this is the kind of thing that gets handled at the application level (your application). There is a feature request for it though. https://issues.apache.org/jira/browse/CASSANDRA-1016 Gary.
Re: java.lang.RuntimeException: java.io.IOException: Value too large for defined data type
*Hopefully* fixed. I was never able to duplicate the problem on my workstation, but I had a pretty good idea what was causing the problem. Julie, if you're in a position to apply and test the fix, it would help help us make sure we've got this one nailed down. Gary. On Thu, Jun 17, 2010 at 00:33, Jonathan Ellis wrote: > That is consistent with the > https://issues.apache.org/jira/browse/CASSANDRA-1169 bug I mentioned, > that is fixed in the 0.6 svn branch. > > On Wed, Jun 16, 2010 at 10:51 PM, Julie wrote: >> The loop is in IncomingStreamReader.java, line 62, a 3-line while loop. >> bytesRead is not changing. pendingFile.getExpectedBytes() returns >> 7,161,538,639 but bytesRead is stuck at 2,147,483,647. >>
Re: Finding new Cassandra data
On Tue, Jun 22, 2010 at 09:59, David Boxenhorn wrote: > In my system, I have a Cassandra front end, and an Oracle back end. Some > information is created in the back end, and pushed out to the front end, and > some information is created in the front end and pulled into the back end. > > Question: How do I locate new rows that have been crated in Cassandra, for > import into Oracle? > > I'm thinking of having a special column family "newRows" that contains only > the keys of the new rows. The offline process would look there to see what's > new, then delete those rows. The "newRows" CF would have no data! (The data > would be in the "real" CF.) I've never tried an empty row, but I'm pretty sure you need at least one column. > > Is this a good solution? It seems weird to have a CF with rows but no data. > But I can't think of a better way. > > Any thoughts? Another approach would be to have a CF with a single row whose column names refer to the new row ids. This would allow you efficient slicing. The downside is that you'd need to make sure the row doesn't get too wide. So depending on your throughput and application behavior, this may or may not work. Gary.
Re: nodetool loadbalance : Strerams Continue on Non Acceptance of New Token
On Tue, Jun 22, 2010 at 20:16, Arya Goudarzi wrote: > Hi, > > Please confirm if this is an issue and should be reported or I am doing > something wrong. I could not find anything relevant on JIRA: > > Playing with 0.7 nightly (today's build), I setup a 3 node cluster this way: > > - Added one node; > - Loaded default schema with RF 1 from YAML using JMX; > - Loaded 2M keys using py_stress; > - Bootstrapped a second node; > - Cleaned up the first node; > - Bootstrapped a third node; > - Cleaned up the second node; > > I got the following ring: > > Address Status Load Range > Ring > 154293670372423273273390365393543806425 > 10.50.26.132 Up 518.63 MB 69164917636305877859094619660693892452 > |<--| > 10.50.26.134 Up 234.8 MB > 111685517405103688771527967027648896391 | | > 10.50.26.133 Up 235.26 MB > 154293670372423273273390365393543806425 |-->| > > Now I ran: > > nodetool --host 10.50.26.132 loadbalance > > It's been going for a while. I checked the streams > > nodetool --host 10.50.26.134 streams > Mode: Normal > Not sending any streams. > Streaming from: /10.50.26.132 > Keyspace1: > /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-3-Data.db/[(0,22206096), > (22206096,27271682)] > Keyspace1: > /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-4-Data.db/[(0,15180462), > (15180462,18656982)] > Keyspace1: > /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-5-Data.db/[(0,353139829), > (353139829,433883659)] > Keyspace1: > /var/lib/cassandra/data/Keyspace1/Standard1-tmp-d-6-Data.db/[(0,366336059), > (366336059,450095320)] > > nodetool --host 10.50.26.132 streams > Mode: Leaving: streaming data to other nodes > Streaming to: /10.50.26.134 > /var/lib/cassandra/data/Keyspace1/Standard1-d-48-Data.db/[(0,366336059), > (366336059,450095320)] > Not receiving any streams. > > These have been going for the past 2 hours. > > I see in the logs of the node with 134 IP address and I saw this: > > INFO [GOSSIP_STAGE:1] 2010-06-22 16:30:54,679 StorageService.java (line 603) > Will not change my token ownership to /10.50.26.132 A node will give this message when it sees another node (usually for the first time) that is trying to claim the same token but whose startup time is much earlier (i.e., this isn't a token replacement). It would follow that you would see this during a rebalance. > > So, to my understanding from wikis loadbalance supposed to decommission and > re-bootstrap again by sending its tokens to other nodes and then bootstrap > again. It's been stuck in streaming for the past 2 hours and the size of ring > has not changed. The log in the first node says it has started streaming for > the past hours: > > INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,255 StreamOut.java (line 72) > Beginning transfer process to /10.50.26.134 for ranges > (154293670372423273273390365393543806425,69164917636305877859094619660693892452] > INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,255 StreamOut.java (line 82) > Flushing memtables for Keyspace1... > INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,266 StreamOut.java (line 128) > Stream context metadata > [/var/lib/cassandra/data/Keyspace1/Standard1-d-48-Data.db/[(0,366336059), > (366336059,450095320)]] 1 sstables. > INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,267 StreamOut.java (line 135) > Sending a stream initiate message to /10.50.26.134 ... > INFO [STREAM-STAGE:1] 2010-06-22 16:35:56,267 StreamOut.java (line 140) > Waiting for transfer to /10.50.26.134 to complete > INFO [FLUSH-TIMER] 2010-06-22 17:36:53,370 ColumnFamilyStore.java (line 359) > LocationInfo has reached its threshold; switching in a fresh Memtable at > CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1277249454413.log', > position=720) > INFO [FLUSH-TIMER] 2010-06-22 17:36:53,370 ColumnFamilyStore.java (line 622) > Enqueuing flush of Memtable(LocationInfo)@1637794189 > INFO [FLUSH-WRITER-POOL:1] 2010-06-22 17:36:53,370 Memtable.java (line 149) > Writing Memtable(LocationInfo)@1637794189 > INFO [FLUSH-WRITER-POOL:1] 2010-06-22 17:36:53,528 Memtable.java (line 163) > Completed flushing /var/lib/cassandra/data/system/LocationInfo-d-9-Data.db > INFO [MEMTABLE-POST-FLUSHER:1] 2010-06-22 17:36:53,529 > ColumnFamilyStore.java (line 374) Discarding 1000 > > > Nothing more after this line. > > Am I doing something wrong? If the output you get from `nodetool streams` isn't changing, then I'd say we have a bug. You're data sizes weren't that large--I'd expect 2 hrs would be more than enough time. I've created https://issues.apache.org/jira/browse/CASSANDRA-1221 to track this problem. Gary. > > > Best Regards, > -Arya > >
Re: KeyRange deletion
See CASSANDRA-293 and CASSANDRA-494. Key-range updates touch multiple rows and cannot be performed atomically, so the consistency promise you get would be the same as with batch_mutate. Another other problem is that the update needs to be sent to *any* node that is responsible for part of the range. That isn't so bad though. The real problem is dealing with failure. The current implementation of hinted-handoff is key based. I've looked at extending it for ranges and there isn't a good way of doing it without rewriting all of hinted-handoff. If you're mainly interested in avoid round-trips, have you tried using batch_mutate? I know there was a limitation when specifying a Deletion, but I think that has been resolved. On Wed, Jun 23, 2010 at 17:39, Carlos Sanchez wrote: > Would there be any support for a KeyRange (start/end key) deletion in 0.7? > > Carlos > > This email message and any attachments are for the sole use of the intended > recipients and may contain proprietary and/or confidential information which > may be privileged or otherwise protected from disclosure. Any unauthorized > review, use, disclosure or distribution is prohibited. If you are not an > intended recipient, please contact the sender by reply email and destroy the > original message and any copies of the message as well as any attachments to > the original message. >
Re: Bootstrap Token collision
Did you add a new node to the cluster at the time you restarted it? If not, I would think that each node already had a token that would make such a collision impossible, unless we have a new bug to troubleshoot. Gary. On Wed, Jul 14, 2010 at 20:46, Mubarak Seyed wrote: > The cluster nodes were running fine. When i restarted to modify the JVM heap > settings, two of the nodes are not joining the cluster and throws Bootstrap > Token collision > Any idea how to fix this error? > ERROR [GMFD:1] 2010-07-15 01:23:13,756 DebuggableThreadPoolExecutor.java > (line 101) Error in ThreadPoolExecutor > java.lang.RuntimeException: Bootstrap Token collision between > /xx.xxx.xxx. and /xx.yyy.zzz.aaa (token > 141231821124659524366835775276207938883 > at > org.apache.cassandra.locator.TokenMetadata.addBootstrapToken(TokenMetadata.java:130) > at > org.apache.cassandra.service.StorageService.handleStateBootstrap(StorageService.java:529) > at > org.apache.cassandra.service.StorageService.onChange(StorageService.java:492) > at > org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:730) > at > org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:695) > at > org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:649) > at > org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:60) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > -- > Thanks, > Mubarak Seyed. >
Re: Increasing Replication Factor in 0.7
Arya, That is not currently possible in trunk. It would be a good feature though. Care to file a ticket? Gary. On Thu, Jul 15, 2010 at 22:13, Arya Goudarzi wrote: > I recall jbellis in his training showing us how to increase the replication > factor and repair data on a cluster in 0.6. How is that possible in 0.7 when > you cannot change schemas from config and there is no alter keyspace method > in api? >
Re: Define keyspaces in cassandra 0.7
Defining at runtime is, very intentionally, an absolute must. It would have been very simple and perhaps user-friendly to add a flag that loads the schema specified in yaml when cassandra starts up. I decided against it when implementing the feature because I figured it would have been a disservice if users were constantly over-writing existing schema unintentionally. Gary. On Mon, Jul 19, 2010 at 08:39, Boris Spasojevic wrote: > Hi all, > Can someone post an example of how to define keyspaces in Cassandra 0.7? > My initial Cassandra node does not load the keyspaces defined at > Cassandra.yaml. Is there a way to define the keyspaces at startup or is > runtime defining an absolute must? > > thanks, > BoriS > >
Re: Define keyspaces in cassandra 0.7
I just realized that I didn't answer the "how" part of your question. :) http://svn.apache.org/repos/asf/cassandra/trunk/contrib/py_stress/stress.py and http://svn.apache.org/repos/asf/cassandra/trunk/test/system/__init__.py both contain examples of how to use the system_* methods to manipulate keyspaces. Gary. On Mon, Jul 19, 2010 at 09:46, Gary Dusbabek wrote: > Defining at runtime is, very intentionally, an absolute must. It > would have been very simple and perhaps user-friendly to add a flag > that loads the schema specified in yaml when cassandra starts up. I > decided against it when implementing the feature because I figured it > would have been a disservice if users were constantly over-writing > existing schema unintentionally. > > Gary. > > > On Mon, Jul 19, 2010 at 08:39, Boris Spasojevic > wrote: >> Hi all, >> Can someone post an example of how to define keyspaces in Cassandra 0.7? >> My initial Cassandra node does not load the keyspaces defined at >> Cassandra.yaml. Is there a way to define the keyspaces at startup or is >> runtime defining an absolute must? >> >> thanks, >> BoriS >> >> >
Re: Bootstrap question
Anthony, I think you're seeing the results of CASSANDRA-1221. Each node has two connections with its peers. One connection is used for gossip, the other for exchanging commands. What you see with 1221 is the command socket getting 'stuck' after a peer is convicted by gossip and then recovers. It doesn't happen every time, but it happens much of the time, especially with streaming. I was able to reproduce this at will using loadbalance, but never tried it under bootstrap (where the bootstrapping IP was previously visible on the cluster), but it seems very plausible. Any chance you could apply the patch for 1221 and test? Gary. On Tue, Jul 20, 2010 at 16:45, Anthony Molinaro wrote: > I see this in the old nodes > > DEBUG [WRITE-/10.220.198.15] 2010-07-20 21:15:50,366 > OutboundTcpConnection.java (line 142) attempting to connect to /10.220.198.15 > INFO [GMFD:1] 2010-07-20 21:15:50,391 Gossiper.java (line 586) Node > /10.220.198.15 is now part of the cluster > INFO [GMFD:1] 2010-07-20 21:15:51,369 Gossiper.java (line 578) InetAddress > /10.220.198.15 is now UP > INFO [HINTED-HANDOFF-POOL:1] 2010-07-20 21:15:51,369 > HintedHandOffManager.java (line 153) Started hinted handoff for endPoint > /10.220.198.15 > INFO [HINTED-HANDOFF-POOL:1] 2010-07-20 21:15:51,371 > HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to > endpoint /10.220.198.15 > DEBUG [GMFD:1] 2010-07-20 21:17:20,551 StorageService.java (line 512) Node > /10.220.198.15 state bootstrapping, token > 28356863910078205288614550619314017621 > DEBUG [GMFD:1] 2010-07-20 21:17:20,656 > StorageService.java (line 746) Pending ranges: > /10.220.198.15:(21604748163853165203168832909938143241,28356863910078205288614550619314017621] > /10.220.198.15:(10637639655367601517656788464652024082,21604748163853165203168832909938143241] > > 10.220.198.15 is the new node > > The key ranges seem to be for the primary and replica ranges. > > So after that, I would expect some AntiCompaction to happen on some of the > other nodes, but I don't see anything. > > Any clues from that output? > > I did not muck around with the Location tables. > > -Anthony > > On Mon, Jul 19, 2010 at 09:36:22PM -0500, Jonathan Ellis wrote: >> What gets logged on the old nodes at debug, when you try to add a >> single new machine after a full cluster restart? >> >> Removing Location would blow away the nodes' token information... It >> should be safe if you set the InitialToken to what it used to be on >> each machine before bringing it up after nuking those. Better >> snapshot the system keyspace first, just in case. >> >> On Sun, Jul 18, 2010 at 2:01 PM, Anthony Molinaro >> wrote: >> > Yeah, I tried all that already and it didn't seem to work, no new nodes >> > will bootstrap, which makes me think there's some saved state somewhere, >> > preventing a new node from bootstrapping. I think maybe the Location >> > sstables? Is it safe to nuke those on all hosts and restart everything? >> > (I just don't want to lose actual data). >> > >> > Thanks for the ideas, >> > >> > -Anthony >> > >> > On Sun, Jul 18, 2010 at 08:09:45PM +0300, shimi wrote: >> >> If I have problems with never ending bootstraping I do the following. I >> >> try >> >> each one if it doesn't help I try the next. It might not be the right >> >> thing >> >> to do but it worked for me. >> >> >> >> 1. Restart the bootstraping node >> >> 2. If I see streaming 0/ I restart the node and all the streaming >> >> nodes >> >> 3. Restart all the nodes >> >> 4. If there is data in the bootstraing node I delete it before I restart. >> >> >> >> Good luck >> >> Shimi >> >> >> >> On Sun, Jul 18, 2010 at 12:21 AM, Anthony Molinaro < >> >> antho...@alumni.caltech.edu> wrote: >> >> >> >> > So still waiting for any sort of answer on this one. The cluster still >> >> > refuses to do anything when I bring up new nodes. I shut down all the >> >> > new nodes and am waiting. I'm guessing that maybe the old nodes have >> >> > some state which needs to get cleared out? Is there anything I can do >> >> > at this point? Are there alternate strategies for bootstrapping I can >> >> > try? (For instance can I just scp all the sstables to all the new >> >> > nodes and do a repair, would that actually work?). >> >> > >> >> > Anyone seen this
Re: Bootstrap question
On Wed, Jul 21, 2010 at 14:14, Anthony Molinaro wrote: > Sure, looks like that's in 0.6.4, so I'll probably just rebuild my server > based on the 0.6 branch, unless you want me to test just the patch for > 1221? Most likely won't get a chance to try until tomorrow, so let me > know. > Either way works for me.
Re: 0.6.4 tag
The vote is in process. http://permalink.gmane.org/gmane.comp.db.cassandra.devel/2010 Gary. On Thu, Jul 29, 2010 at 11:34, B. Todd Burruss wrote: > i see a 0.6.4 tag in SVN, but not on cassandra's download page. is this > ready for use if building from SVN? > >
Re: Changing schema w/o losing data - Can it be done?
NullPointerException--it sounds like you're restarting? If so, you should look into `nodetool drain`. http://wiki.apache.org/cassandra/FAQ#modify_cf_config Gary On Wed, Aug 4, 2010 at 03:27, David Boxenhorn wrote: > When I change schema I need to delete the commit logs - otherwise I get a > null pointer exception. > > Is there a way that I can force Cassandra to empty the commit logs so I > won't lose data??? > > (There has not been much recent activity, nor is there much current > activity. There was only one log and it was only 54K.) >
Re: set ReplicationFactor and Token at Column Family/SuperColumn level.
On Thu, Aug 5, 2010 at 14:59, Zhong Li wrote: > All, > > Thanks for Apache Cassandra Project, it is great project. > > This is my first time to use it. We install it on 10 nodes and runs great. > The 10 nodes cross all 5 datacenters around the world. > > The big thing bother me is initial ring token. We have some Column Families. > It is very hard to choose one token suitable for all CFs. Also some Column > Families need higher Consistent Level and some don't. If we set > ReplicationFactor too high, it is too costy for crossing datacenter, > especially in otherside the world. > > I know we can setup multiple rings, but it costs more hardware. > > if Cassandra can implement Ring,Token and RF on the CF level, or even > SuperColumn level, it will make design much easier and more efficiency. You can get separate replication factors by putting your column families in separate keyspaces. Token per CF (or even token per KS) isn't on the roadmap though. You can mitigate this to some extent by choosing row keys that play nicely with your ring though. Gary > > Is it possible? > > Thanks, > > Zhong
Re: TokenRange contains endpoints without any port information?
On Sun, Aug 8, 2010 at 07:21, Carsten Krebs wrote: > > I'm wondering why a TokenRange returned by describe_ring(keyspace) of the > thrift API just returns endpoints consisting only of an address but omits any > port information? > My first thought was, this method could be used to expose some information > about the ring structure to the client, i.e. to do some client side load > balancing. But now, I'm not sure about this anymore. Additionally, when > looking into the code, I guess the address returned as part of the TokenRange > is the address of the storage service which could differ from the thrift > address, which in turn would make the returned endpoint useless for the > client. > What is the purpose of this method To give a picture of the ring topology. >or respectively why is the port information omitted? You already knew the thrift port to make the query connection. The only other port you *might* need to be concerned with is the storage port, which is assumed to be constant across the cluster. But really, from a client perspective it does you no good to know this port, so why bother exposing it? Gary. > > TIA, > > Carsten > >
Re: Reload/Update schema 0.7
You should take a look at http://wiki.apache.org/cassandra/LiveSchemaUpdates loadSchemaFromYaml() is intended to initialize the schema on a seed node in a new cluster (or one that has been upgraded from 0.6). It is an operation that should only be performed one time *per cluster.* Gary On Wed, Aug 11, 2010 at 20:56, Mark wrote: > How is this accomplished? > > I tried using the > org.apache.cassandra.service.StorageService.loadSchemaFromYAML() method but > I am receiving the following error. > > java.util.concurrent.ExecutionException: > org.apache.cassandra.config.ConfigurationException: Cannot load from XML on > top of pre-existing schemas. > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:637) > > Thanks again >
Re: Count rows
Should we close https://issues.apache.org/jira/browse/CASSANDRA-653 then? Fetching a count of all rows is just a specific instance of fetching the count of a range or rows. I spoke to a programmer at the summit who was working on this ticket mainly as a way of getting familiar with the codebase. He didn't want to submit a patch though, if there was zero chance of it ever being committed. Gary On Fri, Aug 13, 2010 at 12:52, Jonathan Ellis wrote: > because it would work amazingly poorly w/ billions of rows. it's an > antipattern. > > On Fri, Aug 13, 2010 at 10:50 AM, Mark wrote: >> On 8/13/10 10:44 AM, Jonathan Ellis wrote: >>> >>> not without fetching all of them with get_range_slices >>> >>> On Fri, Aug 13, 2010 at 10:37 AM, Mark wrote: >>> >>>> >>>> Is there some way I can count the number of rows in a CF.. CLI, MBean? >>>> >>>> Gracias >>>> >>>> >>> >>> >>> >> >> Im guessing you would advise against this? Any reason this wouldn't be >> exposed via a MBean? >> > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com >
Re: Index package
Please see ColumnFamilyStore.scan(). On Sat, Aug 14, 2010 at 15:01, Carlos Sanchez wrote: > All, > > Could I know what package or classes are used in Cassandra for secondary > column indices? > > Thanks > > Carlos > > This email message and any attachments are for the sole use of the intended > recipients and may contain proprietary and/or confidential information which > may be privileged or otherwise protected from disclosure. Any unauthorized > review, use, disclosure or distribution is prohibited. If you are not an > intended recipient, please contact the sender by reply email and destroy the > original message and any copies of the message as well as any attachments to > the original message. >
Re: MIGRATION-STAGE: IllegalArgumentException: value already present
Yes. definitely open a ticket. Gary On Mon, Aug 16, 2010 at 20:05, Arya Goudarzi wrote: > While inserting into a 3 node cluster, one of the nodes got this exception in > its log: > > ERROR [MIGRATION-STAGE:1] 2010-08-16 17:46:24,090 CassandraDaemon.java (line > 82) Uncaught exception in thread Thread[MIGRATION-STAGE:1,5,main] > java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: > value already present: 1017 > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > Caused by: java.lang.IllegalArgumentException: value already present: 1017 > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) > at > com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:109) > at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:94) > at com.google.common.collect.HashBiMap.put(HashBiMap.java:83) > at org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:170) > at > org.apache.cassandra.db.migration.AddColumnFamily.applyModels(AddColumnFamily.java:78) > at > org.apache.cassandra.db.migration.Migration.apply(Migration.java:157) > at > org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:729) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > ... 2 more > > What does it mean? Is it something I should open a JIRA for? > > -Arya >
Re: Errors with Cassandra 0.7
On Thu, Aug 19, 2010 at 16:30, Alaa Zubaidi wrote: > Hi, > > I am trying to run Cassandra 0.7 and I am getting different errors: First it > was while calling client.insert and now while calling set_keyspace (see > below). > > Note: I get the following when I start Cassandra: > *10/08/19 12:58:26 INFO utils.Mx4jTool: Will not load MX4J, mx4j-tools.jar > is not in the classpath* > MX4J exposes JMX operations over a simple web interface. Cassandra attempts to load it if the jar is in the classpath. By default is is *not* present. Hence, the message. Nothing to worry about. Gary.
Re: Replication factor and other schema changes in >= 0.7
It is coming. In fact, I started working on this ticket yesterday. Most of the settings that you could change before will be modifiable. Unfortunately, you must still manually perform the repair operations, etc., afterward. https://issues.apache.org/jira/browse/CASSANDRA-1285 Gary. On Thu, Aug 19, 2010 at 18:00, Andres March wrote: > How should we go about changing the replication factor and other keyspace > settings now that it and other KSMetaData are no longer managed in > cassandra.yaml? > > I found makeDefinitionMutation() in the Migration class and see that it is > called for the other schema migrations. There just seems to be a big gap in > the management API for the KSMetaData we might want to change. > -- > Andres March > ama...@qualcomm.com > Qualcomm Internet Services
Re: 0.7 beta 1 - "error in row mutation" and NPE
It looks like you're running into https://issues.apache.org/jira/browse/CASSANDRA-1403, which was fixed last week and will be included in beta2. If you are experiencing this on trunk, please do file another ticket, or comment on the existing one. Gary. On Mon, Aug 23, 2010 at 13:33, B. Todd Burruss wrote: > i see the following in my server logs quite closely while doing a lot of > batch_mutations and reads. i create keyspaces and column families using > thrift api, not cassandra.yaml. did not migrate anything from 0.6. > > 4 node cluster, RF = 3, QUORUM read/write. > > happens immediately on a fresh start of the cluster with no data - removed > all data and commit logs before starting. > > i believe the problem is because the creation of the keyspace/CFs has > possible not propagated to all nodes before i start slamming with the > batch_mutates and gets. if i put a 1 second sleep after the keyspace has > been created, but before i start using it, all is well. > > > ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659 > RowMutationVerbHandler.java (line 78) Error in row mutation > org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find > cfId=1001 > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115) > at > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197 > DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor > java.lang.NullPointerException > at org.apache.cassandra.db.Table.open(Table.java:104) > at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > >
Re: 0.7 beta 1 - "error in row mutation" and NPE
On second thought, that doesn't look like the problem at all. Have you waited long enough for the schema to propagate? What does check_schema_agreement tell you? One thing to make sure about: you should funnel all your schema updates through a single coordinator node or make sure that the cluster is quiescent before sending schema updates to a random node. Gary. On Mon, Aug 23, 2010 at 14:03, Gary Dusbabek wrote: > It looks like you're running into > https://issues.apache.org/jira/browse/CASSANDRA-1403, which was fixed > last week and will be included in beta2. > > If you are experiencing this on trunk, please do file another ticket, > or comment on the existing one. > > Gary. > > > On Mon, Aug 23, 2010 at 13:33, B. Todd Burruss wrote: >> i see the following in my server logs quite closely while doing a lot of >> batch_mutations and reads. i create keyspaces and column families using >> thrift api, not cassandra.yaml. did not migrate anything from 0.6. >> >> 4 node cluster, RF = 3, QUORUM read/write. >> >> happens immediately on a fresh start of the cluster with no data - removed >> all data and commit logs before starting. >> >> i believe the problem is because the creation of the keyspace/CFs has >> possible not propagated to all nodes before i start slamming with the >> batch_mutates and gets. if i put a 1 second sleep after the keyspace has >> been created, but before i start using it, all is well. >> >> >> ERROR [ROW-MUTATION-STAGE:19] 2010-08-23 11:14:30,659 >> RowMutationVerbHandler.java (line 78) Error in row mutation >> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find >> cfId=1001 >> at >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:115) >> at >> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372) >> at >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382) >> at >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340) >> at >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> ERROR [ROW-READ-STAGE:6] 2010-08-23 11:26:04,197 >> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor >> java.lang.NullPointerException >> at org.apache.cassandra.db.Table.open(Table.java:104) >> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70) >> at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:41) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> >
Re: StackOverflowError on startup!
I'm willing to bet this has something to do with the system table initialization trying to delete its stats, which it can't do until it's initialized, which means it tries to delete its stats, which it can't do until it's initialized, which... you get the picture. The stack trace is very similar to https://issues.apache.org/jira/browse/CASSANDRA-1382, which is fixed and will be included in the next beta. Gary. On Thu, Aug 26, 2010 at 13:15, Mark wrote: > Using 0.7beta1 and I noticed one of my nodes was not responding.. wtf? Went > to restart and I got the following error. Any clues? > > Exception encountered during startup. > java.lang.StackOverflowError > at java.util.Vector.ensureCapacityHelper(Vector.java:238) > at java.util.Vector.addElement(Vector.java:593) > at java.lang.ClassLoader.addClass(ClassLoader.java:184) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:632) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:277) > at java.net.URLClassLoader.access$000(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:212) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:205) > at java.lang.ClassLoader.loadClass(ClassLoader.java:319) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) > at java.lang.ClassLoader.loadClass(ClassLoader.java:264) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:332) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:208) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:342) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:308) > at org.apache.cassandra.db.Table.(Table.java:245) > at org.apache.cassandra.db.Table.open(Table.java:102) > at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:196) > at > org.apache.cassandra.db.StatisticsTable.deleteSSTableStatistics(StatisticsTable.java:81) > at > org.apache.cassandra.io.sstable.SSTable.deleteIfCompacted(SSTable.java:136) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:202) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:342) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:308) > at org.apache.cassandra.db.Table.(Table.java:245) > at org.apache.cassandra.db.Table.open(Table.java:102) > at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:196) > at > org.apache.cassandra.db.StatisticsTable.deleteSSTableStatistics(StatisticsTable.java:81) > at > org.apache.cassandra.io.sstable.SSTable.deleteIfCompacted(SSTable.java:136) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:202) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:342) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:308) > at org.apache.cassandra.db.Table.(Table.java:245) > at org.apache.cassandra.db.Table.open(Table.java:102) > at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:196) > at > org.apache.cassandra.db.StatisticsTable.deleteSSTableStatistics(StatisticsTable.java:81) > at > org.apache.cassandra.io.sstable.SSTable.deleteIfCompacted(SSTable.java:136) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:202) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:342) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:308) > at org.apache.cassandra.db.Table.(Table.java:245) > at org.apache.cassandra.db.Table.open(Table.java:102) > at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:196) > at > org.apache.cassandra.db.StatisticsTable.deleteSSTableStatistics(StatisticsTable.java:81) > at > org.apache.cassandra.io.sstable.SSTable.deleteIfCompacted(SSTable.java:136) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:202) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:342) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:308) > at org.apache.cassandra.db.Table.(Table.java:245) > at org.apache.cassandra.db.Table.open(Table.java:102) > at org.apache.cassandra.db.RowMu
Re: RowMutationVerbHandler.java (line 78) Error in row mutation
Is it possible this was a new node with a manual token and autobootstrap turned off? If not, could you give more details about the node? Gary. On Fri, Aug 27, 2010 at 17:58, B. Todd Burruss wrote: > i got the latest code this morning. i'm testing with 0.7 > > > ERROR [ROW-MUTATION-STAGE:388] 2010-08-27 15:54:58,053 > RowMutationVerbHandler.java (line 78) Error in row mutation > org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find > cfId=1002 > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:113) > at > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:372) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:382) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:340) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > >
Re: Calls block when using Thrift API
If you're only interested in accessing data natively, I suggest you try the "fat client." It brings up a node that participates in gossip, exposes the StorageProxy API, but does not receive a token and so does not have storage responsibilities. StorageService.instance.initClient(); in 0.7 you will want to loop until the node receives its storage definitions via gossip. Calling SS.instance.initServer() directly bypasses some of the routine startup operations. I don't recommend it unless you really know what's going on (it might work now, but it's not guaranteed to in the future). Gary. On Sun, Aug 29, 2010 at 10:28, Ruben de Laat wrote: > Just for the people looking to run Cassandra embedded and access > directly (not via Thrift/Avro). > > This works: > > StorageService.instance.initServer(); > > And then just use the StorageProxy for data access. > > I have no idea if this is the right way, but is works. > > Kind regards, > Ruben > >
Re: Few questions regarding cassandra deployment on windows
On Tue, Sep 7, 2010 at 17:42, kannan chandrasekaran wrote: > Hi All, > > We are currently considering Cassandra for our application. > > Platform: > * a single-node cluster. > * windows '08 > * 64-bit jvm > > For the sake of brevity let, > Cassandra service = a single node cassandra server running as an embedded > service inside a JVM > > > My use cases: > 1) Start with a schema ( keyspace and set of column families under it) in a > cassandra service > 2) Need to be able to replicate the same schema structure (add new > keyspace/columnfamilies with different names ofcourse). > 3) Because of some existing limitations in my application, I need to be able > to write to the keyspace/column-families from a cassandra service and read > the written changes from a different cassandra service. Both the write and > the read "cassandra-services" are sharing the same Data directory. I > understand that the application has to take care of any naming collisions. > > > Couple Questions related to the above mentioned usecases: > 1) I want to spawn a new JVM and launch Cassandra as an embedded service > programatically instead of using the startup.bat. I would like to know if > that is possible and any pointers in that direction would be really helpful. > ( use-case1) There are a couple ways to do this. I've used two of them (tanuki and objectweb). Tanuki is better imo. I've never used it, but Apache procrun is probably worth checking out. > 2) I understand that there are provisions for live schema changes in 0.7 ( > thank you guys !!!), but since I cant use a beta version in production, I am > restricted to 0.6 for now. Is it possible to to support use-case 2 in 0.6.5 > ? More specifically, I am planning to make runtime changes to the > storage.conf xml file followed by a cassandra service restart Yes, but this is a manual process and will not scale well. > 3) Can I switch the data directory at run-time ? (use-case 3). In order to > not disrupt read while the writes are in progress, I am thinking something > like, copy the existing data-dir into a new location; write to a new data > directory; once the write is complete; switch pointers and restart the > cassandra service to read from the new directory to pick up the updated > changes > Pointing two cassandra instances at the same data directory? This is a bad idea. I've never tried it, so I don't know exactly what will happen, but I imagine you would corrupt your system tables pretty quickly and your commit log wouldn't be too happy either. This is a completely unsupported way of using cassandra. Gary
Re: Few questions regarding cassandra deployment on windows
On Thu, Sep 9, 2010 at 22:23, kannan chandrasekaran wrote: > > Thanks for the replies My comments in Bold... > Kannan > > > From: Gary Dusbabek > To: user@cassandra.apache.org > Sent: Thu, September 9, 2010 5:43:31 AM > Subject: Re: Few questions regarding cassandra deployment on windows > > On Tue, Sep 7, 2010 at 17:42, kannan chandrasekaran > wrote: > > Hi All, > > > > We are currently considering Cassandra for our application. > > > > Platform: > > * a single-node cluster. > > * windows '08 > > * 64-bit jvm > > > > For the sake of brevity let, > > Cassandra service = a single node cassandra server running as an embedded > > service inside a JVM > > > > > > My use cases: > > 1) Start with a schema ( keyspace and set of column families under it) in a > > cassandra service > > 2) Need to be able to replicate the same schema structure (add new > > keyspace/columnfamilies with different names ofcourse). > > 3) Because of some existing limitations in my application, I need to be able > > to write to the keyspace/column-families from a cassandra service and read > > the written changes from a different cassandra service. Both the write and > > the read "cassandra-services" are sharing the same Data directory. I > > understand that the application has to take care of any naming collisions. > > > > > > Couple Questions related to the above mentioned usecases: > > 1) I want to spawn a new JVM and launch Cassandra as an embedded service > > programatically instead of using the startup.bat. I would like to know if > > that is possible and any pointers in that direction would be really helpful. > > ( use-case1) > > There are a couple ways to do this. I've used two of them (tanuki and > objectweb). Tanuki is better imo. I've never used it, but Apache > procrun is probably worth checking out. > > > Someone pointed out > http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/ as > way to start it inside a jvm.. Do you still think tanuki/procrun is needed ? I misinterpreted your first email to be that you wanted to run as a windows service. You don't need tanuki/procrun if you just wish to simply run Cassandra. > > > 2) I understand that there are provisions for live schema changes in 0.7 ( > > thank you guys !!!), but since I cant use a beta version in production, I am > > restricted to 0.6 for now. Is it possible to to support use-case 2 in 0.6.5 > > ? More specifically, I am planning to make runtime changes to the > > storage.conf xml file followed by a cassandra service restart > > Yes, but this is a manual process and will not scale well. > > I agree, but I believe 0.7 solves this problem without a restart . Correct me > if I am wrong. Any ideas when the 0.7 is set to release ? When it's ready. :) > > > 3) Can I switch the data directory at run-time ? (use-case 3). In order to > > not disrupt read while the writes are in progress, I am thinking something > > like, copy the existing data-dir into a new location; write to a new data > > directory; once the write is complete; switch pointers and restart the > > cassandra service to read from the new directory to pick up the updated > > changes > > > > > Pointing two cassandra instances at the same data directory? This is > a bad idea. I've never tried it, so I don't know exactly what will > happen, but I imagine you would corrupt your system tables pretty > quickly and your commit log wouldn't be too happy either. This is a > completely unsupported way of using cassandra. > > Noted and will avoid it. But Can you please explain what happens if I write > to a different location and then copy the keyspace & the system directories > into an existing location ? Any ideas if that will work ? > As long as you take care to ensure that the replacement node uses the same token as the replaced node, nothing adverse will happen. Hints living throughout the cluster destined for the replaced node will become invalid if the IP is changed on the replacement node. Gary.
Re: Embedded Cassandra in JBoss
On Wed, Sep 15, 2010 at 09:17, Sandor Molnar wrote: > Hello guys! > > I just started researching today whether we can use Cassandra in our software > or not and I need to know if it is possible to use embedded Cassandra in > Jboss somehow. I googled it and I found several pages like > http://wiki.apache.org/cassandra/StorageProxy > http://wiki.apache.org/cassandra/Embedding > http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/ > http://itdevworld.wordpress.com/2010/04/04/embedding-cassandra-into-your-javaee-appserver/ > > but the picture is still not clear. Taking a look at o.a.c.s.EmbeddedCassandraService should help. > > What I would need is to have N JBoss instances; all of them have its own > Cassandra instance and a web application writes data into the embedded DB. > There is another JBoss instance which will use another web application to > read data from the previously mentioned Cassandra instances. > Is it complicated to achieve? > Any suggestions are appreciated. If each Cassandra instance will be on separate a node you should have no problem embedding cassandra inside another java process. With JBoss and Cassandra both using JMX, there might be a few things to trip over, but nothing insurmountable. If you want to run multiple embedded Cassandra instances on a single host (inside of a single parent java process or several), you are in for a fun ride. (Multiple Cassandras in a single process isn't possible yet.) Gary.
Re: Getting client only example to work
I discovered some problems with the fat client earlier this week when
I tried using it. It needs some fixes to keep up with all the 0.7
changes.
Gary.
On Thu, Sep 16, 2010 at 05:48, Asif Jan wrote:
>
> Hi
> I am using 0.7.0-beta1 , and trying to get the contrib/client_only example
> to work.
> I am running cassandra on host1, and trying to access it from host2.
> When using thirft (via cassandra-cli) and in my application; I am able to
> connect and do all operations as expected.
> But I am not able to connect to cassandra when using the code in client_only
> (or far that matter using contrib/bmt_example). Since my test requires to
> do bulk insertion of about 1.4 TB of data, so I need to use a non-thirft
> interface.
> The error that I am getting is follows (the keyspace and the column family
> exist and can be used via Thirft) :
> 10/09/16 12:35:31 INFO config.DatabaseDescriptor: DiskAccessMode 'auto'
> determined to be mmap, indexAccessMode is mmap
> 10/09/16 12:35:31 INFO service.StorageService: Starting up client gossip
> Exception in thread "main" java.lang.IllegalArgumentException: Unknown
> ColumnFamily Standard1 in keyspace Keyspace1
> at
> org.apache.cassandra.config.DatabaseDescriptor.getComparator(DatabaseDescriptor.java:1009)
> at
> org.apache.cassandra.db.ColumnFamily.getComparatorFor(ColumnFamily.java:418)
> at gaia.cu7.cassandra.input.Ingestor.testWriting(Ingestor.java:103)
> at gaia.cu7.cassandra.input.Ingestor.main(Ingestor.java:187)
> I am using the following code (from client_only example) (also passing JVM
> parameter -Dstorage-config=path_2_cassandra.yaml)
>
>
> public static void main(String[] args) throws Exception {
> System.setProperty("storage-config","cassandra.yaml");
> testWriting();
> }
>
> // from client_only example
> private static void testWriting() throws Exception
> {
> StorageService.instance.initClient();
> // sleep for a bit so that gossip can do its thing.
> try
> {
> Thread.sleep(1L);
> }
> catch (Exception ex)
> {
> throw new AssertionError(ex);
> }
> // do some writing.
> final AbstractType comp = ColumnFamily.getComparatorFor("Keyspace1",
> "Standard1", null);
> for (int i = 0; i < 100; i++)
> {
> RowMutation change = new RowMutation("Keyspace1", ("key" +
> i).getBytes());
> ColumnPath cp = new
> ColumnPath("Standard1").setColumn(("colb").getBytes());
> change.add(new QueryPath(cp), ("value" + i).getBytes(), new
> TimestampClock(0));
> // don't call change.apply(). The reason is that is makes a
> static call into Table, which will perform
> // local storage initialization, which creates local
> directories.
> // change.apply();
> StorageProxy.mutate(Arrays.asList(change));
> System.out.println("wrote key" + i);
> }
> System.out.println("Done writing.");
> StorageService.instance.stopClient();
> }
>
>
>
>
>
>
Re: 0.7 live schema updates
beta-2 will include the ability to set these values and others. Look for the system_update_column_family() and system_update_keyspace() methods. Gary. On Thu, Sep 16, 2010 at 02:38, Marc Canaleta wrote: > Hi! > I like the new feature of making live schema updates. You can add, drop and > rename columns and keyspaces via thrift, but how do you modify column > attributes like key_cache_size or rows_cached? > Thank you.
Re: FatClient Gossip error and some other problems
X.X has been silent for 360ms, removing from gossip > ERROR [Timer-0] 2010-09-20 13:56:11,421 Gossiper.java (line 99) Gossip error > java.util.ConcurrentModificationException > at java.util.Hashtable$Enumerator.next(Hashtable.java:1031) > at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:383) > at > org.apache.cassandra.gms.Gossiper$GossipTimerTask.run(Gossiper.java:93) > at java.util.TimerThread.mainLoop(Timer.java:512) > at java.util.TimerThread.run(Timer.java:462) > INFO [GMFD:1] 2010-09-20 13:56:43,251 Gossiper.java (line 586) Node > /X.X.X.X is now part of the cluster > > Does anyone have any idea how can I cleanup the problematic node? You may just need to be patient. Have you tried monitoring the CompactionManager in jmx to see if it is doing things? > Does anyone have any idea how can I get rid of the Gossip error? This is CASSANDRA-1494. You can ignore it. Gary.
Re: Building a local cluster in Linux
On Mon, Sep 20, 2010 at 10:02, Rodrigo Hjort wrote: > Hello all, > > In order to build a Cassandra cluster exclusively for availability and > replication testings, I thought of a simple solution, based on a single > Linux instance, with no virtualization at all. > > The idea was to initialize every node, run a testing client, and manually > kill some nodes processes in order to check the service availability and > data replication with several RFs (1,2,3). > > a) What do you think of this approach? Is there any improvement or step you > think that should be added? This approach has worked well for others that use it. One thing I recommend is making sure that all your nodes use a *different* port for JMX. > > b) The Thrift-based client in Java I used always pointed to node1, a single > point of failure threatening availability. What do you recommend using > instead, stress.py? Or come up with a way to randomize the host in your test client. Gary. > > Thanks in advance! > > > I created the following structure (with 3 nodes) on Linux: > > /opt/apache-cassandra-0.6.5/nodes > | > |-- node1 > | |-- bin > | |-- conf > | |-- data > | |-- log > | `-- txs > |-- node2 > | |-- bin > | |-- conf > | |-- data > | |-- log > | `-- txs > `-- node3 > |-- bin > |-- conf > |-- data > |-- log > `-- txs > > And below are the steps I took. > > > 1) create additional network interfaces using alias to loopback (lo) > > # ifconfig lo:2 127.0.0.2 up > # ifconfig lo:3 127.0.0.3 up > > $ ifconfig > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:30848 errors:0 dropped:0 overruns:0 frame:0 > TX packets:30848 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:2946793 (2.9 MB) TX bytes:2946793 (2.9 MB) > > lo:2 Link encap:Local Loopback > inet addr:127.0.0.2 Mask:255.0.0.0 > UP LOOPBACK RUNNING MTU:16436 Metric:1 > > lo:3 Link encap:Local Loopback > inet addr:127.0.0.3 Mask:255.0.0.0 > UP LOOPBACK RUNNING MTU:16436 Metric:1 > > That way, node1's IP is 127.0.0.1, node2's 127.0.0.2, and so forth. > > > 2) register node hostnames locally > > /etc/hosts: > 127.0.0.1 localhost node1 > 127.0.0.2 node2 > 127.0.0.3 node3 > > $ ping node2 > PING node2 (127.0.0.2) 56(84) bytes of data. > 64 bytes from node2 (127.0.0.2): icmp_seq=1 ttl=64 time=0.018 ms > 64 bytes from node2 (127.0.0.2): icmp_seq=2 ttl=64 time=0.015 ms > ^C > --- node2 ping statistics --- > 2 packets transmitted, 2 received, 0% packet loss, time 999ms > rtt min/avg/max/mdev = 0.015/0.016/0.018/0.004 ms > > > 3) create first node structure > > $ cd /opt/apache-cassandra-0.6.5/ > $ mkdir -p nodes/node1 > $ cp -R bin/ conf/ nodes/node1/ > $ cd nodes/node1 > > I opted to use relative paths instead of absolute for simplification > purposes. > > conf/log4j.properties: > > # Edit the next line to point to your logs directory > log4j.appender.R.File=./log/system.log > > conf/storage-conf.xml: > > ./txs > > ./data > > > node1 > 7000 > > > 9160 > false > > bin/cassandra.in.sh: > > for jar in $cassandra_home/../../lib/*.jar; do > CLASSPATH=$CLASSPATH:$jar > done > > > 4) create remaining nodes by cloning the first one > > $ cd .. ; mkdir node2 node3 > $ cp -R node1/* node2 > $ cp -R node1/* node3 > > $ tree -L 2 > . > |-- node1 > | |-- bin > | `-- conf > |-- node2 > | |-- bin > | `-- conf > `-- node3 > |-- bin > `-- conf > > Remaining directories (log, data, txs) are to be created automatically on > server startup. > > > 5) edit specific node settings > > Each node must listen to it's own hostname (i.e., node1, node2, node3). > > conf/storage-conf.xml: > > node2 > > JMX interfaces must be bound to the same host, thus we must change the port. > First node will be on 8081, node2 on 8082, and node3 on 8083. > > node1/bin/cassandra.in.sh: > > # Arguments to pass to the JVM > JVM_OPTS=" \ > -ea \ > -Xms1G \ > -Xmx1G \ > -XX:+UseParNewGC \ > -XX:+UseConcMarkSweepGC \ > -XX:+CMSParallelRemarkEnabled \ > -XX:SurvivorRatio=8 \ > -XX:MaxTenuringThreshold=1 \ > -XX:+HeapDumpOnOutOfMemoryError \ > -Dcom.sun.management.
Re: inter node protocol for 0.7 nightly
1465 maybe? On Mon, Sep 20, 2010 at 16:00, Aaron Morton wrote: > Just took a look upgrading from from 31/08 nightly to the 20/09 and noticed > the news.txt says... > "The Cassandra inter-node protocol is incompatible with 0.6.x releases (and > with 0.7 beta1)" > Could someone point me to the ticket(s) for this change so I can see if I > can do a rolling upgrade. > Thanks > Aaron >
Re: avro + cassandra + ruby
We have a system test that tests this (in avro python). see test/system/test_avro_standard.py:TestStandardOperations.test_multiget_slice_simple. On Wed, Sep 29, 2010 at 01:06, Gabor Torok wrote: > Hi, > I'm attempting to use avro to talk to cassandra because the ruby thrift > client's read performance is pretty bad (I measured 4x slower than java). > > However, I run into a problem when calling multiget_slice. > The server gives a KeyspaceNotDefinedException because > clientState.getKeyspace() returns null. > It seems this is because ClientState stores the keyspace in a ThreadLocal. > > I call set_keyspace and clientState stores the keyspace value. I guess the > next avro call to multiget_slice runs in a different thread so it can't > retrieve the value. > > In ruby, I use Avro::IPC::HTTPTransceiver as the transport which I believe is > a stateless transport. I also tried SocketTransport, but that died with a > malloc exception. > > Is this a problem with the ruby avro library (I use avro 1.4.0), or how the > server handles avro threads? > Any help would be appreciated! > > Thanks, > --Gabor >
Re: Hardware change of a node in the cluster
It should work this way: 1. Move your data to the new node (scp, etc.) 2. Make sure the new node is configured to use the same token as the new node. 3. Stand up the new node. 4. Turn off the old node. If your environment is volatile, it's probably best to run `nodetool repair` on the new node. Gary. On Mon, Oct 4, 2010 at 04:58, Utku Can Topçu wrote: > Hey All, > > Recently I've tried to upgrade (hw upgrade) one of the nodes in my cassandra > cluster from ec2-small to ec2-large. > > However, there were problems and since the IP of the new instance was > different from the previous instance. The other nodes didnot recognize it in > the ring. > > So what should be the best practice for a complete hardware change of one > node in the cluster while keeping the data that it has. > > Regards, > > Utku >
Re: replacing a dead node
On Mon, Oct 11, 2010 at 03:41, Chen Xinli wrote: > Hi, > > We have a cassandra cluster of 6 nodes with RF=3, read-repair enabled, > hinted handoff disabled, WRITE with QUORUM, READ with ONE. > we want to rely on read-repair totally for node failure, as returning > inconsistent result temporarily is ok for us. > > If a node is temporarily dead and returneded to the cluster later, > everything seems fine. > If a node is damaged and a new node with different IP is added, can I just > assign dead node's initialToken to him and have read-repair handle > everything? Yes. Just make sure auto-bootstrap is turned off, as you indicate below. Gary. > > I have read the wiki: > http://www.riptano.com/docs/0.6.5/operations/clustering . > But in order to keep it simple, I don't want to use autobootstrap, > noderepair, removetoken, etc > > -- > Best Regards, > Chen Xinli >
Re: Cassandra newbie question
On Mon, Oct 11, 2010 at 04:01, Arijit Mukherjee wrote: > Hi All > > I've just started reading about Cassandra and writing simple tests > using Cassandra 0.6.5 to see if we can use it for our product. > > I have a data store with a set of columns, like C1, C2, C3, and C4, > but the columns aren't mandatory. For example, there can be a list of > (k.v) pairs with only C1 and C2, but no C3 and C4. At the same time, > there can be a set of records with all the columns present. It's > possible to consider them as three sets A (with all columns), B (with > C1 and C2) and C (with C3 and C4). And I'm trying to find out the > following: > > 1. A - B (all records who don't have C3 and C4) and A - C (all record > who don't have C1 and C2) > 2. records for whom C2 != C4 > > It's possible to pick all records and do this processing in my client > code - but that won't perform well. Is there any way to do these > within Cassandra? For example, by passing a list of column names so > that cassandra returns the records with only those columns? multiget_slice with the SlicePredicate specified using column_names can do the lookups. As far as doing the set operations: no, Cassandra doesn't have the ability to do this server-side. Gary. > > Regards > Arijit >
Re: Null Pointer Exception / Secondary Indices
If you're not seeing the NullPointerException at this point, things are probably good. These messages are expected when logging at DEBUG. Gary. On Tue, Oct 12, 2010 at 02:35, J T wrote: > I rinsed & repeated after updating to the latest trunk version and checking > if the 1571 patch was included (which it appeared to be) and the start of > the repeating error sequence was: > DEBUG [pool-1-thread-21] 2010-10-12 08:30:31,143 ClientState.java (line 81) > logged in: # > DEBUG [pool-1-thread-21] 2010-10-12 08:30:31,166 CassandraServer.java (line > 543) scan > DEBUG [pool-1-thread-21] 2010-10-12 08:30:31,169 StorageProxy.java (line > 563) restricted single token match for query [0,0] > DEBUG [pool-1-thread-21] 2010-10-12 08:30:31,169 StorageProxy.java (line > 649) scan ranges are [0,0] > DEBUG [pool-1-thread-21] 2010-10-12 08:30:31,179 StorageProxy.java (line > 669) reading org.apache.cassandra.db.indexscancomm...@b3a5a0 from > 1...@localhost/127.0.0.1 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,194 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,198 SliceQueryFilter.java (line 122) > collecting 0 of 2147483647: is_confirmed:false:5...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,198 SliceQueryFilter.java (line 122) > collecting 1 of 2147483647: request_type:false:6...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,199 SliceQueryFilter.java (line 122) > collecting 2 of 2147483647: requested:false:5...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,200 SliceQueryFilter.java (line 122) > collecting 3 of 2147483647: requestor:false:5...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,200 SliceQueryFilter.java (line 122) > collecting 4 of 2147483647: requestor_network:false:5...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,206 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,209 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,211 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,214 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,217 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,220 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,223 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > DEBUG [ReadStage:6] 2010-10-12 08:30:31,225 SliceQueryFilter.java (line 122) > collecting 0 of 1: null:false:0...@1286840713 > > On Tue, Oct 12, 2010 at 7:50 AM, J T wrote: >> >> Hi, >> >> Looks like I was premature in my response. >> >> I had cause today to wipe my datastore and restart cassandra and reload >> the .yaml containing the schema definition. >> >> After doing a restart of my app which essentially inserted into a CF with >> a 2ndary idx and then queried that CF I was left with log files full of the >> following. >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,849 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,850 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,851 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,852 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:3] 2010-10-12 07:29:26,798 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,853 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:3] 2010-10-12 07:29:26,854 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,854 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:3] 2010-10-12 07:29:26,855 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:1] 2010-10-12 07:29:26,855 SliceQueryFilter.java (line >> 122) collecting 0 of 1: null:false:0...@1286840713 >> >> DEBUG [ReadStage:3] 2010-10-12 07:29:26,
Re: 0.7 beta2 - Fatal exception from RowMutationVerbHandler
Looks like a bug with hinted-handoff. Will you file a ticket? Gary On Tue, Oct 12, 2010 at 15:51, Michael Moores wrote: > I have a cluster of 8 nodes with a replication factor of 3 and consistency of > QUORUM. > When I stop one node in the cluster I end up with socket read timeouts to > other nodes: > org.apache.thrift.transport.TTransportException: > java.net.SocketTimeoutException: Read timed out > The timeout is set to 4 seconds. > > The recent write latency on the other nodes becomes elevated, reaching 5-6 > seconds. > The load on the cluster is minimal. > > I looked at the system log on an adjacent node in the ring and found this > output: > > INFO [WRITE-kv2-app02.dev.real.com/172.27.109.32] 2010-10-12 12:13:36,712 > OutboundTcpConnection.java (line 115) error writing to > kv2-app02.dev.real.com/172.27.109.32 > INFO [WRITE-kv2-app02.dev.real.com/172.27.109.32] 2010-10-12 12:13:50,336 > OutboundTcpConnection.java (line 115) error writing to > kv2-app02.dev.real.com/172.27.109.32 > INFO [Timer-0] 2010-10-12 12:14:22,792 Gossiper.java (line 196) InetAddress > /172.27.109.32 is now dead. > ERROR [MUTATION_STAGE:1315] 2010-10-12 12:14:24,917 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > ERROR [MUTATION_STAGE:1315] 2010-10-12 12:14:24,918 > AbstractCassandraDaemon.java (line 88) Fatal exception in thread > Thread[MUTATION_STAGE:1315,5,main] > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > ERROR [MUTATION_STAGE:1605] 2010-10-12 12:14:28,919 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > > > I restarted the previously stopped node, and the system recovers, but with a > few more underlflow exceptions: > > INFO [GOSSIP_STAGE:1] 2010-10-12 12:15:44,537 Gossiper.java (line 594) Node > /172.27.109.32 has restarted, now UP again > INFO [HINTED-HANDOFF-POOL:1] 2010-10-12 12:15:44,537 > HintedHandOffManager.java (line 196) Started hinted handoff for endpoint > /172.27.109.32 > INFO [GOSSIP_STAGE:1] 2010-10-12 12:15:44,537 StorageService.java (line 643) > Node /172.27.109.32 state jump to normal > INFO [HINTED-HANDOFF-POOL:1] 2010-10-12 12:15:44,538 > HintedHandOffManager.java (line 252) Finished hinted handoff of 0 rows to > endpoint /172.27.109.32 > INFO [GOSSIP_STAGE:1] 2010-10-12 12:15:44,538 StorageService.java (line 650) > Will not change my token ownership to /172.27.109.32 > ERROR [MUTATION_STAGE:1635] 2010-10-12 12:15:45,083 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > > >
Re: how to CompareWith LongType
Pycassa should just take your long and do the right thing with it
(packing it into a binary string) before passing it off to thrift.
The system tests in the source (test/system/test_thrift_server.py)
will give you a very good indication of how to do this. The long is
packed into a string using struct.pack('>p', n), where 'n' is your
long.
On Thu, Oct 14, 2010 at 06:53, shiwei ye wrote:
> How to insert a value to a column family CompareWith Longtype in python.I'm
> using pycassa.
> What should the type of the column be?Thx
>
>
Re: Hung Repair
Can you produce a thread dump on the machine? kill -3 ought to do it. JConsole can be your friend at a time like this too. It might be painstaking, but you can check the CPU time used by each thread using the java.lang.Threading mbean. There's an interesting jconsole plugin that is supposed to make this easier: http://lsd.luminis.nl/top-threads-plugin-for-jconsole/ Gary. On Fri, Oct 22, 2010 at 16:42, Dan Hendry wrote: > I am currently running a 4 node cluster on Cassandra beta 2. Yesterday, I > ran into a number of problems and the one of my nodes went down for a few > hours. I tried to run a nodetool repair and at least at a data level, > everything seems to be consistent and alright. The problem is that the node > is still chewing up 100% of its available CPU, 20 hours after I started the > repair. Load averages are 8-9 which is crazy given it is a single core ec2 > m1.small. > > > > Besides sitting at 100% cpu, the node on which I ran the repair seems to be > fine. The Cassandra logs appear normal. Based on bandwidth patterns between > nodes, it does not seem like they are transferring any repair related data > (as they did initially). No pending tasks are being shown in any of the > services when inspecting via jmx. I have a reasonable amount of data in the > cluster (~6 gb * 2 replication factor) but nothing crazy. The last repair > related entry in the logs is as follows: > > > > INFO [Thread-145] 2010-10-22 00:24:10,561 AntiEntropyService.java (line 828) > # /10.192.227.6, (kikmetrics,PacketEventsByPacket)> completed successfully: 14 > outstanding. > > > > Any idea what is going on? Could the CPU usage STILL be related to the > repair? Is there any way to check? I hesitate to simply kill the node given > the “14 outstanding” log message and as doing so has caused me problems in > the past when using beta versions. > > > > > > Dan Hendry > >
Re: batch_mutate in 0.7
client.set_keyspace() On Mon, Oct 25, 2010 at 12:59, Chris Oei wrote: > So, I'm a bit puzzled about how to change my old 0.6 code to 0.7. > In 0.6, I used: > client.batch_mutate(keySpace, mutationMap, ConsistencyLevel.ONE); > But in 0.7, batch_mutate no longer has a keyspace argument, so I used: > client.batch_mutate(mutationMap, ConsistencyLevel.ONE); > Not surprisingly, I got an error: > InvalidRequestException(why:You have not set a keyspace for this session) > Maybe I'm having Monday-morning brain-freeze, but I can't figure out how to > tell Cassandra which keyspace > to use. > Suggestions, anyone? > Thanks, > Chris
