Re: Creating counter columns in cassandra

[Abhijit Chanda]

There should be at least one "=" (equals) in the WHERE case on key or
secondary index column, this is the Cassandra limitation.

Re: Creating counter columns in cassandra

[Paolo Bernardi]

On Sun, Jul 29, 2012 at 9:30 AM, Abhijit Chanda
 wrote:
> There should be at least one "=" (equals) in the WHERE case on key or
> secondary index column, this is the Cassandra limitation.

Yep, it's still there (see validateFilterClauses from line 531):

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/thrift/ThriftValidation.java

It's not probably going to change anytime soon (or later, probably ;-)
), secondary indexes are still CFs underneath:

http://grokbase.com/t/cassandra/user/1099pty59r/indexoperator-lt-lte-gt-gte-implementation#20100910dx1fwm8z9ek1cvec10jvmpzfa4

>From that message you also get an hint on how to proceed in this situation:
1) Add a "fake" equality condition to the query;
2) Go for a range scan, which is more efficient than the hack above.

Depending on the speed than you need on the writing side compared to
the speed required on the reading side, you might also consider
keeping an ad-hoc index of the counter columns with the counter
greater than your threshold, but it's surely requiring more
housekeeping on your side.

Paolo

Re: RF on per column family basis ?

[Tyler Hobbs]

On Sat, Jul 28, 2012 at 4:21 PM, Ertio Lew  wrote:

> I heard that it is* not highly recommended* to create more than a single
> keyspace for an application or on a single cluster !?
>

The only possible issue there is with connections and connection pooling,
where you need to set a working keyspace for the connection.


>
> Moreover I fail to understand that why Cassandra puts this limitation to
> set RF on keyspace when, I guess, it makes more sense to do this on per CF
> basis !?
>

The understanding is that you will typically create different keyspaces for
data that has different replication needs.  Keyspace really only serve as a
level at which you set replication settings, nothing more.

-- 
Tyler Hobbs
DataStax 

Error occures while doing the draing Cassandra 1.1.2

[Roshan]

Hi

I got the below exception to Cassandra log while doing the *drain* via
*nodetool* operation before shutting down one node in 3 node development
Cassandra 1.1.2 cluster.

2012-07-30 09:37:45,347 ERROR [CustomTThreadPoolServer] Thrift error
occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in
readMessageBegin, old client?
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2012-07-30 09:38:02,322 INFO  [ColumnFamilyStore] Enqueuing flush of
Memtable-schema_keyspaces@29290924(202/252 serialized/live bytes, 4 ops)
2012-07-30 09:38:02,323 INFO  [Memtable] Writing
Memtable-schema_keyspaces@29290924(202/252 serialized/live bytes, 4 ops)
2012-07-30 09:38:02,470 INFO  [Memtable] Completed flushing
/data/cassandradb/data/system/schema_keyspaces/system-schema_keyspaces-hd-6-Data.db
(252 bytes) for commitlog position ReplayPosition(segmentId=245321525874562,
position=3546)
2012-07-30 09:58:31,015 INFO  [StorageService] DRAINING: starting drain
process
2012-07-30 09:58:31,015 INFO  [CassandraDaemon] Stop listening to thrift
clients
2012-07-30 09:58:31,020 INFO  [Gossiper] Announcing shutdown
2012-07-30 09:58:32,024 INFO  [MessagingService] Waiting for messaging
service to quiesce
2012-07-30 09:58:32,027 INFO  [MessagingService] MessagingService shutting
down server thread.
2012-07-30 09:58:32,088 INFO  [StorageService] DRAINED

Any thoughts?



--
View this message in context: 
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Error-occures-while-doing-the-draing-Cassandra-1-1-2-tp7581493.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
Nabble.com.

Practical node size limits

[Dustin Wenz]

I'm trying to determine if there are any practical limits on the amount of data 
that a single node can handle efficiently, and if so, whether I've hit that 
limit or not.

We've just set up a new 7-node cluster with Cassandra 1.1.2 running under 
OpenJDK6. Each node is 12-core Xeon with 24GB of RAM and is connected to a 
stripe of 10 3TB disk mirrors (a total of 20 spindles each) and connected via 
dual SATA-3 interconnects. I can read and write around 900MB/s sequentially on 
the arrays. I started out with Cassandra tuned with all-default values, with 
the exception of the compaction throughput which was increased from 16MB/s to 
100MB/s. These defaults will set the heap size to 6GB.

Our schema is pretty simple; only 4 column families and each has one secondary 
index. The replication factor was set to four, and compression disabled. Our 
access patterns are intended to be about equal numbers of inserts and selects, 
with no updates, and the occasional delete.

The first thing we did was begin to load data into the cluster. We could 
perform about 3000 inserts per second, which stayed mostly flat. Things started 
to go wrong around the time the nodes exceeded 800GB. Cassandra began to 
generate a lot of "mutations messages dropped" warnings, and was complaining 
that the heap was over 75% capacity.

At that point, we stopped all activity on the cluster and attempted a repair. 
We did this so we could be sure that the data was fully consistent before 
continuing. Our mistake was probably trying to repair all of the nodes 
simultaneously - within an hour, Java terminated on one of the nodes with a 
heap out-of-memory message. I then increased all of the heap sizes to 8GB, and 
reduced the heap_newsize to 800MB. All of the nodes were restarted, and there 
was no no outside activity on the cluster. I then began a repair on a single 
node. Within a few hours, it OOMed again and exited. I then increased the heap 
to 12GB, and attempted the same thing. This time, the repair ran for about 7 
hours before exiting from an OOM condition.

By now, the repair had increased the amount of data on some of the nodes to 
over 1.2TB. There is no going back to a 6GB heap size - Cassandra now exits 
with an OOM during startup unless the heap is set higher. It's at 16GB now, and 
a single node has been repairing for a couple of days. Though I have no 
personal experience with this, I've been told that Java's garbage collector 
doesn't perform well with heaps above 8GB. I'm wary of setting it higher, but I 
can add up to 192GB of RAM to each node if necessary.

How much heap does cassandra need for this amount of data with only four CFs? 
Am I scaling this cluster in completely the wrong direction? Is there a magic 
garbage collection setting that I need to add in cassandra-env that isn't there 
by default?

Thanks,

  - .Dustin

Re: Practical node size limits

[Edward Capriolo]

Yikes. You should read:

http://wiki.apache.org/cassandra/LargeDataSetConsiderations

Essentially what it sounds like your are now running into is this:

The BloomFilters for each SSTable must exist in main memory. Repair
tends to create some extra data which normally gets compacted away
later.

Your best bet is to temporarily raise the Xmx heap and adjust the
index sampling size. If you need to save the data (if it is just test
data you may want to give up and start fresh)

Generally the issue with the large disk configurations it is hard to
keep a good ram/disk ratio. Then most reads turn into disk seeks and
the throughput is low. I get the vibe people believe large stripes are
going to help Cassandra. The issue is that stripes generally only
increase sequential throughput, but Cassandra is a random read system.

How much ram/disk you need is case dependent but 1/5 ratio of RAM to
disk is where I think most people want to be, unless their system is
carrying SSD disks.

Again you have to keep your bloom filters in java heap memory so and
design that tries to create a quatrillion small rows is going to have
memory issues as well.

On Sun, Jul 29, 2012 at 10:40 PM, Dustin Wenz  wrote:
> I'm trying to determine if there are any practical limits on the amount of 
> data that a single node can handle efficiently, and if so, whether I've hit 
> that limit or not.
>
> We've just set up a new 7-node cluster with Cassandra 1.1.2 running under 
> OpenJDK6. Each node is 12-core Xeon with 24GB of RAM and is connected to a 
> stripe of 10 3TB disk mirrors (a total of 20 spindles each) and connected via 
> dual SATA-3 interconnects. I can read and write around 900MB/s sequentially 
> on the arrays. I started out with Cassandra tuned with all-default values, 
> with the exception of the compaction throughput which was increased from 
> 16MB/s to 100MB/s. These defaults will set the heap size to 6GB.
>
> Our schema is pretty simple; only 4 column families and each has one 
> secondary index. The replication factor was set to four, and compression 
> disabled. Our access patterns are intended to be about equal numbers of 
> inserts and selects, with no updates, and the occasional delete.
>
> The first thing we did was begin to load data into the cluster. We could 
> perform about 3000 inserts per second, which stayed mostly flat. Things 
> started to go wrong around the time the nodes exceeded 800GB. Cassandra began 
> to generate a lot of "mutations messages dropped" warnings, and was 
> complaining that the heap was over 75% capacity.
>
> At that point, we stopped all activity on the cluster and attempted a repair. 
> We did this so we could be sure that the data was fully consistent before 
> continuing. Our mistake was probably trying to repair all of the nodes 
> simultaneously - within an hour, Java terminated on one of the nodes with a 
> heap out-of-memory message. I then increased all of the heap sizes to 8GB, 
> and reduced the heap_newsize to 800MB. All of the nodes were restarted, and 
> there was no no outside activity on the cluster. I then began a repair on a 
> single node. Within a few hours, it OOMed again and exited. I then increased 
> the heap to 12GB, and attempted the same thing. This time, the repair ran for 
> about 7 hours before exiting from an OOM condition.
>
> By now, the repair had increased the amount of data on some of the nodes to 
> over 1.2TB. There is no going back to a 6GB heap size - Cassandra now exits 
> with an OOM during startup unless the heap is set higher. It's at 16GB now, 
> and a single node has been repairing for a couple of days. Though I have no 
> personal experience with this, I've been told that Java's garbage collector 
> doesn't perform well with heaps above 8GB. I'm wary of setting it higher, but 
> I can add up to 192GB of RAM to each node if necessary.
>
> How much heap does cassandra need for this amount of data with only four CFs? 
> Am I scaling this cluster in completely the wrong direction? Is there a magic 
> garbage collection setting that I need to add in cassandra-env that isn't 
> there by default?
>
> Thanks,
>
>   - .Dustin

Cassandra 1.0.6 nodetool drain gives lots of batch_mutate exceptions

[Roshan]

Hi

As a part of the Cassandra upgrade to 1.1.2 from 1.0.6, I am running
*nodetool drain* node by node to empty the commit logs. When draining a
particular node, that node accepting READ+WRITE request from the clients and
giving below exceptions.

2012-07-30 23:08:18,169 ERROR [Cassandra$Processor] Internal error
processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
at 
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
at 
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
at
org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2012-07-30 23:08:18,174 ERROR [AbstractCassandraDaemon] Fatal exception in
thread Thread[Thread-6,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
2012-07-30 23:08:18,177 ERROR [AbstractCassandraDaemon] Fatal exception in
thread Thread[Thread-10,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:511)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
2012-07-30 23:08:18,183 ERROR [Cassandra$Processor] Internal error
processing batch_mutate
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut
down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:420)
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:308)
at 
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:120)
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:255)
at 
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:194)
at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:638)
at
org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:589)
at
org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:597)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.Custo

DROP keyspace doesn't delete the files

[Drew Kutcharian]

Hi,

What's the correct procedure to drop a keyspace? When I drop a keyspace, the 
files of that keyspace don't get deleted. There is a JIRA on this:

https://issues.apache.org/jira/browse/CASSANDRA-4075

Is this a bug or I'm missing something?

I'm using Cassandra 1.1.2 on Ubuntu Linux with Sun JVM 1.6, 64bit

Thanks,

Drew